[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [libvirt] blkio cgroup



On Mon, Feb 21, 2011 at 09:30:32AM +0100, Dominik Klein wrote:
> On 02/21/2011 09:19 AM, Dominik Klein wrote:
> >>> - Is it possible to capture 10-15 second blktrace on your underlying
> >>>   physical device. That should give me some idea what's happening.
> >>
> >> Will do, read on.
> > 
> > Just realized I missed this one ... Had better done it right away.
> > 
> > So here goes.
> > 
> > Setup as in first email. 8 Machines, 2 important, 6 not important ones
> > with a throttle of ~10M. group_isolation=1. Each vm dd'ing zeroes.
> > 
> > blktrace -d /dev/sdb -w 30
> > === sdb ===
> >   CPU  0:                 4769 events,      224 KiB data
> >   CPU  1:                28079 events,     1317 KiB data
> >   CPU  2:                 1179 events,       56 KiB data
> >   CPU  3:                 5529 events,      260 KiB data
> >   CPU  4:                  295 events,       14 KiB data
> >   CPU  5:                  649 events,       31 KiB data
> >   CPU  6:                  185 events,        9 KiB data
> >   CPU  7:                  180 events,        9 KiB data
> >   CPU  8:                   17 events,        1 KiB data
> >   CPU  9:                   12 events,        1 KiB data
> >   CPU 10:                    6 events,        1 KiB data
> >   CPU 11:                   55 events,        3 KiB data
> >   CPU 12:                28005 events,     1313 KiB data
> >   CPU 13:                 1542 events,       73 KiB data
> >   CPU 14:                 4814 events,      226 KiB data
> >   CPU 15:                  389 events,       19 KiB data
> >   CPU 16:                 1545 events,       73 KiB data
> >   CPU 17:                  119 events,        6 KiB data
> >   CPU 18:                 3019 events,      142 KiB data
> >   CPU 19:                   62 events,        3 KiB data
> >   CPU 20:                  800 events,       38 KiB data
> >   CPU 21:                   17 events,        1 KiB data
> >   CPU 22:                  243 events,       12 KiB data
> >   CPU 23:                    1 events,        1 KiB data
> >   Total:                 81511 events (dropped 0),     3822 KiB data
> > 
> > Very constant 296 blocked processes in vmstat during this run. But...
> > apparently no data is written at all (see "bo" column).

Hm..., this sounds bad. If you have put a limit of ~10Mb/s then no
"bo" is bad. That would explain that why your box is not responding
and you need to do power reset.

- I am assuming that you have not put any throttling limits on root group.
  Is your system root also on /dev/sdb or on a separate disk altogether.

- This sounds like a bug in throttling logic. To narrow it down can you
  start running "deadline" on end device. If it still happens, it is more
  or less in throttling layer.

- We can also try to remove dm layers and just create partitions on 
  /dev/sdb and export as virtio disks to virtual machines and take
  dm layer out of picture and see if it still happens.

- In one of the mails you mentioned that with 1 virutal machine throttling
  READs and WRITEs is working for you. So it looks like 1 virtual machine
  does not hang but once you launch 8 virtual machines it hangs. Can we
  try increasing the number of vitual machines gragually and confirm that
  it happens only if some certain number of virtual machines are launched.

- Can you also paste me the rules you have put on important and non-important
  groups. Somehow I suspect that some of the rule has gone horribly bad
  in the sense that it is very low and effectively no virtual machine
  is making any progress.

- How long does it take to reach in this locked state where bo=0.

- you can also try to redirect blktrace output to blkparse and redirect
  it to standard output and see capture some output by copying pasting
  last messages.

In the mean time, I will try to launch more machines and see if I can
reproduce the issue.

Thanks
Vivek 


> > 
> > vmstat 2
> > procs -----------memory---------- ---swap-- -----io---- -system--
> > ----cpu----
> >  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
> > id wa
> >  0 296      0 125254224  21432 142016    0    0    16   633  181  331  0
> >  0 93  7
> >  0 296      0 125253728  21432 142016    0    0     0     0 17115 33794
> >  0  0 25 75
> >  0 296      0 125254112  21432 142016    0    0     0     0 17084 33721
> >  0  0 25 74
> >  1 296      0 125254352  21440 142012    0    0     0    18 17047 33736
> >  0  0 25 75
> >  0 296      0 125304224  21440 131060    0    0     0     0 17630 33989
> >  0  1 23 76
> >  1 296      0 125306496  21440 130260    0    0     0     0 16810 33401
> >  0  0 20 80
> >  4 296      0 125307208  21440 129856    0    0     0     0 17169 33744
> >  0  0 26 74
> >  0 296      0 125307496  21448 129508    0    0     0    14 17105 33650
> >  0  0 36 64
> >  0 296      0 125307712  21452 129672    0    0     2  1340 17117 33674
> >  0  0 22 78
> >  1 296      0 125307752  21452 129520    0    0     0     0 16875 33438
> >  0  0 29 70
> >  1 296      0 125307776  21452 129520    0    0     0     0 16959 33560
> >  0  0 21 79
> >  1 296      0 125307792  21460 129520    0    0     0    12 16700 33239
> >  0  0 15 85
> >  1 296      0 125307808  21460 129520    0    0     0     0 16750 33274
> >  0  0 25 74
> >  1 296      0 125307808  21460 129520    0    0     0     0 17020 33601
> >  0  0 26 74
> >  1 296      0 125308272  21460 129520    0    0     0     0 17080 33616
> >  0  0 20 80
> >  1 296      0 125308408  21460 129520    0    0     0     0 16428 32972
> >  0  0 42 58
> >  1 296      0 125308016  21460 129524    0    0     0     0 17021 33624
> >  0  0 22 77
> 
> While we're on that ... It is impossible for me now to recover from this
> state without pulling the power plug.
> 
> On the VMs console I see messages like
> INFO: task (kjournald|flush-254|dd|rs:main|...) blocked for more than
> 120 seconds.

If VMs are completely blocked and not making any progress, it is expected.
> 
> While the ssh sessions through which the dd was started seem intact
> (pressing enter gives a new line), it is impossible to cancel the dd
> command. Logging in on the VMs console also is impossible.
> 
> Opening a new ssh session to the host also does not work. Killing the
> qemu-kvm processes from a session opened earlier leaves zomby processes.
> Moving the VMs back to the root cgroup makes no difference either.
> 
> Regards
> Dominik
> 
> --
> libvir-list mailing list
> libvir-list redhat com
> https://www.redhat.com/mailman/listinfo/libvir-list


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]