[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [libvirt] blkio cgroup



Hi

to be as verbose and clear as possible, I will write every command I use.

This sets everything up for the test:

# setup start

importantvms="kernel1 kernel2"
notimportantvms="kernel3 kernel4 kernel5 kernel6 kernel7 kernel8"

for vm in $importantvms $notimportantvms; do
virsh domstate $vm|grep -q running || virsh start $vm;
done

echo 1 > /sys/block/sdb/queue/iosched/group_isolation

mount -t cgroup -o blkio none /mnt
cd /mnt
mkdir important
mkdir notimportant

for vm in $notimportantvms; do
cd /proc/$(pgrep -f "qemu-kvm.*$vm")/task
for task in *; do
/bin/echo $task > /mnt/notimportant/tasks
done
done

for vm in $importantvms; do
cd /proc/$(pgrep -f "qemu-kvm.*$vm")/task
for task in *; do
/bin/echo $task > /mnt/important/tasks
done
done

#ls -lH /dev/vdisks/kernel[3-8]
#brw-rw---- 1 root root 254, 25 Feb 22 13:42 kernel3
#brw-rw---- 1 root root 254, 26 Feb 22 13:42 kernel4
#brw-rw---- 1 root root 254, 27 Feb 22 13:42 kernel5
#brw-rw---- 1 root root 254, 28 Feb 22 13:42 kernel6
#brw-rw---- 1 root root 254, 29 Feb 22 13:43 kernel7
#brw-rw---- 1 root root 254, 30 Feb 22 13:43 kernel8

for i in $(seq 25 30); do
/bin/echo 254:$i 10000000 >
/mnt/notimportant/blkio.throttle.write_bps_device
done

# setup complete

> Hm..., this sounds bad. If you have put a limit of ~10Mb/s then no
> "bo" is bad. That would explain that why your box is not responding
> and you need to do power reset.
> 
> - I am assuming that you have not put any throttling limits on root group.
>   Is your system root also on /dev/sdb or on a separate disk altogether.

No throttling on root. Correct.

system root is on sda
vms are on sdb

> - This sounds like a bug in throttling logic. To narrow it down can you
>   start running "deadline" on end device. If it still happens, it is more
>   or less in throttling layer.

cat /sys/block/sdb/queue/scheduler
noop deadline [cfq]
echo deadline > /sys/block/sdb/queue/scheduler
cat /sys/block/sdb/queue/scheduler
noop [deadline] cfq

This changes things:

vmstat before test (nothing going on):
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
id wa
 1  0      0 130231696  17968  69424    0    0     0     0 16573 32834
0  0 100  0

vmstat during test while all 8 vms are writing (2 unthrottled, 6 at 10M):
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
id wa
 2  9      0 126257984  17968  69424    0    0     8 164462 20114 36751
 5  4 52 39

vmstat during test when only throttled vms are still dd'ing
 2 21      0 124518928  17976  69424    0    0     0 63876 17410 33670
1  1 59 39

Load is 12-ish during the test and the results are as expected:
Throttled VMs write with approx 10M, unthrottled get like 80 each, which
sums up to about the 200M max capacity of the device for direct io.

> - We can also try to remove dm layers and just create partitions on 
>   /dev/sdb and export as virtio disks to virtual machines and take
>   dm layer out of picture and see if it still happens.

Did that. Also happens when scheduler is cfq. Also does not happen when
scheduler is deadline. So it does not seem to be a dm issue.

> - In one of the mails you mentioned that with 1 virutal machine throttling
>   READs and WRITEs is working for you. So it looks like 1 virtual machine
>   does not hang but once you launch 8 virtual machines it hangs. Can we
>   try increasing the number of vitual machines gragually and confirm that
>   it happens only if some certain number of virtual machines are launched.

Back on cfq here.

1 throttled vm: works, load ~4
2 throttled vms: works, load ~6
3 throttled vms: works, load ~9
4 throttled vms: works, load ~12
6 throttled vms: works, load ~20

The number of blocked threads increases with the number of vms dd'ing.

At the beginning of each test, the blocked threads number goes really
high (4 vms 160, 6 vms 220), but then drops significantly and stays low.

So it seems that when only throttled vms are running, the problem does
not occur.

1 throttled + 1 unthrottled vm: works, load ~5
2 throttled + 1 unthrottled vm: boom

Constantly 144 blocked threads, bo=0, load increasing to 144. System
needs power reset.

So, thinking about what I did in the initial setup, I re-tested without the

for vm in $importantvms; do
cd /proc/$(pgrep -f "qemu-kvm.*$vm")/task
for task in *; do
/bin/echo $task > /mnt/important/tasks
done
done

since I don't do anything with that "important" cgroup (yet) anyway. It
did not make a difference though.

> - Can you also paste me the rules you have put on important and non-important
>   groups. Somehow I suspect that some of the rule has gone horribly bad
>   in the sense that it is very low and effectively no virtual machine
>   is making any progress.

See setup commands in the beginning of this email.

> - How long does it take to reach in this locked state where bo=0.

It goes there "instantly", right after the dd commands start.

> - you can also try to redirect blktrace output to blkparse and redirect
>   it to standard output and see capture some output by copying pasting
>   last messages.

I hope this is what you meant:

blktrace -d /dev/sdb -o - | blkparse -i -

Output is in attachment.

Regards
Dominik
  8,16   3     3927    55.798466536  3693  A  WS 3407889504 + 480 <- (8,17) 3407887456
  8,16   3     3928    55.798467619  3693  Q  WS 3407889504 + 480 [qemu-kvm]
  8,16   3     3929    55.798470220  3693  G  WS 3407889504 + 480 [qemu-kvm]
  8,16   3     3930    55.798471438  3693  I   W 3407889504 + 480 [qemu-kvm]
  8,16   3        0    55.798471876     0  m   N cfq3693 insert_request
  8,17   3     3931    55.798491328  3693  A  WS 3407887936 + 480 <- (254,23) 33569984
  8,16   3     3932    55.798491556  3693  A  WS 3407889984 + 480 <- (8,17) 3407887936
  8,16   3     3933    55.798491904  3693  Q  WS 3407889984 + 480 [qemu-kvm]
  8,16   3     3934    55.798492752  3693  G  WS 3407889984 + 480 [qemu-kvm]
  8,16   3     3935    55.798493994  3693  I   W 3407889984 + 480 [qemu-kvm]
  8,16   3        0    55.798494516     0  m   N cfq3693 insert_request
  8,17   3     3936    55.798510421  3693  A  WS 3407888416 + 480 <- (254,23) 33570464
  8,16   3     3937    55.798510610  3693  A  WS 3407890464 + 480 <- (8,17) 3407888416
  8,16   3     3938    55.798510926  3693  Q  WS 3407890464 + 480 [qemu-kvm]
  8,16   3     3939    55.798511583  3693  G  WS 3407890464 + 480 [qemu-kvm]
  8,16   3     3940    55.798512645  3693  I   W 3407890464 + 480 [qemu-kvm]
  8,16   3        0    55.798518244     0  m   N cfq3693 insert_request
  8,17   3     3941    55.798535943  3693  A  WS 3407888896 + 480 <- (254,23) 33570944
  8,16   3     3942    55.798536140  3693  A  WS 3407890944 + 480 <- (8,17) 3407888896
  8,16   3     3943    55.798536454  3693  Q  WS 3407890944 + 480 [qemu-kvm]
  8,16   3     3944    55.798537257  3693  G  WS 3407890944 + 480 [qemu-kvm]
  8,16   3     3945    55.798538417  3693  I   W 3407890944 + 480 [qemu-kvm]
  8,16   3        0    55.798538844     0  m   N cfq3693 insert_request
  8,16   9      404    55.798556926  3564  U   N [qemu-kvm] 132
  8,17   3     3946    55.798557461  3693  A  WS 3407889376 + 480 <- (254,23) 33571424
  8,16   3     3947    55.798557645  3693  A  WS 3407891424 + 480 <- (8,17) 3407889376
  8,16   3     3948    55.798557990  3693  Q  WS 3407891424 + 480 [qemu-kvm]
  8,16   3     3949    55.798558736  3693  G  WS 3407891424 + 480 [qemu-kvm]
  8,16   3     3950    55.798560030  3693  I   W 3407891424 + 480 [qemu-kvm]
  8,16   3        0    55.798560712     0  m   N cfq3693 insert_request
  8,17   3     3951    55.798576762  3693  A  WS 3407889856 + 152 <- (254,23) 33571904
  8,16   3     3952    55.798577074  3693  A  WS 3407891904 + 152 <- (8,17) 3407889856
  8,16   3     3953    55.798577380  3693  Q  WS 3407891904 + 152 [qemu-kvm]
  8,16   3     3954    55.798578078  3693  G  WS 3407891904 + 152 [qemu-kvm]
  8,16   3     3955    55.798578734  3693  I   W 3407891904 + 152 [qemu-kvm]
  8,16   3        0    55.798579207     0  m   N cfq3693 insert_request
  8,16   3     3956    55.798580865  3693  U   N [qemu-kvm] 134
  8,17  19      950    55.798590132  3698  A  WS 3407891072 + 480 <- (254,23) 33573120
  8,16  19      951    55.798591023  3698  A  WS 3407893120 + 480 <- (8,17) 3407891072
  8,16  19      952    55.798592021  3698  Q  WS 3407893120 + 480 [qemu-kvm]
  8,16  19      953    55.798600061  3698  S  WS 3407893120 + 480 [qemu-kvm]
  8,16   6      688    55.799223015  3699  U   N [qemu-kvm] 134
  8,16   6      689    55.799819390  3700  U   N [qemu-kvm] 134
  8,16   6      690    55.800236635  3701  U   N [qemu-kvm] 134
  8,16   7      880    55.800279272  3702  U   N [qemu-kvm] 134
  8,16   6      691    55.800831081  3703  U   N [qemu-kvm] 134
  8,16   6      692    55.801516229  3704  U   N [qemu-kvm] 134
  8,16   6      693    55.801860219  3705  U   N [qemu-kvm] 134
  8,16   6      694    55.802110337  3706  U   N [qemu-kvm] 134
  8,16   6      695    55.804130110  3707  U   N [qemu-kvm] 134
  8,16   1        0    55.806298452     0  m   N cfq idle timer fired
  8,16   1        0    55.806300924     0  m   N cfq3564 slice expired t=0
  8,16   1        0    55.806303380     0  m   N cfq3564 sl_used=2 disp=1 charge=2 iops=0 sect=8
  8,16   1        0    55.806303770     0  m   N cfq3564 del_from_rr
  8,16   1        0    55.806304555     0  m   N cfq schedule dispatch
  8,17   9      405    55.806320403   484  A  WS 3659277360 + 480 <- (254,29) 33301168
  8,16   9      406    55.806321212   484  A  WS 3659279408 + 480 <- (8,17) 3659277360
  8,17   1    13713    55.806326506   454  A  WS 3701251800 + 480 <- (254,30) 33332568
  8,16   1    13714    55.806327201   454  A  WS 3701253848 + 480 <- (8,17) 3701251800
  8,16   9      407    55.806328664   484  Q  WS 3659279408 + 480 [kworker/9:1]
  8,16   1    13715    55.806335493   454  Q  WS 3701253848 + 480 [kworker/1:1]
  8,17  19      954    55.806340403   520  A  WS 3533640576 + 480 <- (254,26) 33493504
  8,16   9      408    55.806340854   484  S  WS 3659279408 + 480 [kworker/9:1]
  8,16  19      955    55.806341348   520  A  WS 3533642624 + 480 <- (8,17) 3533640576
  8,16   1    13716    55.806347196   454  S  WS 3701253848 + 480 [kworker/1:1]
  8,16  19      956    55.806356670   520  Q  WS 3533642624 + 480 [kworker/19:1]
  8,16  19      957    55.806364627   520  S  WS 3533642624 + 480 [kworker/19:1]
  8,17  14     3809    55.808056903  3708  A  WS 3449578144 + 480 <- (254,24) 33317152
  8,16  14     3810    55.808057746  3708  A  WS 3449580192 + 480 <- (8,17) 3449578144
  8,16  14     3811    55.808058575  3708  Q  WS 3449580192 + 480 [qemu-kvm]
  8,16  14     3812    55.808065313  3708  S  WS 3449580192 + 480 [qemu-kvm]
  8,16  19      958    55.808281456  3709  U   N [qemu-kvm] 134
  8,16  14     3813    55.808479046  3710  U   N [qemu-kvm] 134
  8,16  14     3814    55.808610181  3711  U   N [qemu-kvm] 134
  8,17  14     3815    55.808643307  3712  A  WS 3449581176 + 104 <- (254,24) 33320184
  8,16  14     3816    55.808643709  3712  A  WS 3449583224 + 104 <- (8,17) 3449581176
  8,16  14     3817    55.808644114  3712  Q  WS 3449583224 + 104 [qemu-kvm]
  8,16  14     3818    55.808645156  3712  S  WS 3449583224 + 104 [qemu-kvm]
  8,16  14     3819    55.808941624  3713  U   N [qemu-kvm] 134
  8,17   6      696    55.809300401  3714  A  WS 3449581296 + 480 <- (254,24) 33320304
  8,16   6      697    55.809301256  3714  A  WS 3449583344 + 480 <- (8,17) 3449581296
  8,16   6      698    55.809302071  3714  Q  WS 3449583344 + 480 [qemu-kvm]
  8,16   6      699    55.809307149  3714  S  WS 3449583344 + 480 [qemu-kvm]
  8,16   7      881    55.809415138  3715  U   N [qemu-kvm] 134
  8,17  14     3820    55.809630538  3717  A  WS 3407893096 + 152 <- (254,23) 33575144
  8,16  14     3821    55.809631395  3717  A  WS 3407895144 + 152 <- (8,17) 3407893096
  8,16  14     3822    55.809632498  3717  Q  WS 3407895144 + 152 [qemu-kvm]
  8,16  14     3823    55.809637373  3717  S  WS 3407895144 + 152 [qemu-kvm]
  8,16  19      959    55.809652187  3716  U   N [qemu-kvm] 134
  8,17   6      700    55.809726441  3718  A  WS 3449583320 + 480 <- (254,24) 33322328
  8,16   6      701    55.809726919  3718  A  WS 3449585368 + 480 <- (8,17) 3449583320
  8,16   6      702    55.809727583  3718  Q  WS 3449585368 + 480 [qemu-kvm]
  8,16   6      703    55.809728880  3718  S  WS 3449585368 + 480 [qemu-kvm]
  8,17   6      704    55.810330315  3720  A  WS 3449586352 + 480 <- (254,24) 33325360
  8,16   6      705    55.810330740  3720  A  WS 3449588400 + 480 <- (8,17) 3449586352
  8,16   6      706    55.810330979  3720  Q  WS 3449588400 + 480 [qemu-kvm]
  8,16   6      707    55.810331793  3720  S  WS 3449588400 + 480 [qemu-kvm]
  8,17   7      882    55.810342846  3719  A  WS 3407893248 + 152 <- (254,23) 33575296
  8,16   7      883    55.810343814  3719  A  WS 3407895296 + 152 <- (8,17) 3407893248
  8,16   7      884    55.810344524  3719  Q  WS 3407895296 + 152 [qemu-kvm]
  8,16   7      885    55.810348787  3719  S  WS 3407895296 + 152 [qemu-kvm]
  8,16   7      886    55.810963240  3721  U   N [qemu-kvm] 134
  8,17   7      887    55.811078088  3722  A  WS 3449589952 + 480 <- (254,24) 33328960
  8,16   7      888    55.811078908  3722  A  WS 3449592000 + 480 <- (8,17) 3449589952
  8,16   7      889    55.811079530  3722  Q  WS 3449592000 + 480 [qemu-kvm]
  8,16   7      890    55.811082649  3722  S  WS 3449592000 + 480 [qemu-kvm]
  8,16   7      891    55.811537529  3723  U   N [qemu-kvm] 134
  8,16   7      892    55.812222597  3724  U   N [qemu-kvm] 134
  8,16   6      708    55.812245576  3725  U   N [qemu-kvm] 134
  8,16   7      893    55.812274987  3726  U   N [qemu-kvm] 134
  8,16   6      709    55.812797561  3727  U   N [qemu-kvm] 134
  8,16  23       64    55.812929114  3728  U   N [qemu-kvm] 134
  8,16   7      894    55.813528023  3729  U   N [qemu-kvm] 134
  8,16   6      710    55.813687257  3730  U   N [qemu-kvm] 134
  8,17   7      895    55.814660300  3731  A  WS 3407893408 + 480 <- (254,23) 33575456
  8,16   7      896    55.814661094  3731  A  WS 3407895456 + 480 <- (8,17) 3407893408
  8,16   7      897    55.814661536  3731  Q  WS 3407895456 + 480 [qemu-kvm]
  8,16   7      898    55.814665009  3731  S  WS 3407895456 + 480 [qemu-kvm]
  8,16   6      711    55.814888432  3732  U   N [qemu-kvm] 134
  8,16   6      712    55.815519196  3733  U   N [qemu-kvm] 134
  8,16   6      713    55.815945610  3734  U   N [qemu-kvm] 134
  8,16   6      714    55.816133848  3735  U   N [qemu-kvm] 134
  8,16  14     3824    55.816456454  3736  U   N [qemu-kvm] 134
  8,16   4     1525    55.820278197  3737  U   N [qemu-kvm] 134
  8,16   4     1526    55.821778871  3738  U   N [qemu-kvm] 134
  8,16   4     1527    55.821966424  3739  U   N [qemu-kvm] 134
  8,16   4     1528    55.822460767  3740  U   N [qemu-kvm] 134
  8,16   4     1529    55.822582532  3741  U   N [qemu-kvm] 134
  8,16   4     1530    55.823182859  3742  U   N [qemu-kvm] 134
  8,17   7      899    55.823232794  3743  A  WS 3407895432 + 480 <- (254,23) 33577480
  8,16   7      900    55.823233549  3743  A  WS 3407897480 + 480 <- (8,17) 3407895432
  8,16   7      901    55.823234161  3743  Q  WS 3407897480 + 480 [qemu-kvm]
  8,16   7      902    55.823239257  3743  S  WS 3407897480 + 480 [qemu-kvm]
  8,17   7      903    55.823890130  3744  A  WS 3407898464 + 480 <- (254,23) 33580512
  8,16   7      904    55.823890790  3744  A  WS 3407900512 + 480 <- (8,17) 3407898464
  8,16   7      905    55.823891069  3744  Q  WS 3407900512 + 480 [qemu-kvm]
  8,16   7      906    55.823895342  3744  S  WS 3407900512 + 480 [qemu-kvm]
  8,17   7      907    55.824300171  3745  A  WS 3407901448 + 152 <- (254,23) 33583496
  8,16   7      908    55.824300596  3745  A  WS 3407903496 + 152 <- (8,17) 3407901448
  8,16   7      909    55.824300817  3745  Q  WS 3407903496 + 152 [qemu-kvm]
  8,16   7      910    55.824302076  3745  S  WS 3407903496 + 152 [qemu-kvm]
  8,16   4     1531    55.824325070  3746  U   N [qemu-kvm] 134
  8,16  19      960    55.824848417  3747  U   N [qemu-kvm] 134
  8,16   2     3006    55.824991838  3748  U   N [qemu-kvm] 134
  8,16   4     1532    55.825498152  3749  U   N [qemu-kvm] 134
  8,16   2     3007    55.825612338  3750  U   N [qemu-kvm] 134
  8,16   4     1533    55.825768633  3751  U   N [qemu-kvm] 134
  8,16   4     1534    55.826446820  3752  U   N [qemu-kvm] 134
  8,17   4     1535    55.826521140  3753  A  WS 3407901608 + 480 <- (254,23) 33583656
  8,16   4     1536    55.826522033  3753  A  WS 3407903656 + 480 <- (8,17) 3407901608
  8,16   4     1537    55.826522763  3753  Q  WS 3407903656 + 480 [qemu-kvm]
  8,16   4     1538    55.826527018  3753  S  WS 3407903656 + 480 [qemu-kvm]
  8,16   4     1539    55.827843895  3754  U   N [qemu-kvm] 134
  8,17   7      911    55.836641685  3755  A  WS 3407903632 + 480 <- (254,23) 33585680
  8,16   7      912    55.836642701  3755  A  WS 3407905680 + 480 <- (8,17) 3407903632
  8,16   7      913    55.836643478  3755  Q  WS 3407905680 + 480 [qemu-kvm]
  8,16   7      914    55.836649708  3755  S  WS 3407905680 + 480 [qemu-kvm]
  8,17   4     1540    55.837357705  3756  A  WS 3407906664 + 480 <- (254,23) 33588712
  8,16   4     1541    55.837358612  3756  A  WS 3407908712 + 480 <- (8,17) 3407906664
  8,16   4     1542    55.837359329  3756  Q  WS 3407908712 + 480 [qemu-kvm]
  8,16   4     1543    55.837363803  3756  S  WS 3407908712 + 480 [qemu-kvm]
  8,17   7      915    55.838540760  3757  A  WS 3407907456 + 192 <- (254,23) 33589504
  8,16   7      916    55.838541575  3757  A  WS 3407909504 + 192 <- (8,17) 3407907456
  8,16   7      917    55.838542282  3757  Q  WS 3407909504 + 192 [qemu-kvm]
  8,16   7      918    55.838548208  3757  S  WS 3407909504 + 192 [qemu-kvm]
  8,17   4     1544    55.850108060  3758  A  WS 3407907840 + 480 <- (254,23) 33589888
  8,16   4     1545    55.850108927  3758  A  WS 3407909888 + 480 <- (8,17) 3407907840
  8,16   4     1546    55.850109476  3758  Q  WS 3407909888 + 480 [qemu-kvm]
  8,16   4     1547    55.850113608  3758  S  WS 3407909888 + 480 [qemu-kvm]
  8,17   4     1548    55.850480286  3759  A  WS 3407908856 + 480 <- (254,23) 33590904
  8,16   4     1549    55.850480658  3759  A  WS 3407910904 + 480 <- (8,17) 3407908856
  8,16   4     1550    55.850480901  3759  Q  WS 3407910904 + 480 [qemu-kvm]
  8,16   4     1551    55.850481992  3759  S  WS 3407910904 + 480 [qemu-kvm]
  8,17   7      919    55.851235411  3760  A  WS 3407910096 + 152 <- (254,23) 33592144
  8,16   7      920    55.851236200  3760  A  WS 3407912144 + 152 <- (8,17) 3407910096
  8,16   7      921    55.851236889  3760  Q  WS 3407912144 + 152 [qemu-kvm]
  8,16   7      922    55.851243432  3760  S  WS 3407912144 + 152 [qemu-kvm]
  8,17   4     1552    55.854625105  3761  A  WS 3407910256 + 480 <- (254,23) 33592304
  8,16   4     1553    55.854625872  3761  A  WS 3407912304 + 480 <- (8,17) 3407910256
  8,16   4     1554    55.854626567  3761  Q  WS 3407912304 + 480 [qemu-kvm]
  8,16   4     1555    55.854630912  3761  S  WS 3407912304 + 480 [qemu-kvm]
  8,17   4     1556    55.863546935  3762  A  WS 3407912280 + 480 <- (254,23) 33594328
  8,16   4     1557    55.863547767  3762  A  WS 3407914328 + 480 <- (8,17) 3407912280
  8,16   4     1558    55.863548414  3762  Q  WS 3407914328 + 480 [qemu-kvm]
  8,16   4     1559    55.863552803  3762  S  WS 3407914328 + 480 [qemu-kvm]
  8,17   4     1560    55.864224593  3763  A  WS 3407915312 + 480 <- (254,23) 33597360
  8,16   4     1561    55.864225104  3763  A  WS 3407917360 + 480 <- (8,17) 3407915312
  8,16   4     1562    55.864225377  3763  Q  WS 3407917360 + 480 [qemu-kvm]
  8,16   4     1563    55.864228934  3763  S  WS 3407917360 + 480 [qemu-kvm]
  8,17   4     1564    55.866627863  3764  A  WS 3407918296 + 152 <- (254,23) 33600344
  8,16   4     1565    55.866628591  3764  A  WS 3407920344 + 152 <- (8,17) 3407918296
  8,16   4     1566    55.866628983  3764  Q  WS 3407920344 + 152 [qemu-kvm]
  8,16   4     1567    55.866633183  3764  S  WS 3407920344 + 152 [qemu-kvm]
  8,17   4     1568    55.877039182  3765  A  WS 3407918456 + 480 <- (254,23) 33600504
  8,16   4     1569    55.877040027  3765  A  WS 3407920504 + 480 <- (8,17) 3407918456
  8,16   4     1570    55.877040609  3765  Q  WS 3407920504 + 480 [qemu-kvm]
  8,16   4     1571    55.877044815  3765  S  WS 3407920504 + 480 [qemu-kvm]
  8,17   4     1572    55.877686947  3766  A  WS 3407920480 + 480 <- (254,23) 33602528
  8,16   4     1573    55.877687518  3766  A  WS 3407922528 + 480 <- (8,17) 3407920480
  8,16   4     1574    55.877687831  3766  Q  WS 3407922528 + 480 [qemu-kvm]
  8,16   4     1575    55.877691496  3766  S  WS 3407922528 + 480 [qemu-kvm]
  8,17   4     1576    55.878521409  3767  A  WS 3407921616 + 480 <- (254,23) 33603664
  8,16   4     1577    55.878522026  3767  A  WS 3407923664 + 480 <- (8,17) 3407921616
  8,16   4     1578    55.878522333  3767  Q  WS 3407923664 + 480 [qemu-kvm]
  8,16   4     1579    55.878526111  3767  S  WS 3407923664 + 480 [qemu-kvm]
  8,17   3     3957    55.886319962   487  A  WS 3491537488 + 480 <- (254,25) 33333456
  8,16   3     3958    55.886320831   487  A  WS 3491539536 + 480 <- (8,17) 3491537488
  8,16   3     3959    55.886328182   487  Q  WS 3491539536 + 480 [kworker/3:1]
  8,16   3     3960    55.886335232   487  S  WS 3491539536 + 480 [kworker/3:1]
  8,17   7      923    55.890487690  3768  A  WS 3407923648 + 480 <- (254,23) 33605696
  8,16   7      924    55.890488716  3768  A  WS 3407925696 + 480 <- (8,17) 3407923648
  8,16   7      925    55.890489350  3768  Q  WS 3407925696 + 480 [qemu-kvm]
  8,16   7      926    55.890494793  3768  S  WS 3407925696 + 480 [qemu-kvm]
  8,16   1        1 1266874889.707163736     0  C   R 3390363000 + 48 [0]
  8,16   1        0 1266874889.707181482     0  m   N cfq3273 complete rqnoidle 0
CPU0 (8,16):
 Reads Queued:        1133,    20864KiB	 Writes Queued:         161,      912KiB
 Read Dispatches:     1131,    20872KiB	 Write Dispatches:      221,     3388KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:     1551,    27852KiB	 Writes Completed:      481,     5332KiB
 Read Merges:            0,        0KiB	 Write Merges:            8,       32KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:          1370        	 Timer unplugs:           0
CPU1 (8,16):
 Reads Queued:        1249,    22376KiB	 Writes Queued:         200,     1364KiB
 Read Dispatches:     1250,    22412KiB	 Write Dispatches:      290,     3080KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:     1760,    31589KiB	 Writes Completed:      545,     5712KiB
 Read Merges:            0,        0KiB	 Write Merges:            8,       32KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:          1527        	 Timer unplugs:           0
CPU2 (8,16):
 Reads Queued:         309,     4792KiB	 Writes Queued:          79,      456KiB
 Read Dispatches:      307,     4580KiB	 Write Dispatches:       76,      424KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:            0,        0KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:           420        	 Timer unplugs:           0
CPU3 (8,16):
 Reads Queued:         354,     5708KiB	 Writes Queued:         160,     4056KiB
 Read Dispatches:      357,     5700KiB	 Write Dispatches:      134,     1056KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:            3,       12KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:           595        	 Timer unplugs:           0
CPU4 (8,16):
 Reads Queued:          82,     1652KiB	 Writes Queued:         136,     3040KiB
 Read Dispatches:       82,     1652KiB	 Write Dispatches:       74,      340KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:           19,       76KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:           243        	 Timer unplugs:           0
CPU5 (8,16):
 Reads Queued:         101,     2633KiB	 Writes Queued:          91,      508KiB
 Read Dispatches:      102,     2637KiB	 Write Dispatches:       86,      388KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:            0,        0KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:           212        	 Timer unplugs:           0
CPU6 (8,16):
 Reads Queued:          23,      364KiB	 Writes Queued:          75,     7432KiB
 Read Dispatches:       22,      348KiB	 Write Dispatches:       32,      140KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:            0,        0KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:           137        	 Timer unplugs:           0
CPU7 (8,16):
 Reads Queued:          38,      492KiB	 Writes Queued:          95,     5840KiB
 Read Dispatches:       38,      492KiB	 Write Dispatches:       42,      448KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:            4,       16KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:           142        	 Timer unplugs:           0
CPU8 (8,16):
 Reads Queued:           3,      148KiB	 Writes Queued:          66,      272KiB
 Read Dispatches:        3,      148KiB	 Write Dispatches:       33,      140KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:            8,       32KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:            79        	 Timer unplugs:           0
CPU9 (8,16):
 Reads Queued:           7,       92KiB	 Writes Queued:          52,     3612KiB
 Read Dispatches:        6,       88KiB	 Write Dispatches:       29,      120KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:            0,        0KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:            64        	 Timer unplugs:           0
CPU10 (8,16):
 Reads Queued:           1,       32KiB	 Writes Queued:         104,      416KiB
 Read Dispatches:        1,       32KiB	 Write Dispatches:       19,      336KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:           65,      260KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:            60        	 Timer unplugs:           0
CPU11 (8,16):
 Reads Queued:           7,       48KiB	 Writes Queued:          16,       68KiB
 Read Dispatches:        7,       48KiB	 Write Dispatches:        8,       36KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:            0,        0KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:            40        	 Timer unplugs:           0
CPU12 (8,16):
 Reads Queued:        1424,    23672KiB	 Writes Queued:         139,     2780KiB
 Read Dispatches:     1415,    23640KiB	 Write Dispatches:      180,     6788KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:     1962,    31868KiB	 Writes Completed:      367,     8440KiB
 Read Merges:            0,        0KiB	 Write Merges:            1,        8KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:          1616        	 Timer unplugs:           0
CPU13 (8,16):
 Reads Queued:        1223,    23770KiB	 Writes Queued:         113,      656KiB
 Read Dispatches:     1230,    23982KiB	 Write Dispatches:      141,     4020KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:     1821,    37370KiB	 Writes Completed:      334,     5256KiB
 Read Merges:            0,        0KiB	 Write Merges:            0,        0KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:          1390        	 Timer unplugs:           0
CPU14 (8,16):
 Reads Queued:         400,     6044KiB	 Writes Queued:          96,     2032KiB
 Read Dispatches:      403,     6048KiB	 Write Dispatches:       76,      580KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:            2,      152KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:           517        	 Timer unplugs:           0
CPU15 (8,16):
 Reads Queued:         404,     9400KiB	 Writes Queued:          96,     2788KiB
 Read Dispatches:      405,     9420KiB	 Write Dispatches:       82,      636KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:            0,        0KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:           513        	 Timer unplugs:           0
CPU16 (8,16):
 Reads Queued:          96,     1568KiB	 Writes Queued:          59,      316KiB
 Read Dispatches:       96,     1568KiB	 Write Dispatches:       56,     1288KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:            0,        0KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:           170        	 Timer unplugs:           0
CPU17 (8,16):
 Reads Queued:         135,     2908KiB	 Writes Queued:          50,      212KiB
 Read Dispatches:      135,     2908KiB	 Write Dispatches:       39,      168KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:            0,        0KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:           202        	 Timer unplugs:           0
CPU18 (8,16):
 Reads Queued:          28,      248KiB	 Writes Queued:          51,     1784KiB
 Read Dispatches:       29,      252KiB	 Write Dispatches:       39,      200KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:            0,        0KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:            86        	 Timer unplugs:           0
CPU19 (8,16):
 Reads Queued:          43,      804KiB	 Writes Queued:          97,     9788KiB
 Read Dispatches:       41,      788KiB	 Write Dispatches:       32,      916KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:            0,        0KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:           138        	 Timer unplugs:           0
CPU20 (8,16):
 Reads Queued:          10,      192KiB	 Writes Queued:          29,     2128KiB
 Read Dispatches:       10,      192KiB	 Write Dispatches:        9,       44KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:            0,        0KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:            39        	 Timer unplugs:           0
CPU21 (8,16):
 Reads Queued:          12,      300KiB	 Writes Queued:          35,     2168KiB
 Read Dispatches:       12,      300KiB	 Write Dispatches:       21,      124KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:            0,        0KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:            50        	 Timer unplugs:           0
CPU22 (8,16):
 Reads Queued:           3,       16KiB	 Writes Queued:           7,      440KiB
 Read Dispatches:        3,       16KiB	 Write Dispatches:        2,       44KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:            0,        0KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:            14        	 Timer unplugs:           0
CPU23 (8,16):
 Reads Queued:           1,        4KiB	 Writes Queued:           7,       40KiB
 Read Dispatches:        1,        4KiB	 Write Dispatches:        6,       36KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:        0,        0KiB	 Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB	 Write Merges:            0,        0KiB
 Read depth:            13        	 Write depth:            17
 IO unplugs:            15        	 Timer unplugs:           0

Total (8,16):
 Reads Queued:        7086,   128127KiB	 Writes Queued:        2014,    53108KiB
 Read Dispatches:     7086,   128127KiB	 Write Dispatches:     1727,    24740KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:     7094,   128679KiB	 Writes Completed:     1727,    24740KiB
 Read Merges:            0,        0KiB	 Write Merges:          118,      620KiB
 IO unplugs:          9639        	 Timer unplugs:           0

Throughput (R/W): 0KiB/s / 0KiB/s
Events (8,16): 193565 entries
Skips: 21833 forward (8033027806 - 100.0%)


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]