[libvirt] Qemu/KVM guest boots 2x slower with vhost_net
Reeted
reeted at shiftmail.org
Sun Oct 9 21:47:26 UTC 2011
On 10/05/11 01:12, Reeted wrote:
> .....
> I found that virtual machines in my host booted 2x slower ... to the
> vhost_net presence
> ...
Just a small update,
Firstly: I cannot reproduce any slowness after boot by doing:
# time /etc/init.d/chrony restart
Restarting time daemon: Starting /usr/sbin/chronyd...
chronyd is running and online.
real 0m3.022s
user 0m0.000s
sys 0m0.000s
since this is a network service I expected it to show the problem, but
it doesn't. It takes exactly same time with and without vhost_net.
Secondly, vhost_net appears to work correctly, because I have performed
a NPtcp performance test between two guests in the same host, and these
are the results:
vhost_net deactivated for both hosts:
NPtcp -h 192.168.7.81
Send and receive buffers are 16384 and 87380 bytes
(A bug in Linux doubles the requested buffer sizes)
Now starting the main loop
0: 1 bytes 917 times --> 0.08 Mbps in 92.07 usec
1: 2 bytes 1086 times --> 0.18 Mbps in 86.04 usec
2: 3 bytes 1162 times --> 0.27 Mbps in 85.08 usec
3: 4 bytes 783 times --> 0.36 Mbps in 85.34 usec
4: 6 bytes 878 times --> 0.54 Mbps in 85.42 usec
5: 8 bytes 585 times --> 0.72 Mbps in 85.31 usec
6: 12 bytes 732 times --> 1.07 Mbps in 85.52 usec
7: 13 bytes 487 times --> 1.16 Mbps in 85.52 usec
8: 16 bytes 539 times --> 1.43 Mbps in 85.26 usec
9: 19 bytes 659 times --> 1.70 Mbps in 85.43 usec
10: 21 bytes 739 times --> 1.77 Mbps in 90.71 usec
11: 24 bytes 734 times --> 2.13 Mbps in 86.13 usec
12: 27 bytes 822 times --> 2.22 Mbps in 92.80 usec
13: 29 bytes 478 times --> 2.35 Mbps in 94.02 usec
14: 32 bytes 513 times --> 2.60 Mbps in 93.75 usec
15: 35 bytes 566 times --> 3.15 Mbps in 84.77 usec
16: 45 bytes 674 times --> 4.01 Mbps in 85.56 usec
17: 48 bytes 779 times --> 4.32 Mbps in 84.70 usec
18: 51 bytes 811 times --> 4.61 Mbps in 84.32 usec
19: 61 bytes 465 times --> 5.08 Mbps in 91.57 usec
20: 64 bytes 537 times --> 5.22 Mbps in 93.46 usec
21: 67 bytes 551 times --> 5.73 Mbps in 89.20 usec
22: 93 bytes 602 times --> 8.28 Mbps in 85.73 usec
23: 96 bytes 777 times --> 8.45 Mbps in 86.70 usec
24: 99 bytes 780 times --> 8.71 Mbps in 86.72 usec
25: 125 bytes 419 times --> 11.06 Mbps in 86.25 usec
26: 128 bytes 575 times --> 11.38 Mbps in 85.80 usec
27: 131 bytes 591 times --> 11.60 Mbps in 86.17 usec
28: 189 bytes 602 times --> 16.55 Mbps in 87.14 usec
29: 192 bytes 765 times --> 16.80 Mbps in 87.19 usec
30: 195 bytes 770 times --> 17.11 Mbps in 86.94 usec
31: 253 bytes 401 times --> 22.04 Mbps in 87.59 usec
32: 256 bytes 568 times --> 22.64 Mbps in 86.25 usec
33: 259 bytes 584 times --> 22.68 Mbps in 87.12 usec
34: 381 bytes 585 times --> 33.19 Mbps in 87.58 usec
35: 384 bytes 761 times --> 33.54 Mbps in 87.36 usec
36: 387 bytes 766 times --> 33.91 Mbps in 87.08 usec
37: 509 bytes 391 times --> 44.23 Mbps in 87.80 usec
38: 512 bytes 568 times --> 44.70 Mbps in 87.39 usec
39: 515 bytes 574 times --> 45.21 Mbps in 86.90 usec
40: 765 bytes 580 times --> 66.05 Mbps in 88.36 usec
41: 768 bytes 754 times --> 66.73 Mbps in 87.81 usec
42: 771 bytes 760 times --> 67.02 Mbps in 87.77 usec
43: 1021 bytes 384 times --> 88.04 Mbps in 88.48 usec
44: 1024 bytes 564 times --> 88.30 Mbps in 88.48 usec
45: 1027 bytes 566 times --> 88.63 Mbps in 88.40 usec
46: 1533 bytes 568 times --> 71.75 Mbps in 163.00 usec
47: 1536 bytes 408 times --> 72.11 Mbps in 162.51 usec
48: 1539 bytes 410 times --> 71.71 Mbps in 163.75 usec
49: 2045 bytes 204 times --> 95.40 Mbps in 163.55 usec
50: 2048 bytes 305 times --> 95.26 Mbps in 164.02 usec
51: 2051 bytes 305 times --> 95.33 Mbps in 164.14 usec
52: 3069 bytes 305 times --> 141.16 Mbps in 165.87 usec
53: 3072 bytes 401 times --> 142.19 Mbps in 164.83 usec
54: 3075 bytes 404 times --> 150.68 Mbps in 155.70 usec
55: 4093 bytes 214 times --> 192.36 Mbps in 162.33 usec
56: 4096 bytes 307 times --> 193.21 Mbps in 161.74 usec
57: 4099 bytes 309 times --> 213.24 Mbps in 146.66 usec
58: 6141 bytes 341 times --> 330.80 Mbps in 141.63 usec
59: 6144 bytes 470 times --> 328.09 Mbps in 142.87 usec
60: 6147 bytes 466 times --> 330.53 Mbps in 141.89 usec
61: 8189 bytes 235 times --> 437.29 Mbps in 142.87 usec
62: 8192 bytes 349 times --> 436.23 Mbps in 143.27 usec
63: 8195 bytes 349 times --> 436.99 Mbps in 143.08 usec
64: 12285 bytes 349 times --> 625.88 Mbps in 149.75 usec
65: 12288 bytes 445 times --> 626.27 Mbps in 149.70 usec
66: 12291 bytes 445 times --> 626.15 Mbps in 149.76 usec
67: 16381 bytes 222 times --> 793.58 Mbps in 157.48 usec
68: 16384 bytes 317 times --> 806.90 Mbps in 154.91 usec
69: 16387 bytes 322 times --> 796.81 Mbps in 156.90 usec
70: 24573 bytes 318 times --> 1127.58 Mbps in 166.26 usec
71: 24576 bytes 400 times --> 1125.20 Mbps in 166.64 usec
72: 24579 bytes 400 times --> 1124.84 Mbps in 166.71 usec
73: 32765 bytes 200 times --> 1383.86 Mbps in 180.64 usec
74: 32768 bytes 276 times --> 1376.05 Mbps in 181.68 usec
75: 32771 bytes 275 times --> 1377.47 Mbps in 181.51 usec
76: 49149 bytes 275 times --> 1824.90 Mbps in 205.48 usec
77: 49152 bytes 324 times --> 1813.95 Mbps in 206.73 usec
78: 49155 bytes 322 times --> 1765.68 Mbps in 212.40 usec
79: 65533 bytes 156 times --> 2193.44 Mbps in 227.94 usec
80: 65536 bytes 219 times --> 2186.79 Mbps in 228.65 usec
81: 65539 bytes 218 times --> 2186.98 Mbps in 228.64 usec
82: 98301 bytes 218 times --> 2831.01 Mbps in 264.92 usec
83: 98304 bytes 251 times --> 2804.76 Mbps in 267.40 usec
84: 98307 bytes 249 times --> 2824.62 Mbps in 265.53 usec
85: 131069 bytes 125 times --> 3106.48 Mbps in 321.90 usec
86: 131072 bytes 155 times --> 3033.71 Mbps in 329.63 usec
87: 131075 bytes 151 times --> 3044.89 Mbps in 328.43 usec
88: 196605 bytes 152 times --> 4196.94 Mbps in 357.40 usec
89: 196608 bytes 186 times --> 4358.25 Mbps in 344.17 usec
90: 196611 bytes 193 times --> 4362.34 Mbps in 343.86 usec
91: 262141 bytes 96 times --> 4654.49 Mbps in 429.69 usec
92: 262144 bytes 116 times --> 4727.16 Mbps in 423.09 usec
93: 262147 bytes 118 times --> 4697.22 Mbps in 425.79 usec
94: 393213 bytes 117 times --> 5452.51 Mbps in 550.20 usec
95: 393216 bytes 121 times --> 5360.27 Mbps in 559.67 usec
96: 393219 bytes 119 times --> 5358.03 Mbps in 559.91 usec
97: 524285 bytes 59 times --> 5053.83 Mbps in 791.47 usec
98: 524288 bytes 63 times --> 5033.86 Mbps in 794.62 usec
99: 524291 bytes 62 times --> 5691.44 Mbps in 702.81 usec
100: 786429 bytes 71 times --> 5750.68 Mbps in 1043.35 usec
101: 786432 bytes 63 times --> 5809.21 Mbps in 1032.84 usec
102: 786435 bytes 64 times --> 5864.45 Mbps in 1023.12 usec
103: 1048573 bytes 32 times --> 5755.24 Mbps in 1390.03 usec
104: 1048576 bytes 35 times --> 6001.51 Mbps in 1333.00 usec
105: 1048579 bytes 37 times --> 6099.40 Mbps in 1311.61 usec
106: 1572861 bytes 38 times --> 6061.69 Mbps in 1979.64 usec
107: 1572864 bytes 33 times --> 6144.15 Mbps in 1953.08 usec
108: 1572867 bytes 34 times --> 6108.20 Mbps in 1964.58 usec
109: 2097149 bytes 16 times --> 6128.72 Mbps in 2610.65 usec
110: 2097152 bytes 19 times --> 6271.35 Mbps in 2551.29 usec
111: 2097155 bytes 19 times --> 6273.55 Mbps in 2550.39 usec
112: 3145725 bytes 19 times --> 6146.28 Mbps in 3904.79 usec
113: 3145728 bytes 17 times --> 6288.29 Mbps in 3816.62 usec
114: 3145731 bytes 17 times --> 6234.73 Mbps in 3849.41 usec
115: 4194301 bytes 8 times --> 5852.76 Mbps in 5467.50 usec
116: 4194304 bytes 9 times --> 5886.74 Mbps in 5435.94 usec
117: 4194307 bytes 9 times --> 5887.35 Mbps in 5435.39 usec
118: 6291453 bytes 9 times --> 4502.11 Mbps in 10661.67 usec
119: 6291456 bytes 6 times --> 4541.26 Mbps in 10569.75 usec
120: 6291459 bytes 6 times --> 4465.98 Mbps in 10747.93 usec
121: 8388605 bytes 3 times --> 4601.84 Mbps in 13907.47 usec
122: 8388608 bytes 3 times --> 4590.50 Mbps in 13941.84 usec
123: 8388611 bytes 3 times --> 4195.17 Mbps in 15255.65 usec
vhost_net activated for both hosts:
NPtcp -h 192.168.7.81
Send and receive buffers are 16384 and 87380 bytes
(A bug in Linux doubles the requested buffer sizes)
Now starting the main loop
0: 1 bytes 1013 times --> 0.10 Mbps in 75.89 usec
1: 2 bytes 1317 times --> 0.21 Mbps in 74.03 usec
2: 3 bytes 1350 times --> 0.30 Mbps in 76.90 usec
3: 4 bytes 866 times --> 0.43 Mbps in 71.27 usec
4: 6 bytes 1052 times --> 0.60 Mbps in 76.02 usec
5: 8 bytes 657 times --> 0.79 Mbps in 76.88 usec
6: 12 bytes 812 times --> 1.24 Mbps in 73.72 usec
7: 13 bytes 565 times --> 1.40 Mbps in 70.60 usec
8: 16 bytes 653 times --> 1.58 Mbps in 77.05 usec
9: 19 bytes 730 times --> 1.90 Mbps in 76.25 usec
10: 21 bytes 828 times --> 1.98 Mbps in 80.85 usec
11: 24 bytes 824 times --> 2.47 Mbps in 74.22 usec
12: 27 bytes 954 times --> 2.73 Mbps in 75.45 usec
13: 29 bytes 589 times --> 3.06 Mbps in 72.23 usec
14: 32 bytes 668 times --> 3.26 Mbps in 74.84 usec
15: 35 bytes 709 times --> 3.46 Mbps in 77.09 usec
16: 45 bytes 741 times --> 4.50 Mbps in 76.35 usec
17: 48 bytes 873 times --> 4.83 Mbps in 75.90 usec
18: 51 bytes 905 times --> 5.50 Mbps in 70.72 usec
19: 61 bytes 554 times --> 6.36 Mbps in 73.14 usec
20: 64 bytes 672 times --> 6.28 Mbps in 77.77 usec
21: 67 bytes 663 times --> 6.39 Mbps in 80.06 usec
22: 93 bytes 671 times --> 9.44 Mbps in 75.15 usec
23: 96 bytes 887 times --> 9.52 Mbps in 76.90 usec
24: 99 bytes 880 times --> 10.55 Mbps in 71.57 usec
25: 125 bytes 508 times --> 12.63 Mbps in 75.49 usec
26: 128 bytes 657 times --> 12.30 Mbps in 79.38 usec
27: 131 bytes 639 times --> 12.72 Mbps in 78.57 usec
28: 189 bytes 660 times --> 18.36 Mbps in 78.55 usec
29: 192 bytes 848 times --> 18.84 Mbps in 77.75 usec
30: 195 bytes 864 times --> 18.91 Mbps in 78.69 usec
31: 253 bytes 443 times --> 24.04 Mbps in 80.28 usec
32: 256 bytes 620 times --> 26.61 Mbps in 73.40 usec
33: 259 bytes 686 times --> 26.09 Mbps in 75.75 usec
34: 381 bytes 672 times --> 40.04 Mbps in 72.59 usec
35: 384 bytes 918 times --> 39.67 Mbps in 73.86 usec
36: 387 bytes 906 times --> 40.68 Mbps in 72.58 usec
37: 509 bytes 469 times --> 51.70 Mbps in 75.11 usec
38: 512 bytes 664 times --> 51.55 Mbps in 75.77 usec
39: 515 bytes 662 times --> 49.61 Mbps in 79.19 usec
40: 765 bytes 637 times --> 75.91 Mbps in 76.89 usec
41: 768 bytes 867 times --> 76.03 Mbps in 77.07 usec
42: 771 bytes 866 times --> 76.21 Mbps in 77.19 usec
43: 1021 bytes 436 times --> 99.46 Mbps in 78.32 usec
44: 1024 bytes 637 times --> 100.04 Mbps in 78.10 usec
45: 1027 bytes 641 times --> 100.06 Mbps in 78.31 usec
46: 1533 bytes 641 times --> 113.15 Mbps in 103.36 usec
47: 1536 bytes 644 times --> 127.72 Mbps in 91.75 usec
48: 1539 bytes 727 times --> 102.87 Mbps in 114.14 usec
49: 2045 bytes 293 times --> 177.68 Mbps in 87.81 usec
50: 2048 bytes 569 times --> 103.58 Mbps in 150.85 usec
51: 2051 bytes 331 times --> 107.53 Mbps in 145.52 usec
52: 3069 bytes 344 times --> 204.05 Mbps in 114.75 usec
53: 3072 bytes 580 times --> 207.53 Mbps in 112.93 usec
54: 3075 bytes 590 times --> 211.37 Mbps in 110.99 usec
55: 4093 bytes 301 times --> 285.23 Mbps in 109.48 usec
56: 4096 bytes 456 times --> 317.27 Mbps in 98.50 usec
57: 4099 bytes 507 times --> 332.92 Mbps in 93.93 usec
58: 6141 bytes 532 times --> 462.96 Mbps in 101.20 usec
59: 6144 bytes 658 times --> 451.75 Mbps in 103.76 usec
60: 6147 bytes 642 times --> 478.19 Mbps in 98.07 usec
61: 8189 bytes 340 times --> 743.81 Mbps in 84.00 usec
62: 8192 bytes 595 times --> 695.89 Mbps in 89.81 usec
63: 8195 bytes 556 times --> 702.95 Mbps in 88.94 usec
64: 12285 bytes 562 times --> 945.94 Mbps in 99.08 usec
65: 12288 bytes 672 times --> 870.86 Mbps in 107.65 usec
66: 12291 bytes 619 times --> 954.94 Mbps in 98.20 usec
67: 16381 bytes 339 times --> 1003.02 Mbps in 124.60 usec
68: 16384 bytes 401 times --> 652.84 Mbps in 191.47 usec
69: 16387 bytes 261 times --> 872.02 Mbps in 143.37 usec
70: 24573 bytes 348 times --> 1105.61 Mbps in 169.57 usec
71: 24576 bytes 393 times --> 1037.52 Mbps in 180.72 usec
72: 24579 bytes 368 times --> 1066.39 Mbps in 175.85 usec
73: 32765 bytes 189 times --> 1271.24 Mbps in 196.64 usec
74: 32768 bytes 254 times --> 1253.73 Mbps in 199.41 usec
75: 32771 bytes 250 times --> 1101.71 Mbps in 226.94 usec
76: 49149 bytes 220 times --> 1704.99 Mbps in 219.93 usec
77: 49152 bytes 303 times --> 1678.17 Mbps in 223.46 usec
78: 49155 bytes 298 times --> 1648.32 Mbps in 227.52 usec
79: 65533 bytes 146 times --> 1940.36 Mbps in 257.67 usec
80: 65536 bytes 194 times --> 1785.37 Mbps in 280.05 usec
81: 65539 bytes 178 times --> 2079.85 Mbps in 240.41 usec
82: 98301 bytes 207 times --> 2840.36 Mbps in 264.04 usec
83: 98304 bytes 252 times --> 3441.30 Mbps in 217.94 usec
84: 98307 bytes 305 times --> 3575.33 Mbps in 209.78 usec
85: 131069 bytes 158 times --> 3145.83 Mbps in 317.87 usec
86: 131072 bytes 157 times --> 3283.65 Mbps in 304.54 usec
87: 131075 bytes 164 times --> 3610.07 Mbps in 277.01 usec
88: 196605 bytes 180 times --> 4921.05 Mbps in 304.81 usec
89: 196608 bytes 218 times --> 4953.98 Mbps in 302.79 usec
90: 196611 bytes 220 times --> 4841.76 Mbps in 309.81 usec
91: 262141 bytes 107 times --> 4546.37 Mbps in 439.91 usec
92: 262144 bytes 113 times --> 4730.30 Mbps in 422.81 usec
93: 262147 bytes 118 times --> 5211.50 Mbps in 383.77 usec
94: 393213 bytes 130 times --> 7191.67 Mbps in 417.15 usec
95: 393216 bytes 159 times --> 7423.89 Mbps in 404.10 usec
96: 393219 bytes 164 times --> 7321.70 Mbps in 409.74 usec
97: 524285 bytes 81 times --> 7631.75 Mbps in 524.12 usec
98: 524288 bytes 95 times --> 7287.79 Mbps in 548.86 usec
99: 524291 bytes 91 times --> 7253.28 Mbps in 551.48 usec
100: 786429 bytes 90 times --> 8451.33 Mbps in 709.94 usec
101: 786432 bytes 93 times --> 8755.43 Mbps in 685.29 usec
102: 786435 bytes 97 times --> 8740.15 Mbps in 686.49 usec
103: 1048573 bytes 48 times --> 9220.97 Mbps in 867.59 usec
104: 1048576 bytes 57 times --> 8512.15 Mbps in 939.83 usec
105: 1048579 bytes 53 times --> 8556.70 Mbps in 934.94 usec
106: 1572861 bytes 53 times --> 9566.40 Mbps in 1254.39 usec
107: 1572864 bytes 53 times --> 10165.18 Mbps in 1180.50 usec
108: 1572867 bytes 56 times --> 11420.63 Mbps in 1050.73 usec
109: 2097149 bytes 31 times --> 11295.29 Mbps in 1416.52 usec
110: 2097152 bytes 35 times --> 11869.30 Mbps in 1348.02 usec
111: 2097155 bytes 37 times --> 11407.22 Mbps in 1402.62 usec
112: 3145725 bytes 35 times --> 12821.47 Mbps in 1871.86 usec
113: 3145728 bytes 35 times --> 11727.57 Mbps in 2046.46 usec
114: 3145731 bytes 32 times --> 12803.10 Mbps in 1874.55 usec
115: 4194301 bytes 17 times --> 10009.28 Mbps in 3197.03 usec
116: 4194304 bytes 15 times --> 10283.54 Mbps in 3111.77 usec
117: 4194307 bytes 16 times --> 10923.95 Mbps in 2929.34 usec
118: 6291453 bytes 17 times --> 11959.10 Mbps in 4013.68 usec
119: 6291456 bytes 16 times --> 10674.76 Mbps in 4496.59 usec
120: 6291459 bytes 14 times --> 10868.07 Mbps in 4416.61 usec
121: 8388605 bytes 7 times --> 9456.16 Mbps in 6768.07 usec
122: 8388608 bytes 7 times --> 9303.58 Mbps in 6879.07 usec
123: 8388611 bytes 7 times --> 10048.79 Mbps in 6368.93 usec
so the case vhost_net is indeed faster in NPtcp by a factor almost 2,
but that's only visible for the very high speeds and buffer sizes. Also
note that the speed in the vhost_net case is much more variable, even if
it's higher on average.
I expected more difference...
Actually I expected no-vhost to be slower than what it is.
Kudos to the developers.
If you have any more ideas for the slower boot please tell. I am of
course not worried about waiting 10-30 more seconds at boot time, I am
worried that if there is some factor 2x or 3x slowness somewhere, that
can bite me in production without me even realizing.
And after having seen the above no-vhost_net tcp benchmarks I guess I
don't really need vhost_net active for these VMs during production, so I
will just disable vhost_net to be on the safe side until I can track
down the boot-time slowness somehow.
Thanks for your help
R.
More information about the libvir-list
mailing list