[virt-tools-list] LUKS decryption slow in KVM

[Cole Robinson]

On 01/13/2014 05:26 AM, Digimer wrote:
> On 13/01/14 04:49 AM, Digimer wrote:
>> Hi all,
>>
>>    I'm trying to sort out a significant performance hit I am getting
>> with a LUKS partition inside a KVM VM (host is RHEL 6.5, guest is also
>> RHEL 6.5).
>>
>>    Inside the VM, I get ~1.1 GB/sec write speed:
>>
>> ====
>> Inside the VM! (48 GB, 4x vCPUs, 500MB /boot, 40 GB /, 4 GB <swap>, RHEL
>> 6 minimal, no selinux/iptables)
>> Write to raw partition using: sync; dd if=/dev/zero of=/dev/vda5 bs=4M
>> count=80000; sync
>> - 335544320000 bytes (336 GB) copied, 302.885 s, 1.1 GB/s
>> - 335544320000 bytes (336 GB) copied, 293.905 s, 1.1 GB/s
>> - 335544320000 bytes (336 GB) copied, 289.298 s, 1.2 GB/s
>>
>> Write to ext4 partition using: sync; dd if=/dev/zero
>> of=/mnt/data/zeros.out bs=4M count=80000; sync
>> - 335544320000 bytes (336 GB) copied, 290.855 s, 1.2 GB/s
>> - 335544320000 bytes (336 GB) copied, 347.901 s, 964 MB/s
>> - 335544320000 bytes (336 GB) copied, 317.877 s, 1.1 GB/s
>> ====
>>
>>    When I created the LUKS partition though, I got substantially lower
>> speed:
>>
>> ====
>> [root at vm01-ng-rhel01 ~]# cryptsetup -v status vda5
>> /dev/mapper/vda5 is active.
>>    type:  LUKS1
>>    cipher:  aes-cbc-essiv:sha256
>>    keysize: 256 bits
>>    device:  /dev/vda5
>>    offset:  4096 sectors
>>    size:    1729118208 sectors
>>    mode:    read/write
>> Command successful.
>>
>> (killed the 'dd' when I realized how slow it was going)
>>
>> sync; dd if=/dev/zero of=/dev/mapper/vda5 bs=4M count=80000; sync
>> 48330964992 bytes (48 GB) copied, 345.72 s, 140 MB/s
>> ====
>>
>> I noticed that the real CPU on the host:
>>
>> ====
>> vendor_id    : GenuineIntel
>> cpu family    : 6
>> model        : 62
>> model name    : Intel(R) Xeon(R) CPU E5-2637 v2 @ 3.50GHz
>> stepping    : 4
>> cpu MHz        : 3500.087
>> cache size    : 15360 KB
>> physical id    : 1
>> siblings    : 8
>> core id        : 4
>> cpu cores    : 4
>> apicid        : 41
>> initial apicid    : 41
>> fpu        : yes
>> fpu_exception    : yes
>> cpuid level    : 13
>> wp        : yes
>> flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
>> cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall
>> nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
>> xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx
>> smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt
>> tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb
>> xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid fsgsbase smep
>> erms
>> bogomips    : 6998.99
>> clflush size    : 64
>> cache_alignment    : 64
>> address sizes    : 46 bits physical, 48 bits virtual
>> power management:
>> ====
>>
>> Has a lot more features (I see 'aes' in particular) than inside the VM:
>>
>> ====
>> processor    : 3
>> vendor_id    : GenuineIntel
>> cpu family    : 6
>> model        : 13
>> model name    : QEMU Virtual CPU version (cpu64-rhel6)
>> stepping    : 3
>> cpu MHz        : 3499.998
>> cache size    : 4096 KB
>> fpu        : yes
>> fpu_exception    : yes
>> cpuid level    : 4
>> wp        : yes
>> flags        : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
>> pse36 clflush mmx fxsr sse sse2 syscall nx lm unfair_spinlock pni cx16
>> hypervisor lahf_lm
>> bogomips    : 6999.99
>> clflush size    : 64
>> cache_alignment    : 64
>> address sizes    : 46 bits physical, 48 bits virtual
>> power management:
>> ====
>>
>> Should I (can I?) be passing some of these features up to the guest to
>> improve LUKS performance?
>>
>> Thanks!
> 
> I copied the extensions of the host, and 'aes' showed in the guest's cpuinfo,
> to the VM and got somewhat better numbers;
> 
> Write to raw LUKS partition using: sync; dd if=/dev/zero of=/dev/mapper/vda5
> bs=4M count=80000; sync
> - 335544320000 bytes (336 GB) copied, 1157.27 s, 290 MB/s
> 
> Still a far cry from what the base system can do.
> 

For all questions like this, please start with sudo virsh dumpxml $vmname, and
/var/log/libvirt/qemu/$vmname.log

- Cole