[libvirt] Stored secrets seem to get corrupted

[Wido den Hollander]

Hi,

On one of my systems I'm having troubles with my RBD storage backend.

At first I thought it was a problem with my code, but after trying the 
same code on a second machine I'm a bit confused.

The problem is that the storage backend tries to retrieve the value of a 
secret and base64 decode it, that fails.

My debug log shows:

debug : virStorageBackendRBDOpenRADOSConn:65 : Using cephx authorization
debug : virStorageBackendRBDOpenRADOSConn:75 : Looking up secret by 
UUID: 322bccea-f2ed-4eae-a7e5-d0793ffb162d
debug : virSecretLookupByUUIDString:14128 : conn=0x7fac9c0009c0, 
uuidstr=322bccea-f2ed-4eae-a7e5-d0793ffb162d
debug : virSecretLookupByUUID:14086 : conn=0x7fac9c0009c0, 
uuid=322bccea-f2ed-4eae-a7e5-d0793ffb162d
debug : virSecretGetValue:14486 : secret=0x7fac94000d30, 
value_size=0x7facad481918, flags=0

debug : virStorageBackendRBDOpenRADOSConn:103 : Found cephx key:  `I^%

debug : virStorageBackendRBDOpenRADOSConn:135 : Found 1 RADOS cluster 
monitors in the pool configuration
debug : virStorageBackendRBDOpenRADOSConn:159 : RADOS mon_host has been 
set to: 31.25.100.131:6789,
error : virStorageBackendRBDOpenRADOSConn:171 : internal error failed to 
connect to the RADOS monitor on: 31.25.100.131:6789,

It goes wrong at "Found cephx key: <garbage>"

So I figured it had to be something in my code and I went over the code 
again, but nothing seemed odd.

I tried the same checkout (tag: 0.9.13-rc1) on a different host (also 
Ubuntu 12.04) and that worked.

The secret and pool XML's are the same, but what I found is that the 
secret storage on disk seems to go wrong on one machine.

Notice this behavior:

root at stack01:~# virsh secret-set-value 
322bccea-f2ed-4eae-a7e5-d0793ffb162d 
AQAE+uJPCFpELBAAkTniQvHabBGj0Quwnu2imA==
Secret value set

root at stack01:~# md5sum 
/etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
b4b147bc522828731f1a016bfa72c073 
/etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
root at stack01:~# virsh secret-set-value 
322bccea-f2ed-4eae-a7e5-d0793ffb162d 
AQAE+uJPCFpELBAAkTniQvHabBGj0Quwnu2imA==
Secret value set

root at stack01:~# md5sum 
/etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
927e2458c32cc3f6754d91694e41333f 
/etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
root at stack01:~#

As you can see, the md5sum of the file changes when I set the value of 
the secret to the same.

I tried the same on the other host:

root at stack02:~# virsh secret-set-value 
322bccea-f2ed-4eae-a7e5-d0793ffb162d 
AQAE+uJPCFpELBAAkTniQvHabBGj0Quwnu2imA==
Secret value set

root at stack02:~# md5sum 
/etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
c30db27f9ebfe3f7903470d4bd542d1d 
/etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
root at stack02:~# virsh secret-set-value 
322bccea-f2ed-4eae-a7e5-d0793ffb162d 
AQAE+uJPCFpELBAAkTniQvHabBGj0Quwnu2imA==
Secret value set

root at stack02:~# md5sum 
/etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
c30db27f9ebfe3f7903470d4bd542d1d 
/etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
root at stack02:~# virsh secret-set-value 
322bccea-f2ed-4eae-a7e5-d0793ffb162d 
AQAE+uJPCFpELBAAkTniQvHabBGj0Quwnu2imA==
Secret value set

root at stack02:~# md5sum 
/etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
c30db27f9ebfe3f7903470d4bd542d1d 
/etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
root at stack02:~#

The md5sum stays the same on stack02.

I verified that stack01 isn't out of disk space or out of inodes, those 
are in the acceptable values range.

Any suggestions?

For the record, the hosts:

OS: Ubuntu 12.04 x864_64
Kernel: 3.2.0-25-generic
Libvirt commit: 0fce94fe1bd782ac4c33fdd59d13ee37b3437413

Thank you,

Wido