[libvirt] [PATCH v4 00/23] Introduce metadata locking

Wed Sep 19 09:17:39 UTC 2018

Bjoern Walk <bwalk at linux.ibm.com> [2018-09-12, 01:17PM +0200]:
> Michal Privoznik <mprivozn at redhat.com> [2018-09-12, 11:32AM +0200]:
> > On 09/12/2018 07:19 AM, Bjoern Walk wrote:
> > > Michal Privoznik <mprivozn at redhat.com> [2018-09-10, 11:36AM +0200]:
> > >> Technically, this is v4 of:
> > >>
> > >> https://www.redhat.com/archives/libvir-list/2018-August/msg01627.html
> > >>
> > >> However, this is implementing different approach than any of the
> > >> previous versions.
> > >>
> > >> One of the problems with previous version was that it was too
> > >> complicated. The main reason for that was that we could not close the
> > >> connection whilst there was a file locked. So we had to invent a
> > >> mechanism that would prevent that (on the client side).
> > >>
> > >> These patches implement different approach. They rely on secdriver's
> > >> transactions which bring all the paths we want to label into one place
> > >> so that they can be relabelled within different namespace.
> > >> I'm extending this idea so that transactions run all the time
> > >> (regardless of domain namespacing) and only at the very last moment is
> > >> decided which namespace would the relabeling run in.
> > >>
> > >> Metadata locking is then as easy as putting lock/unlock calls around one
> > >> function.
> > >>
> > >> You can find the patches at my github too:
> > >>
> > >> https://github.com/zippy2/libvirt/tree/disk_metadata_lock_v4_alt
> > > 
> > > Hey Michal,
> > > 
> > > is was running a quick test with this patch series with two domains
> > > sharing a disk image without <shareable/> and SELinux enabled. When
> > > starting the second domain, the whole libvirtd daemon hangs for almost a
> > > minute until giving the error that the image is locked. I haven't
> > > debugged it yet to figure out what happens.
> > 
> > Is this on two different hosts or one?
> 
> On the same host.
> 
> > I'm unable to reproduce, so can you please attach debugger and share 't
> > a a bt' output with me?
> 
> (gdb) thread apply all bt
> 
> Thread 17 (Thread 0x3ff48dff910 (LWP 193353)):
> #0  0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
> #2  0x000003ff6678c5a8 in udevEventHandleThread (opaque=<optimized out>) at node_device/node_device_udev.c:1603
> #3  0x000003ff8c6bad16 in virThreadHelper () from /lib64/libvirt.so.0
> #4  0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
> #5  0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
> 
> Thread 16 (Thread 0x3ff4acfe910 (LWP 193312)):
> #0  0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
> #2  0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
> #3  0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
> #4  0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
> #5  0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
> 
> Thread 15 (Thread 0x3ff4b4ff910 (LWP 193311)):
> #0  0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
> #2  0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
> #3  0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
> #4  0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
> #5  0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
> 
> Thread 14 (Thread 0x3ff64efd910 (LWP 193310)):
> #0  0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
> #2  0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
> #3  0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
> #4  0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
> #5  0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
> 
> Thread 13 (Thread 0x3ff656fe910 (LWP 193309)):
> #0  0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
> #2  0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
> #3  0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
> #4  0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
> #5  0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
> 
> Thread 12 (Thread 0x3ff65eff910 (LWP 193308)):
> #0  0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
> #2  0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
> #3  0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
> #4  0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
> #5  0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
> 
> Thread 11 (Thread 0x3ff67fff910 (LWP 193307)):
> #0  0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
> #2  0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0
> #3  0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
> #4  0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
> #5  0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
> 
> Thread 10 (Thread 0x3ff84bf7910 (LWP 193306)):
> #0  0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
> #2  0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0
> #3  0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
> #4  0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
> #5  0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
> 
> Thread 9 (Thread 0x3ff853f8910 (LWP 193305)):
> #0  0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
> #2  0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0
> #3  0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
> #4  0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
> #5  0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
> 
> Thread 8 (Thread 0x3ff85bf9910 (LWP 193304)):
> #0  0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
> #2  0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0
> #3  0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
> #4  0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
> #5  0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
> 
> Thread 7 (Thread 0x3ff863fa910 (LWP 193303)):
> #0  0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
> #2  0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0
> #3  0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
> #4  0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
> #5  0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
> 
> Thread 6 (Thread 0x3ff86bfb910 (LWP 193302)):
> #0  0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
> #2  0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
> #3  0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
> #4  0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
> #5  0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
> 
> Thread 5 (Thread 0x3ff873fc910 (LWP 193301)):
> #0  0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
> #2  0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
> #3  0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
> #4  0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
> #5  0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
> 
> Thread 4 (Thread 0x3ff87bfd910 (LWP 193300)):
> #0  0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
> #2  0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
> #3  0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
> #4  0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
> #5  0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
> 
> Thread 3 (Thread 0x3ff883fe910 (LWP 193299)):
> #0  0x000003ff8b7113d4 in read () from /lib64/libpthread.so.0
> #1  0x000003ff8c65a648 in saferead () from /lib64/libvirt.so.0
> #2  0x000003ff8c65a718 in saferead_lim () from /lib64/libvirt.so.0
> #3  0x000003ff8c65abe4 in virFileReadHeaderFD () from /lib64/libvirt.so.0
> #4  0x000003ff8c69cb5a in virProcessRunInMountNamespace () from /lib64/libvirt.so.0
> #5  0x000003ff8c767bd6 in virSecuritySELinuxTransactionCommit () from /lib64/libvirt.so.0
> #6  0x000003ff8c75fb1c in virSecurityManagerTransactionCommit () from /lib64/libvirt.so.0
> #7  0x000003ff8c75bb70 in virSecurityStackTransactionCommit () from /lib64/libvirt.so.0
> #8  0x000003ff8c75fb1c in virSecurityManagerTransactionCommit () from /lib64/libvirt.so.0
> #9  0x000003ff6612c492 in qemuSecuritySetAllLabel () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
> #10 0x000003ff660bcfd6 in qemuProcessLaunch () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
> #11 0x000003ff660c0916 in qemuProcessStart () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
> #12 0x000003ff66124a00 in qemuDomainObjStart.constprop.52 () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
> #13 0x000003ff66125030 in qemuDomainCreateWithFlags () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
> #14 0x000003ff8c87bfca in virDomainCreate () from /lib64/libvirt.so.0
> #15 0x000002aa1f2d516c in remoteDispatchDomainCreate (server=<optimized out>, msg=0x2aa4c488b00, args=<optimized out>, rerr=0x3ff883fdae8, client=<optimized out>) at remote/remote_daemon_dispatch_stubs.h:4434
> #16 remoteDispatchDomainCreateHelper (server=<optimized out>, client=<optimized out>, msg=0x2aa4c488b00, rerr=0x3ff883fdae8, args=<optimized out>, ret=0x3ff78004c80) at remote/remote_daemon_dispatch_stubs.h:4410
> #17 0x000003ff8c78bc80 in virNetServerProgramDispatch () from /lib64/libvirt.so.0
> #18 0x000003ff8c792aba in virNetServerHandleJob () from /lib64/libvirt.so.0
> #19 0x000003ff8c6bbb16 in virThreadPoolWorker () from /lib64/libvirt.so.0
> #20 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
> #21 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
> #22 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
> 
> Thread 2 (Thread 0x3ff88bff910 (LWP 193298)):
> #0  0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
> #2  0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
> #3  0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
> #4  0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
> #5  0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
> 
> Thread 1 (Thread 0x3ff8cbd7970 (LWP 193297)):
> #0  0x000003ff8b5ee502 in poll () from /lib64/libc.so.6
> #1  0x000003ff8c65629c in virEventPollRunOnce () from /lib64/libvirt.so.0
> #2  0x000003ff8c654caa in virEventRunDefaultImpl () from /lib64/libvirt.so.0
> #3  0x000003ff8c7921de in virNetDaemonRun () from /lib64/libvirt.so.0
> #4  0x000002aa1f2a4c12 in main (argc=<optimized out>, argv=<optimized out>) at remote/remote_daemon.c:1461
> 
> > When I try to reproduce I always get one domain running and the other
> > failing to start because qemu is unable to do its locking. When I put
> > <shareable/> in both, they start successfully.
> 
> Yes, I get the same (expected) behaviour, just with the timeout.
> 
> > TBH, I don't run SELinux enabled host so I'm testing DAC only, but the
> > changes to the code are merely the same. So I'm wondering if this is
> > really an issue in my SELinux impl or somewhere else.
> > 
> > BTW: The one minute timeout comes from patch 16/23:
> > 
> >   #define LOCK_ACQUIRE_TIMEOUT 60
> > 
> > Michal
> 
> It's entirely possible that my setup is broken, this was just a quick
> test. If you can't reproduce it or make sense out of the backtrace then
> it's fine, I will try to put some time to debug it next week. I was just
> interested if this would fix a long-standing bug where this scenario
> would crash the first domain because the label changed.
> 
> Bjoern
> 

Still seeing the same timeout. Is this expected behaviour?

-- 
IBM Systems
Linux on Z & Virtualization Development
------------------------------------------------------------------------
IBM Deutschland Research & Development GmbH
Schönaicher Str. 220, 71032 Böblingen
Phone: +49 7031 16 1819
------------------------------------------------------------------------
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 902 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20180919/e888430c/attachment-0001.sig>