[libvirt-users] libvirt hang on domain attemp to create domain
Александр
sss at sss.chaoslab.ru
Mon Jun 24 09:20:00 UTC 2013
В письме от Четверг, 20-июн-2013 15:59:56 пользователь Александр написал:
> В письме от Четверг, 20-июн-2013 13:25:06 пользователь Michal Privoznik
>
> написал:
> > On 20.06.2013 12:56, Alexandr wrote:
> > > On 20.06.2013 10:44, Michal Privoznik wrote:
> > >> On 20.06.2013 09:40, Александр wrote:
> > >>> В письме от Четверг, 20-июн-2013 09:15:14 Вы написали:
> > >>>> On 19.06.2013 21:24, Александр wrote:
> > >>>>> В письме от Среда, 19-июн-2013 09:35:27 пользователь Michal
> > >>>>> Privoznik
> > >>>
> > >>> написал:
> > >>>>>> On 18.06.2013 22:21, Александр wrote:
> > >>>>>>> В письме от Вторник, 18-июн-2013 12:10:41 пользователь Michal
> > >>>>>>> Privoznik
> > >>>>>>>
> > >>>>>>> написал:
> > >>>>>>>> On 18.06.2013 07:14, Александр wrote:
> > >>>>>>>>> good day, i am trying to create domain via virDomainDefineXML.
> > >>>>>>>>> and my
> > >>>>>>>>> program hanging, also i failed to caonnect to libvirt daemon
> > >>>>>>>>> with
> > >>>>>>>>> virt-manager after this, only killing libvirtd via SIGKILL and
> > >>>>>>>>> restarting
> > >>>>>>>>> it solveing hang (i mean after restart i able to connect with
> > >>>>>>>>> virt-manager), i using following domain xml
> > >>>>>>>>>
> > >>>>>>>>> <?xml version="1.0" encoding="UTF-8"?>
> > >>>>>>>>> <domain type="kvm">
> > >>>>>>>>>
> > >>>>>>>>> <tittle>Automatically created domain with default
> > >>>>>>>>> configuration</tittle>
> > >>>>>>>>> <name>vm1</name>
> > >>>>>>>>> <description>no description</description>
> > >>>>>>>>> <os>
> > >>>>>>>>>
> > >>>>>>>>> <type>hvm</type>
> > >>>>>>>>>
> > >>>>>>>>> <boot dev="cdrom"/>
> > >>>>>>>>> <boot dev="hd"/>
> > >>>>>>>>>
> > >>>>>>>>> </os>
> > >>>>>>>>>
> > >>>>>>>>> <vcpu>1</vcpu>
> > >>>>>>>>> <memory unit="MB">512</memory>
> > >>>>>>>>> <cpu model="host-model">
> > >>>>>>>>>
> > >>>>>>>>> <topology sockets="1" cores="1" threads="1"/>
> > >>>>>>>>>
> > >>>>>>>>> </cpu>
> > >>>>>>>>>
> > >>>>>>>>> <clock offset="localtime"/>
> > >>>>>>>>> <devices>
> > >>>>>>>>>
> > >>>>>>>>> <graphics type="vnc" autoport="yes" password="admin"/>
> > >>>>>>>>>
> > >>>>>>>>> <channel type="unix">
> > >>>>>>>>>
> > >>>>>>>>> <source mode="bind" path="/var/run/vm1.sock"/>
> > >>>>>>>>>
> > >>>>>>>>> <target type="virtio" name="org.guest_agent.0"/>
> > >>>>>>>>>
> > >>>>>>>>> </channel>
> > >>>>>>>>>
> > >>>>>>>>> </devices>
> > >>>>>>>>>
> > >>>>>>>>> </domain>
> > >>>>>>>>>
> > >>>>>>>>> i have this calls on stack
> > >>>>>>>>>
> > >>>>>>>>> 0 0x00007ffff48c1750 __poll_nocancel
> > >>>>>>>>> 1 0x00007ffff649e5ca virNetClientIOEventLoop
> > >>>>>>>>
> > >>>>>>>> This is just probably a client waiting for the corresponding
> > >>>>>>>> libvirtd
> > >>>>>>>> to
> > >>>>>>>> provide a reply message. Are you able to get libvirtd debug logs
> > >>>>>>>> so we
> > >>>>>>>> can see what's going on?
> > >>>>>>>>
> > >>>>>>>> http://wiki.libvirt.org/page/DebugLogs
> > >>>>>>>>
> > >>>>>>>> BTW, what's the libvirt version you are running?
> > >>>>>>>>
> > >>>>>>>> Michal
> > >>>>>>>>
> > >>>>>>>>> 2 0x00007ffff649ed96 virNetClientSendInternal
> > >>>>>>>>> 3 0x00007ffff64a0443 virNetClientSendWithReply
> > >>>>>>>>> 4 0x00007ffff64a0daa virNetClientProgramCall
> > >>>>>>>>> 5 0x00007ffff6478566 callFull.isra.2
> > >>>>>>>>> 6 0x00007ffff6487fa2 remoteDomainDefineXML
> > >>>>>>>>> 7 0x00007ffff6449d2f virDomainDefineXML
> > >>>>>>>>> 8 0x0000000000426941 virt_domain::create
> > >>>>>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp
> > >>>>>>>>> 295
> > >>>>>>>>> 9 0x000000000040c1a9 main
> > >>>>>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp 40
> > >>>>>>>>>
> > >>>>>>>>> what i am doing wrong ?
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> _______________________________________________
> > >>>>>>>>> libvirt-users mailing list
> > >>>>>>>>> libvirt-users at redhat.com
> > >>>>>>>>> https://www.redhat.com/mailman/listinfo/libvirt-users
> > >>>>>>>
> > >>>>>>> i have libvirt version 1.0.6, debug log in attachment.
> > >>>>>>
> > >>>>>> Yuck, this is 1.0.6; Well, from the logs it seems like the daemon
> > >>>>>> is
> > >>>>>> stuck. Can you attach to the libvirtd while it's being stuck and
> > >>>>>> run 't
> > >>>>>> a a bt' and post the result? I wonder if we have a deadlock
> > >>>>>> somewhere.
> > >>>>>>
> > >>>>>> Michal
> > >>>>>
> > >>>>> if i correctly understand you, you need debugger backtrace after
> > >>>>> libvirtd
> > >>>>> stuck ?, if so, here is bt from gdb
> > >>>>
> > >>>> You understand correctly. However, simple 'bt' doesn't show what's
> > >>>> going
> > >>>> on in other threads - libvirtd is multi threaded.
> > >>>>
> > >>>>> (gdb) bt
> > >>>>> #0 0x00007f00c1ae876d in poll () from /lib64/libc.so.6
> > >>>>> #1 0x00007f00c203cfcb in virEventPollRunOnce () from
> > >>>>> /usr/lib64/libvirt.so.0 #2 0x00007f00c203ba4d in
> > >>>>> virEventRunDefaultImpl
> > >>>>> () from
> > >>>>> /usr/lib64/libvirt.so.0
> > >>>>> #3 0x00007f00c214a5cd in virNetServerRun () from
> > >>>>> /usr/lib64/libvirt.so.0
> > >>>>> #4 0x00007f00c2aea08a in main ()
> > >>>>
> > >>>> So all we see here is the libvirtd event loop. That's why we need 't
> > >>>> a a
> > >>>> bt' which is 'thread apply all backtrace' which will run 'bt' over
> > >>>> all
> > >>>> threads.
> > >>>>
> > >>>> Michal
> > >>>
> > >>> how can i make it ?
> > >>
> > >> The very same way as you did 'bt'. Just instead of typing 'bt' you'll
> > >> type 't a a bt'.
> > >>
> > >> Michal
> > >
> > > here is backtrace from all threads:
> > >
> > > (gdb) t a a bt
> > >
> > > Thread 11 (Thread 0x7fdd8db82700 (LWP 20321)):
> > > #0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/libpthread.so.0
> > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > > #2 0x00007fdd9476582b in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #3 0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > >
> > > Thread 10 (Thread 0x7fdd8d381700 (LWP 20322)):
> > > #0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/libpthread.so.0
> > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > > #2 0x00007fdd9476582b in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #3 0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > >
> > > Thread 9 (Thread 0x7fdd8cb80700 (LWP 20323)):
> > > #0 0x00007fdd944bc344 in __lll_lock_wait () from /lib64/libpthread.so.0
> > > #1 0x00007fdd944b7e56 in _L_lock_511 () from /lib64/libpthread.so.0
> > > #2 0x00007fdd944b7caa in pthread_mutex_lock () from
> > > /lib64/libpthread.so.0
> >
> > Bingo! So we've a deadlock here. Somebody has forgotten to unlock a
> > domain. So as we go over the domain list to check for name (if it exists
> > already or not), we deadlock. The process goes like this:
> >
> > 1) if adding a domain the the internal list (I should write rather hash
> > table where UUID is the key, and the pointer to the internal structure
> > representing domain is the value), we search for UUID. Since this is the
> > key, we don't have to lock anything (besides the hash table itself).
> >
> > 2) Okay, no matching UUID. But what about domain name? We go through the
> > table again, over each element, and:
> > 2a) lock the element
> > 2b) compare the names
> > 2c) unlock the element
> >
> > And since a domain withing the table is locked (and all none of the
> > threads is using it) we deadlock in step 2a) as there's no one to come
> > and unlock the domain. Sigh. Hopefully, I'll have some time to dig into
> > the log you've provided and get more precise info.
> >
> > > #3 0x00007fdd94773861 in virDomainObjListSearchName () from
> > > /usr/lib64/libvirt.so.0
> > > #4 0x00007fdd9473c8a5 in virHashSearch () from /usr/lib64/libvirt.so.0
> > > #5 0x00007fdd9478cc3d in virDomainObjListAddLocked.isra.43 () from
> > > /usr/lib64/libvirt.so.0
> > > #6 0x00007fdd9478cd47 in virDomainObjListAdd () from
> > > /usr/lib64/libvirt.so.0
> > > #7 0x00007fdd875a311c in qemuDomainDefineXML () from
> > > /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
> > > #8 0x00007fdd947ead2f in virDomainDefineXML () from
> > > /usr/lib64/libvirt.so.0
> > > #9 0x00007fdd95206b4d in remoteDispatchDomainDefineXMLHelper ()
> > > #10 0x00007fdd94848d27 in virNetServerProgramDispatch () from
> > > /usr/lib64/libvirt.so.0
> > > #11 0x00007fdd94844068 in virNetServerHandleJob () from
> > > /usr/lib64/libvirt.so.0
> > > #12 0x00007fdd94765795 in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #13 0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #14 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #15 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > >
> > > Thread 8 (Thread 0x7fdd8c37f700 (LWP 20324)):
> > > #0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/libpthread.so.0
> > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > > #2 0x00007fdd9476582b in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #3 0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > >
> > > Thread 7 (Thread 0x7fdd8bb7e700 (LWP 20325)):
> > > #0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/libpthread.so.0
> > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > > #2 0x00007fdd9476582b in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #3 0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > >
> > > Thread 6 (Thread 0x7fdd8b37d700 (LWP 20326)):
> > > #0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/libpthread.so.0
> > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > > #2 0x00007fdd9476584b in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #3 0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > >
> > > Thread 5 (Thread 0x7fdd8ab7c700 (LWP 20327)):
> > > #0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/libpthread.so.0
> > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > > #2 0x00007fdd9476584b in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #3 0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > >
> > > Thread 4 (Thread 0x7fdd8a37b700 (LWP 20328)):
> > > #0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/libpthread.so.0
> > > ---Type <return> to continue, or q <return> to quit---
> > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > > #2 0x00007fdd9476584b in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #3 0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > >
> > > Thread 3 (Thread 0x7fdd89b7a700 (LWP 20329)):
> > > #0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/libpthread.so.0
> > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > > #2 0x00007fdd9476584b in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #3 0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > >
> > > Thread 2 (Thread 0x7fdd89379700 (LWP 20330)):
> > > #0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/libpthread.so.0
> > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > > #2 0x00007fdd9476584b in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #3 0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > >
> > > Thread 1 (Thread 0x7fdd95166740 (LWP 20320)):
> > > #0 0x00007fdd941e376d in poll () from /lib64/libc.so.6
> > > #1 0x00007fdd94737fcb in virEventPollRunOnce () from
> > > /usr/lib64/libvirt.so.0
> > > #2 0x00007fdd94736a4d in virEventRunDefaultImpl () from
> > > /usr/lib64/libvirt.so.0
> > > #3 0x00007fdd948455cd in virNetServerRun () from
> > > /usr/lib64/libvirt.so.0
> > > #4 0x00007fdd951e508a in main ()
> > >
> > > and also from my program if it important:
> > >
> > > 0 0x00007ffff48c1750 __poll_nocancel
> > > 1 0x00007ffff649e5ca virNetClientIOEventLoop
> > > 2 0x00007ffff649ed96 virNetClientSendInternal
> > > 3 0x00007ffff64a0443 virNetClientSendWithReply
> > > 4 0x00007ffff64a0daa virNetClientProgramCall
> > > 5 0x00007ffff6478566 callFull.isra.2
> > > 6 0x00007ffff6487fa2 remoteDomainDefineXML
> > > 7 0x00007ffff6449d2f virDomainDefineXML
> > > 8 0x0000000000426941 virt_domain::create
> > > /home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp 295
> > > 9 0x000000000040c1a9 main
> > > /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp 40
> >
> > Michal
>
> thx for quick reply, i will wait.
i have tried to workaround this, added uuid to xml, but nothing changed,
currently my xml looks like
<?xml version="1.0" encoding="UTF-8"?>
<domain type="kvm">
<tittle>Automatically created domain with default configuration</tittle>
<uuid>2961e3ca-6318-4af7-85e5-72affde5dc6e</uuid>
<name>vm1</name>
<description>no description</description>
<os>
<type>hvm</type>
<boot dev="cdrom"/>
<boot dev="hd"/>
</os>
<vcpu>1</vcpu>
<memory unit="MB">512</memory>
<cpu model="host-model">
<topology sockets="1" cores="1" threads="1"/>
</cpu>
<clock offset="localtime"/>
<devices>
<graphics type="vnc" autoport="yes" password="admin"/>
<channel type="unix">
<source mode="bind" path="/var/run/vm1.sock"/>
<target type="virtio" name="org.guest_agent.0"/>
</channel>
</devices>
</domain>
but virt-manager can create domain, how can i find difference in configs ?,
how can i obtain config from virt manager (without deep research of code)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: This is a digitally signed message part.
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20130624/c0720b34/attachment.sig>
More information about the libvirt-users
mailing list