[Libvir] Re: support for hvm guests

[Jim Fehlig]

Anthony Liguori wrote:

> Jim Fehlig wrote:
>
>> Daniel Veillard wrote:
>>
>>> On Wed, May 03, 2006 at 09:43:36AM -0600, Jim Fehlig wrote:
>>>  
>>>
>>>> Currently I can get info on hvm domains but unable to create them.  
>>>> The create request is sent to xend but then blocks reading from 
>>>> xend in xend_req().  No problem creating the domain using xm.  I 
>>>> have looked at the config coming into xend and it is identical 
>>>> between xm and virsh/libvirt.  Need to poke around xend and see why 
>>>> there is no response to the virsh/libvirt create request for hvm 
>>>> domains.
>>>>   
>>>
>>>
>>>  Hum, basically to debug this kind of things I spend my time in
>>> /var/log/xend.log especially since it prints what it received at 
>>> creation
>>> time, you can at least check the informations coming from libvirt and
>>> from xm look alike.
>>>  Unfortunately I won't be able to help you, first because I'm about to
>>> take vacations for 3 weeks :-), second because I don't have one of 
>>> those
>>> new Intel CPUs, but I am sure others will give an hand if needed.
>>>  
>>>
>>
>> Well, I have found the problem but not a fix :-).  When posting the 
>> create op to xend, the libvirt code currently waits for a response 
>> from xend (xend_req() is called after posting in xend_post()).  For 
>> hvm guests, xend does not respond and libvirt blocks indefinitely on 
>> read.  If I skip over the call to xend_req() for the create 
>> operation, wait_for_devices() and subsequently unpause() are called 
>> and the hvm guest is launched.
>
>
> What do you mean by Xend does not respond?


OK, let me try this again after some experimentation today.  First, I'm 
using CVS libvirt and it does not use xmlrpc but the "old" S-expression 
over http - so I take a different code path to domain_create().  I set a 
breakpoint (pdb.set_trace()) in domain_create() and attempted to launch 
the hvm guest with xm and virsh/libvirt.  The stack trace from xm is

(Pdb) bt
  /usr/lib/python2.4/threading.py(442)__bootstrap()
-> self.run()
  /usr/lib/python2.4/threading.py(422)run()
-> self.__target(*self.__args, **self.__kwargs)
  /usr/lib/python2.4/SocketServer.py(463)process_request_thread()
-> self.finish_request(request, client_address)
  /usr/lib/python2.4/SocketServer.py(254)finish_request()
-> self.RequestHandlerClass(request, client_address, self)
  /usr/lib/python2.4/SocketServer.py(521)__init__()
-> self.handle()
  /usr/lib/python2.4/BaseHTTPServer.py(316)handle()
-> self.handle_one_request()
  /usr/lib/python2.4/BaseHTTPServer.py(310)handle_one_request()
-> method()
  /usr/lib/python2.4/SimpleXMLRPCServer.py(432)do_POST()
-> response = self.server._marshaled_dispatch(
  
/usr/lib/python2.4/site-packages/xen/util/xmlrpclib2.py(103)_marshaled_dispatch()
-> response = self._dispatch(method, params)
  /usr/lib/python2.4/SimpleXMLRPCServer.py(406)_dispatch()
-> return func(*params)
  
/usr/lib/python2.4/site-packages/xen/xend/server/XMLRPCServer.py(65)domain_create()
-> info = XendDomain.instance().domain_create(config)
 > 
/usr/lib/python2.4/site-packages/xen/xend/XendDomain.py(227)domain_create()
-> self.domains_lock.acquire()
(Pdb)

and the stack trace using virsh/libvirt:

(Pdb) bt
  /usr/lib/python2.4/threading.py(442)__bootstrap()
-> self.run()
  /usr/lib/python2.4/threading.py(422)run()
-> self.__target(*self.__args, **self.__kwargs)
  /usr/lib/python2.4/site-packages/xen/web/httpserver.py(288)run()
-> self.processRequest(sock, addr)
  
/usr/lib/python2.4/site-packages/xen/web/httpserver.py(316)processRequest()
-> rp.process()
  /usr/lib/python2.4/site-packages/xen/web/httpserver.py(110)process()
-> res = req.process()
  /usr/lib/python2.4/site-packages/xen/web/httpserver.py(140)process()
-> return self.render(resource)
  /usr/lib/python2.4/site-packages/xen/web/httpserver.py(180)render()
-> val = resource.render(self)
  /usr/lib/python2.4/site-packages/xen/web/resource.py(59)render()
-> return meth(req)
  
/usr/lib/python2.4/site-packages/xen/xend/server/SrvDomainDir.py(135)render_POST()
-> return self.perform(req)
  /usr/lib/python2.4/site-packages/xen/web/SrvBase.py(85)perform()
-> return op_method(op, req)
  
/usr/lib/python2.4/site-packages/xen/xend/server/SrvDomainDir.py(77)op_create()
-> dominfo = self.xd.domain_create(config)
 > 
/usr/lib/python2.4/site-packages/xen/xend/XendDomain.py(227)domain_create()
-> self.domains_lock.acquire()
(Pdb)

Continuing in the debugger I find that all routines complete up to 
process() in httpserver.py line 110 (note that my line numbers may be 
skewed a little due to insertion of debug code).  After executing the 
statement "res = req.process()", I take the else clause of "if 
isinstance(res, ThreadRequest):" - which calls req.finish() which in 
turn closes the connection.  So no response is sent to libvirt, which is 
blocked waiting to read something from xend.

As mentioned before, if I ignore reading a response from xend on the 
create op and go directly to wait_for_devices and unpause then the guest 
is launched as expected.  BTW, I do get responses from xend on 
wait_for_devices and unpause ops.

> The path is a normal domain_create().  The path within Xend is 
> server/XMLRPCServer.py but that's a very thin wrapper around a direct 
> call to XendDomain.py:domain_create().  domain_create() should always 
> return.  If it's not, there's a bug in Xend I think.


Not in the case of libvirt - see above.

> Can you give me a very concrete test case for this?  The exact 
> parameters being passed to domain_create() (it should just be an 
> S-Expression config).


Sent in a previous post.

I will look into this more tomorrow - have other fires that need 
attention now.

Thanks for the help :-).
Jim