[libvirt] Xen and out of memory stuff, blocks libvirtd

[Daniel P. Berrange]

On Wed, Oct 08, 2008 at 07:57:08PM +0200, Stefan de Konink wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
> 
> Daniel P. Berrange schreef:
> > On Wed, Oct 08, 2008 at 07:32:21PM +0200, Stefan de Konink wrote:
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA512
> >>
> >> [2008-10-08 19:14:05 13001] ERROR (XendCheckpoint:157) Save failed on
> >> domain klant1_monetdb (17) - resuming.
> >> Traceback (most recent call last):
> >>   File "usr/lib64/python2.5/site-packages/xen/xend/XendCheckpoint.py",
> >> line 125, in save
> >>     forkHelper(cmd, fd, saveInputHandler, False)
> >>   File "usr/lib64/python2.5/site-packages/xen/xend/XendCheckpoint.py",
> >> line 358, in forkHelper
> >>     child = xPopen3(cmd, True, -1, [fd, xc.handle()])
> >>   File "usr/lib64/python2.5/site-packages/xen/util/xpopen.py", line 100,
> >> in __init__
> >>     self.pid = os.fork()
> >> OSError: [Errno 12] Cannot allocate memory
> >> [2008-10-08 19:14:05 13001] DEBUG (XendDomainInfo:2456)
> >> XendDomainInfo.resumeDomain(17)
> >>
> >> After this point, I am unable to contact the libvird daemon.
> > 
> > Well your host OS has run out of memory, so pretty much everything
> > will be fubar at this point. If it can't even fork() then there's
> > not much libvirt until this is resolved. libvirt tries to handle
> > OOM scenario but there's some places we might have missed, but
> > even so, the best libvirt can do is to drop all further requests
> > as gracefully as possible. 
> 
> Not quite right, the host OS still has 32MB of memory at this point. But
> I agree strange things, even with later saves. If there is anything that
> is able to remove that painful xend stuff... I'm happy to give it a shot.

The host had 32 MB at the time you measured it. That's not the same
as the available memory at the time the kernel tried to run fork().
When fork() failed, and Xen aborted the restoration attempt, it will
have free'd up a bunch of resources being used for the restore.

This is nothing todo with XenD - the error shows fork() failed due
to lack of OS memory. Whether running XenD or not, if the kernel runs
out of memory you're doomed. You need more RAM allocated to your host.

Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|