[libvirt] serial console runs out of buffer and console messages soft lock the CPU

[Daniel P. Berrange]

On Tue, Nov 25, 2008 at 04:32:00PM -0800, Bryan McLellan wrote:
> I recently started configuring guests to use both a serial console and
> a graphics element. One of the nodes with this configuration kept
> locking up after running for some time. Trying to access the guest
> through virt-manager would leave virt-viewer hung at the connecting to
> screen. The guest would not respond to any network traffic including
> icmp. The user would destroy the domain and restart it to recover.
> Eventually the problem came to me and I found that immediately upon
> running 'virsh console domain' the guest would recover. The console
> contained:
> 
> BUG: soft lockup - CPU#0 stuck for 94393s! [postgresql-8.3:4975]
> 
> I discovered a bug in a puppet recipe caused postgres to reload
> regularly, printing a message to the console each time. Running puppet
> in a loop to continue producing these messages to the console caused
> another lockup that was recoverable by accessing the serial console in
> the same manner. I have since fixed that recipe, but I'm uncomfortable
> about the possibility of this happening again.
> 
> Is there a way to configure libvirt to cache the serial console on the
> host so the buffer on the guest does not fill up when you aren't
> connected to the console, or should I disable the serial console?

This is actually a bug in QEMU. It should never block the VM when
the serial port buffer fills up. Unfortunately when the serial port
was connected to a TTY, QEMU would block on the write :-( The recent
KVM releases fix this, but QEMU goes years without a release so
it still suffers from this. I'd recommend filing a bug against QEMU
in your distro. There's nothing libvirt can do to fix this.

Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|