Draft of Tuesday's News- Virt Interview with Daniel Berrange

[Jack Aboutboul]

Below is the the text of the interview, didn't really think about the 
lead it, but thats trivial...

*1.  Please introduce yourself and what you do and how you got started 
working on virtualization.*

I'm one of the lead developers for the libvirt project and am actively 
involved in many related areas of open source development (qemu/kvm, 
xen, gtk-vnc, virt-manager, to name but a few). I also co-maintain many 
of these packages in Fedora and RHEL, along with many others in Red 
Hat's virtualization team.

More than three years ago (shortly after transferring into Red Hat's 
Engineering team, from consulting services) I was working on the OLPC 
project. We needed a way to easily test the OS images we were building 
without needing real hardware. As a proof of concept, I hacked up a 
simple GTK application to run images them under QEMU. At around the same 
time Daniel Veillard had started the libvirt project and there was a 
desire for a desktop application to manage Xen using libvirt. So I 
switched over to the virtualization team, wrote virt-manager for Fedora 
6, and my involvement in all areas of open source virtualization grew 
from there.

*2.  Many people view the work being done on virtualization as a feature 
set of major importance and significance.  Can you give us a brief 
overview of some of the changes that we can expect to see in F11? 
*
The open source virtualization development effort is so large now, that 
it is useful to discuss each stream in turn.

At the lowest layer is obviously the Linux kernel & KVM/QEMU. There has 
been a major acceleration of development in QEMU and push to merge KVM 
into the official QEMU source repository. There's ever continuing work 
on performance, stability, scalability & reliability in KVM. PCI device 
passthrough is one new feature we're highlighting for Fedora 11. The 
return of Xen Dom0 was not to be, as the Dom0 paravirt_ops merge with 
the upstream Linux kernel is still an ongoing process.

At the middle layer is libvirt, providing a consistent management API 
across different virtualization technologies. New features in libvirt, 
since F10, include PCI device passthrough for Xen and KVM, the sVirt 
security driver using SELinux to protect KVM guests from each other, 
thread safety of all libvirt APIs, improved scalability, reliability and 
debugging for the libvirtd daemon and support for SCSI HBAs and 
copy-on-write volumes in the storage management APIs.

The top layer covers end user tools such as virt-install & virt-manager. 
virt-manager is undergoing a significant (and ongoing) overhaul of its 
user interface. The first improvements arriving for Fedora 11 are in the 
guest installation process and storage management capabilities. As guest 
installation is first task most users try, ensuring this is simple and 
reliable is key to making a good first impression. Guest desktop 
interaction is another historical pain point which has been a focus for 
improvements in Fedora 11.

With every release we also try to make a significant step forward in 
security of the virtualization stack. In Fedora 11 the focus  has been 
on SELinux to protect guests from each other and SASL to authenticate 
VNC users.

*3.  There have been some large changes in virt-manager and libvirt, 
which are at the core of the user experience when it comes to 
virtualization.  Can you talk to use about those some more?*

The guest installation process and desktop interaction are the most 
critical areas for making a good first impression. In the virt-manager 
re-design the wizard used for installing new guests has been 
streamlined, cutting out three redundant steps. Where possible, it will 
automatically detect the type of operating system being installed and 
choose the best configuration options to optimize for this OS, no longer 
requiring the user to figure this out for themselves. The installation 
process now directly utilizes the libvirt storage management APIs to 
allow easy creation of files in a variety of formats (raw, qcow2, vmdk, 
etc), allocation of new local disk partitions or LVM volumes and access 
to LUNs exported by iSCSI targets. This is particularly useful when 
remotely managing virtualization hosts, allowing regular administrator 
tasks to be performed from the virt-manager UI without resorting to 
command line SSH sessions.

The mouse pointer has been a constant source of trouble for 
virtualization management applications. Getting the guest mouse pointer 
to track the host pointer is essentially impossible with the standard 
emulated PS/2 mouse. The solution is to provide a pointer device that 
supports absolute motion events, instead of relative events that the 
PS/2 mouse provides. For KVM and Xen, this means adding a USB tablet 
device, but historically Xorg has not been able to automatically 
configure this correctly. This is resolved with Fedora 11 guests, 
finally providing a guest pointer that moves in perfect sync with the 
host, not requiring the pointer to be confined to the guest window.

Users with non-US layout keyboards have also had a hard time getting 
their guests to support input of accented/special characters. The VNC 
protocol has now been extended to allow the hardware keycodes to be 
passed directly from the VNC client to the guest OS without any 
intermediate translation step. This should allow the guest OS complete 
control over the keyboard layout mapping, without a need for any special 
settings on the host.

The final piece of work was to increase the guest desktop resolution. 
The real Cirrus video card that QEMU emulates would never have done more 
than 800x600, but there are tricks that can be done in a virtual world. 
Thus a simple change to the Xorg cirrus driver allows it to detect that 
it is using a Cirrus card emulated by QEMU and increase the guest 
desktop resolution to 1024x768. Still not great by today's standards, 
but better than before. Longer term plans involve replacing the cirrus 
driver in QEMU with something more virtualization friendly.

*4.   Also, as people should take note of, there has been a lot of work 
done surrounding KVM and getting that well integrated into the whole 
virtualization setup in Fedora.  How has that work been going and has 
anything significant been done in that area in this release?*

Fedora was the first major Linux distribution to integrate KVM back in 
the Fedora 7 release. It became the default virtualization technology in 
Fedora 9, when it became clear we could no longer maintain the separate 
Xen host kernel until it was merged in the upstream Linux kernel. The 
great benefit of KVM from an distro integration point of view, is that 
it is there by default in all new Linux kernels. All that was required 
in Fedora was to turn on the module build and make sure the modules are 
always loaded when compatible CPUs are found. libvirt and virt-manager 
have also both supported KVM since it was first added to Fedora. Thus 
there hasn't been a need for much additional integration work for KVM. 
The focus has simply been on improving features available to KVM users 
via libvirt and virt-manager.
*
5. Glauber Costa has also done significant work merging KVM and QEMU. 
Can you explain to us what QEMU is and why the choice was made to merge 
it with QEMU and how that is of benefit to the user base?*

Earlier Fedora releases have have suffered from the divergence of QEMU 
and KVM code bases. Upstream QEMU has had releases almost 1 year apart, 
while KVM has been releasing at least once a month, if not more, using 
snapshots of the QEMU source repository. Thus the features available in 
QEMU were far behind those available in KVM even though they both shared 
the same fundamental code base and upstream development stream. It also 
doubled the work package maintainers had todo for security & bug fixes.

Since Fedora 10 though, the upstream QEMU community has accelerated its 
release schedule significantly and many of the KVM features have been 
merged back into the main QEMU code base. Thus we judged that the time 
was right to attempt to ship a single package containing both QEMU and 
KVM built from a single code base. For users this means that parity of 
features between QEMU and KVM, while the reduced burden on our Fedora 
package maintainers, ensures more timely security and bug fixes. Glauber 
also took the opportunity to split out all the virtual BIOS files and 
ROMs from QEMU into separate packages and ensure all are fully built 
from source using appropriate upstream source releases.

*6.  Virtualization and Security are two things that are being discussed 
more or less hand in hand these days, as the ability to create and use 
virtualized machines expands there are many security risks involved. Can 
you speak a bit to the work that was done one improving security both at 
the kernel level (sVirt) and also the user level with things like SASL 
for VNC Auth?*

In each Fedora release we try to make at least one significant step 
forward in the security of our virtualization technology. In Fedora 8, 
libvirt gained support for secure remote management using TLS for 
encryption and x509 client certificates for authentication, while 
GTK-VNC, QEMU, KVM and Xen were also all extended to add a VNC extension 
for TLS encryption providing a secure remote desktop. In Fedora 9 
libvirt was further extended to support SASL enabling secure remote 
managment with Kerberos for authentication and PolicyKit for local 
desktop authentication. Fedora 9 and 10 also increased use of SELinux to 
protect the host operating system from a compromised or malicious 
QEMU/KVM process.

The latter still did not provide any protection between guests, so one 
compromised QEMU process would still potentially be able to compromise 
another on the same host. Thus James Morris started work on a project 
known as sVirt, the first results of which are appearing in Fedora 11. 
The focus has been to provide isolation between guests running on a 
single host. libvirt directly integrates with SELinux to ensure every 
QEMU process it launches runs within a dedicated security context, only 
able to access its own assigned disk images. This protection is enabled 
by default on all Fedora 11 hosts using libvirt for management. As well 
as the security benefits, the end user experiance is improved because 
libvirt will automatically manage SELinux labelling for all guest disks, 
eliminating a major source of bug reports from previous Fedora releases.


*7.  These features have all evolved over time and over the previous 
Fedora releases and Fedora, as a distribution, has always been a leader 
in the virtualization realm.  Can you talk a bit now about the actual 
process of developing these features and how many of the improvements 
and enhancements to virtualization have come about as a direct result of 
the work done previously?  Also, what does the future look like?*

Virtualization technology in Fedora is reaping the benefit of very 
active upstream projects and the significant developer resources of Red 
Hat's Virtualization Engineering team. The combination of these provide 
great opportunities for new features to have their debut in Fedora releases.

The ideas for new features come from many sources, some from Fedora 
end-user experiances & consequent bug reports, some magically arrive on 
cue from upstream projects, while others are things that look to be 
important for future RHEL releases. With the PCI device passthrough 
feature in F11, the core support was all already done by the upstream 
KVM community. This is a important feature for future RHEL, so Red Hat 
put resources into a F11 feature to add support to libvirt for PCI 
passthrough with KVM and Xen and then expose this in virt-manager.

The feature aiming to improve the guest desktop interaction was a result 
of the persistent stream of bug reports from Fedora users. We had been 
considering ways to address this over course of several Fedora releases, 
but it was not until Fedora 11 that all the pieces of the solution 
finally came together from the various upstream projects.

The security improvements in virtualization have a different origin. 
Very few end users ever file explicit bug reports asking for the 
addition of more authentication / encryption features or to use more 
SELinux. If anything users ask for ability to more easily turn off 
existing security features. This is a case where the user is not always 
right. They do want more security, but they just don't know it yet!

As a developer it is neccessary to be very proactive with security 
improvements. This can be particularly challenging work because the 
solutions often involve working across multiple upstream communities.

Take the VNC SASL authentication feature in Fedora 11 as an example. The 
first step was to write a specification for a new VNC security 
extension, have it reviewed & get a code for it allocated by RealVNC. 
Work on QEMU was required to implement the server side. Work on GTK-VNC 
was needed for client side. For management tools, work on libvirt was 
required to get the new security type enabled for guests it launches and 
finally virt-manager was extended such that it knows how to login. 
That's give different projects involved for one feature. This is only 
practical by having a close working relationship with all the upstream 
communities and carefully coordinating the work there so it all arrives 
in time for the next Fedora release.

For the future we're happy that libvirt gained support for managing 
VirtualBox recently and likely to soon have a driver for the Open 
Nebular cloud management project. Expect more advancements in sVirt, 
allowing for tighter controls on what a virtual machine can do, for 
example, ability to restrict network access of guests. libvirt will also 
gain the ability to manage host network configuration in Fedora 12, 
removing the need to manually configure bridge devices. Container based 
virt may make a more formal appearance in Fedora 12 as the native Linux 
container (LXC) support improves in the kernel and libvirt. The overhaul 
of the virt-manager user interface also continues.


*8.  Working on Virtualization must be awfully time consuming and very 
involved.  Do you enjoy it?  What do you do to get away from the 
pressures of hacking?*

Working on open source virtualization technology is a great experiance 
because it is a really interesting & challenging field, having plenty of 
talented developers to work with & learn from. There is plenty of work 
still to be done at all levels of the stack from kernel/hypervisor right 
through to end user applications & not nearly enough time todo it all. 
I'm fortunate to be able to spread my work between upstream projects, 
the Fedora community and RHEL releases and maintainence. As for free 
time ? What free time  :-)  I try to find time for a photography, with 4 
out of my 5 cameras still using film, rather than digital.