[libvirt] [RFC PATCH] support multifunction PCI device

Daniel P. Berrange berrange at redhat.com
Wed May 11 07:55:56 UTC 2011


On Wed, May 11, 2011 at 10:30:43AM +0800, Wen Congyang wrote:
> At 05/10/2011 06:00 PM, Daniel P. Berrange Write:
> > On Tue, May 10, 2011 at 02:00:17PM +0800, Wen Congyang wrote:
> >> We want to use more than 200+ device. Libvirt does not use multi
> >> function PCI device and PCI-to-PCI bridge. So we can not use more
> >> than 200+ device if it's a PCI device or it's controller is a PCI
> >> device.
> > 
> > IMHO using multi function support is a dead end, because it
> > makes hotplug completely unusable and is not even possible
> > for most PCI devices. The way we want to raise the device
> > limit is
> > 
> >   a. Supporting multiple PCI domains (multiple PCI root complexes)
> 
> IIUC, qemu do not support multiple PCI domains now.

There are people who are working on making that supported in KVM
real soon.

> >   b. Adding PCI bridges
> 
> I do not know whether qemu supports it. I find hw/pci_bridge.c in qemu's source,
> so qemu may support it. I will confirm it.

I'm told this is already supported, but I've not tested it myself.

> >   c. A virtio-scsi controller to allow > 1 virtio disk per PCI device
> 
> Hmm, do you mean this:
> -device virtio-blk-pci,ports=4,drive0=hda,drive1=hdb,drive2=hdc,...

No it is actually going to be a new device type, and I'm expecting
it will work very like the LSI scsi does in terms of device setup.
eg create a controller device, and then create devices for each drive.

  -device virtio-scsi-pci,id=vscsi1
  -drive file=/some/disk,id=vscsi1.1
  -device virtio-blk-pci,bus=vscsi1,drive=vscsi1.1
  -drive file=/some/disk,id=vscsi1.2
  -device virtio-blk-pci,bus=vscsi1,drive=vscsi1.2
  -drive file=/some/disk,id=vscsi1.2
  -device virtio-blk-pci,bus=vscsi1,drive=vscsi1.2

> > any of those will dramatically increase the number of devices
> > we can use without the horrible hotplug problems that multifunction
> > introduces.
> > 
> >> This patch adds the support of multi function PCI device. It
> >> does not support to hot plug/unplug multi function PCI device.
> >>
> >> TODO:
> >> 1. support to hot plug multi function PCI device
> >>    We only can hot plug one device at one time. I think we should
> >>    introduce another command 'virsh attach-devices XXX' to support
> >>    hot plug more than one device at one time?
> > 
> > Since you can't do practical hotplug of multifunction devices at the
> > QEMU layer at all, there's nothing useful we can do at libvirt either.
> 
> We can hotplug multifunction PCI devices like this:
> 1. Before hot pluging,
> # lspci
> 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
> 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
> 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
> 00:01.2 USB Controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
> 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
> 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
> 00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20)
> 00:04.0 RAM memory: Red Hat, Inc Virtio memory balloon
> 00:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> 
> 2. hot plug multi function PCI devices:
> # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi2,bus=pci.0,addr=0x06.0x07'
> # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi3,bus=pci.0,addr=0x06.0x06'
> # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi4,bus=pci.0,addr=0x06.0x05'
> # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi5,bus=pci.0,addr=0x06.0x04'
> # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi6,bus=pci.0,addr=0x06.0x03'
> # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi7,bus=pci.0,addr=0x06.0x02'
> # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi8,bus=pci.0,addr=0x06.0x01'
> # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi9,multifunction=on,bus=pci.0,addr=0x06.0x00'

Hmm, that's kinda wierd & i'm suprised it works, particularly for LSI
since I thought guest drivers would need support for multifunction too.

> 
> 3. After hot plug multi function PCI devices:
> # lspci
> 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
> 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
> 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
> 00:01.2 USB Controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
> 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
> 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
> 00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20)
> 00:04.0 RAM memory: Red Hat, Inc Virtio memory balloon
> 00:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> 00:06.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> 00:06.1 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> 00:06.2 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> 00:06.3 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> 00:06.4 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> 00:06.5 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> 00:06.6 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> 00:06.7 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> 
> 
> > 
> >>
> >> 2. support to hot unplug multi function PCI device
> >>    hot unpluging multi function PCI device meas that all the other
> >>    devices on the same slot will be hot unpluged. So we should do
> >>    some cleanup and remove the other devices too. If the other
> >>    device on the same slot does not support hot unpluged, or it is a
> >>    a controller and some other devices still use this controller,
> >>    I think we should refuse to hot unplug this mutlti function PCI
> >>    device.
> > 
> > IMHO these kind of restrictions will make life really unpleasant
> > for applications and are a reason we should *not* support the
> > multifunction code. Instead we should focus on one of the other
> > 3 options I mention above.
> 
> Yes, there's too many restrictions about hot plug/unplug multifunction
> PCI devices.


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|




More information about the libvir-list mailing list