[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [libvirt] new libvirt "pci" controller type and pcie/q35 (was Re: [PATCH 4/7] add pci-bridge controller type)



On Fri, 2013-04-12 at 11:46 -0400, Laine Stump wrote:
> On 04/11/2013 07:23 AM, Michael S. Tsirkin wrote:
> > On Thu, Apr 11, 2013 at 07:03:56AM -0400, Laine Stump wrote:
> >> On 04/10/2013 05:26 AM, Daniel P. Berrange wrote:
> >>> On Tue, Apr 09, 2013 at 04:06:06PM -0400, Laine Stump wrote:
> >>>> On 04/09/2013 04:58 AM, Daniel P. Berrange wrote:
> >>>>> On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote:
> >>>>> Actually I do wonder if we should reprent a PCI root as two
> >>>>> <controller> elements, one representing the actual PCI root
> >>>>> device, and the other representing the host bridge that is
> >>>>> built-in.
> >>>>>
> >>>>> Also we should use the actual model names, not 'pci-root' or
> >>>>> 'pcie-root' but rather i440FX for "pc" machine type, and whatever
> >>>>> the q35 model name is.
> >>>>>
> >>>>>  - One PCI root with built-in PCI bus (ie todays' setup)
> >>>>>
> >>>>>    <controller type="pci-root" index="0">
> >>>>>      <model name="i440FX"/>
> >>>>>    </controller>
> >>>>>    <controller type="pci" index="0"> <!-- Host bridge -->
> >>>>>      <address type='pci' domain='0' bus='0' slot='0''/>
> >>>> Isn't this saying that the bridge connects to itself? (since bus 0 is
> >>>> this bus)
> >>>>
> >>>> I understand (again, possibly wrongly) that the builtin PCI bus connects
> >>>> to the chipset using its own slot 0 (that's why it's reserved), but
> >>>> that's its address on itself. How is this bridge associated with the
> >>>> pci-root?
> >>>>
> >>>> Ah, I *think* I see it - the domain attribute of the pci controller is
> >>>> matched to the index of the pci-root controller, correct? But there's
> >>>> still something strange about the <address> of the pci controller being
> >>>> self-referential.
> >>> Yes, the index of the pci-root matches the 'domain' of <address>
> >>
> >> Okay, then the way that libvirt differentiates between a pci bridge that
> >> is connected to the root, and one that is connected to a slot of another
> >> bridge is 1) the "bus" attribute of the bridge's <address> matches the
> >> "index" attribute of the bridge itself, and 2) "slot" is always 0. Correct?
> >>
> >> (The corollary of this is that if slot == 0 and bus != index, or bus ==
> >> index and slot != 0, it is a configuration error).
> >>
> >> I'm still unclear on the usefulness of the pci-root controller though -
> >> all the necessary information is contained in the pci controller, except
> >> for the type of root. But in the case of pcie root, I think you're not
> >> allowed to connect a standard bridge to it, only a "dmi-to-pci-bridge"
> >> (i82801b11-bridge)
> > Yes you can connect a pci bridge to pcie-root.
> > It's represented as a root complex integrated device.

Is this accurate?  Per the PCI express spec, any PCI express device
needs to have a PCI express capability, which our pci-bridge does not.
I think this is one of the main differences for our i82801b11-bridge,
that it exposes itself as a root complex integrated endpoint, so we know
it's effectively a PCIe-to-PCI bridge.  We'll be asking for trouble
if/when we get guest IOMMU support if we are lax about using PCI-to-PCI
bridges where we should have PCIe-to-PCI bridges.  There are plenty of
examples to the contrary of root complex integrated endpoints without an
express capability, but that doesn't make it correct to the spec.

> ARGHH!! Just when I think I'm starting to understand *something* about
> these devices...
> 
> (later edit: after some coaching on IRC, I *think* I've got a bit better
> handle on it.)
> 
> >>>>>    </controller>
> >>>>>    <interface type='direct'>
> >>>>>       ...
> >>>>>      <address type='pci' domain='0' bus='0' slot='3'/>
> >>>>>    </controller>
> >>>>>
> >>>>>  - One PCI root with built-in PCI bus and extra PCI bridge
> >>>>>
> >>>>>    <controller type="pci-root" index="0">
> >>>>>      <model name="i440FX"/>
> >>>>>    </controller>
> >>>>>    <controller type="pci" index="0"> <!-- Host bridge -->
> >>>>>      <address type='pci' domain='0' bus='0' slot='0'/>
> >>>>>    </controller>
> >>>>>    <controller type="pci" index="1"> <!-- Additional bridge -->
> >>>>>      <address type='pci' domain='0' bus='0' slot='1'/>
> >>>>>    </controller>
> >>>>>    <interface type='direct'>
> >>>>>       ...
> >>>>>      <address type='pci' domain='0' bus='1' slot='3'/>
> >>>>>    </controller>
> >>>>>
> >>>>>  - One PCI root with built-in PCI bus, PCI-E bus and and extra PCI bridge
> >>>>>    (ie possible q35 setup)
> >>>> Why would a q35 machine have an i440FX pci-root?
> >>> It shouldn't, that's a typo
> >>>
> >>>>>    <controller type="pci-root" index="0">
> >>>>>      <model name="i440FX"/>
> >>>>>    </controller>
> >>>>>    <controller type="pci" index="0"> <!-- Host bridge -->
> >>>>>      <address type='pci' domain='0' bus='0' slot='0'/>
> >>>>>    </controller>
> >>>>>    <controller type="pci" index="1"> <!-- Additional bridge -->
> >>>>>      <address type='pci' domain='0' bus='0' slot='1'/>
> >>>>>    </controller>
> >>>>>    <controller type="pci" index="1"> <!-- Additional bridge -->
> >>>>>      <address type='pci' domain='0' bus='0' slot='1'/>
> >>>>>    </controller>
> >>>> I think you did a cut-paste here and intended to change something, but
> >>>> didn't - those two bridges are identical.
> >>> Yep, the slot should be 2 in the second one
> >>>
> >>>>>    <interface type='direct'>
> >>>>>       ...
> >>>>>      <address type='pci' domain='0' bus='1' slot='3'/>
> >>>>>    </controller>
> >>>>>
> >>>>> So if we later allowed for mutiple PCI roots, then we'd have something
> >>>>> like
> >>>>>
> >>>>>    <controller type="pci-root" index="0">
> >>>>>      <model name="i440FX"/>
> >>>>>    </controller>
> >>>>>    <controller type="pci-root" index="1">
> >>>>>      <model name="i440FX"/>
> >>>>>    </controller>
> >>>>>    <controller type="pci" index="0"> <!-- Host bridge 1 -->
> >>>>>      <address type='pci' domain='0' bus='0' slot='0''/>
> >>>>>    </controller>
> >>>>>    <controller type="pci" index="0"> <!-- Host bridge 2 -->
> >>>>>      <address type='pci' domain='1' bus='0' slot='0''/>
> >>>>>    </controller>
> 
> 
> There is a problem here - within a given controller type, we will now
> have the possibility of multiple controllers with the same index - the
> differentiating attribute will be in the <address> subelement, which
> could create some awkwardness. Maybe instead this should be handled with
> a different model of pci controller, and we can add a "domain" attribute
> at the toplevel rather than specifying an <address>?

On real hardware, the platform can specify the _BBN (Base Bus Number =
bus) and the _SEG (Segment = domain) of the host bridge.  So perhaps you
want something like:

<controller type="pci-host-bridge">
  <model name="i440FX"/>
  <address type="pci-host-bridge-addr" domain='1' bus='0'/>
</controller>

"index" is confusing to me.

> >>>>>    <interface type='direct'> <!-- NIC on host bridge 2 -->
> >>>>>       ...
> >>>>>      <address type='pci' domain='1' bus='0' slot='3'/>
> >>>>>    </controller>
> >>>>>
> >>>>>
> >>>>> NB this means that 'index' values can be reused against the
> >>>>> <controller>, provided they are setup on different pci-roots.
> >>>>>
> >>>>>> (also note that it might happen that the bus number in libvirt's config
> >>>>>> will correspond to the bus numbering that shows up in the guest OS, but
> >>>>>> that will just be a happy coincidence)
> >>>>>>
> >>>>>> Does this make sense?
> >>>>> Yep, I think we're fairly close.
> >>>> What about the other types of pci controllers that are used by PCIe? We
> >>>> should make sure they fit in this model before we settle on it.
> >>> What do they do ?
> 
> (The descriptions of different models below tell what each of these
> other devices does; in short, they're all just some sort of electronic
> Lego to help connect PCI and PCIe devices into a tree).
> 
> Okay, I'll make yet another attempt at understanding these devices, and
> suggesting how they can all be described in the XML. I'm thinking that
> *all* of the express hubs, switch ports, bridges, etc can be described
> in xml in the manner above, i.e.:
> 
>    <controller type='pci' index='n'>
>      <model type='xxx'/>
>    </controller>
> 
> and that the method for connecting a device to any of them would be by
> specifying:
> 
>      <address type='pci' domain='n' bus='n' slot='n' function='n'/>
> 
> Any limitations about which devices/controllers can connect to which
> controllers, and how many devices can connect to any particular
> controller will be derived from the <model type='xxx'/>. (And, as we've
> said before, although qemu doesn't assign each of these controllers a
> numeric bus id, and although we can make no guarantee that the bus id we
> use for a particular controller is what will be used by the guest
> BIOS/OS, it's still a convenient notation and works well with other
> hypervisors as well as qemu. I'll also note that when I run lspci on an
> X58-based machine I have here, *all* of the relationships between all
> the devices listed below are described with simple bus:slot.function
> numbers.)
> 
> Here is a list of the pci controller model types and their restrictions
> (thanks to mst and aw for repeating these over and over to me; I'm sure
> I still have made mistakes, but at least it's getting closer).
> 
> 
> <controller type='pci-root'>
> ============================
> 
> Upstream:         nothing
> Downstream:       only a single pci-root-bus (implied)
> qemu commandline: nothing (it's implied in the q35 machinetype)
> 
> Explanation:
> 
> Each machine will have a different controller called "pci-root" as
> outlined above by Daniel. Two types of pci-root will be supported:
> i440FX and q35. If a pci-root is not spelled out in the config, one will
> be auto-added (depending on machinetype).
> 
> An i440FX pci-root has an implicitly added pci-bridge at 0:0:0.0 (and
> any bridge that has an address of slot='0' on its own bus is, by
> definition, connected to a pci-root controller - the two are matched by
> setting "domain" in the address of the pci-bridge to "index" of the
> pci-root). This bridge can only have PCI devices added.
> 
> A q35 pci-root also implies a different kind of pci-bridge device - one
> that can only have PCIe devices/controllers attached, but is otherwise
> identical to the pci-bridge added for i440FX. This bus will be called
> "root-bus" (Note that there are generally followed conventions for what
> can be connected to which slot on this bus, and we will probably follow
> those conventions when building a machine, *but* we will not hardcode
> this convention into libvirt; each q35 machine will be an empty slate)
> 
> 
> <controller type='pci'>
> =======================
> 
> This will be used for *all* of the following controller devices
> supported by qemu:
> 
> <model type='pcie-root-bus'/> (implicit/integrated)
> ----------------------------
> 
> Upstream:         connect to pci-root controller *only*
> Downstream:       32 slots, PCIe devices only, no hotplug.
> qemu commandline: nothing (implicit in the q35-* machinetype)
> 
> This controller is the bus described above that connects to a q35's
> pci-root, and provides places for PCIe devices to connect. Examples are
> root-ports, dmi-to-pci-bridges sata controllers, integrated
> sound/usb/ethernet devices (do any of those that can be connected to the
> pcie-root-bus exist yet?).
> 
> There is only one of these controllers, and it will *always* be
> index='0', and will always have the following address:
> 
>   <address type='pci' domain='0' bus='0' slot='0' function='0'/>

Implicit devices make me nervous, why wouldn't this just be a pcie-root
(or pcie-host-bridge)?  If we want to support multiple host bridges,
there can certainly be more than one, so the index='0' assumption seems
to fall apart.

> <model type='root-port'/> (ioh3420)
> -------------------------
> 
> Upstream:         PCIe, connect to pcie-root-bus *only* (?)

yes

> Downstream:       1 slot, PCIe devices only (?)

yes

> qemu commandline: -device ioh3420,...
> 
> These can only connect to the "pcie-root-bus" of of a q35 (implying that
> this bus will need to have a different model name than the simple
> "pci-bridge"
> 
> 
> <model type='dmi-to-pci-bridge'/> (i82801b11-bridge)

I'm worried this name is either too specific or too generic.  What
happens when we add a generic pcie-bridge and want to use that instead
of the i82801b11-bridge?  The guest really only sees this as a
PCIe-to-PCI bridge, it just happens that on q35 this attaches at the DMI
port of the MCH.

> ---------------------------------
> 
> (btw, what does "dmi" mean?)

http://en.wikipedia.org/wiki/Direct_Media_Interface

> Upstream:         pcie-root-bus *only*

And only to a specific q35 slot (1e.0) for the i82801b11-bridge.

> Downstream:       32 slots, any PCI device, no hotplug (?)

Yet, but I think this is where we want to implement ACPI based hotplug.

> qemu commandline: -device i82801b11-bridge,...
> 
> 
> <model type='upstream-switch-port'/> (x3130-upstream)
> ------------------------------------
> 
> Upstream:         PCIe, connect to pcie-root-bus, root-port, or
> downstream-switch-port (?)

yes

> Downstream:       32 slots, connect *only* to downstream-switch-port

I can't verify that there are 32 slots, mst?  I've only setup downstream
ports within slot 0.

> qemu-commandline: -device x3130-upstream
> 
> 
> This is the upper side of a switch that can multiplex multiple devices
> onto a single port. It's only useful when one or more downstream switch
> ports are connected to it.
> 
> <model type='downstream-switch-port'/> (xio3130-downstream)
> --------------------------------------
> 
> Upstream:         connect *only* to upstream-switch-port
> Downstream:       1 slot, any PCIe device
> qemu commandline: -device xio3130-downstream
> 
> You can connect one or more of these to an upstream-switch-port in order
> to effectively plug multiple devices into a single PCIe port.
> 
> <model type='pci-bridge'/> (pci-bridge)
> --------------------------
> 
> Upstream:         PCI, connect to 1) pci-root, 2) dmi-to-pci-bridge, 3)
> another pci-bridge
> Downstream:       any PCI device, 32 slots
> qemu commandline: -device pci-bridge,...
> 
> This differs from dmi-to-pci-bridge in that its upstream connection is
> PCI rather than PCIe (so it will work on an i440FX system, which has no
> root PCIe bus) and that hotplug is supported. In general, if a guest
> will have any PCI devices, one of these controllers should be added, and
> 
> ===============================================================
> 
> 
> Comment: I'm not quite convinced that we really need the separate
> "pci-root" device. Since 1) every pci-root will *always* have either a
> pcie-root-bus or a pci-bridge connected to it, 2) the pci-root-bus will
> only ever be connected to the pci-root, and 3) the pci-bridge that
> connects to it will need special handling within the pci-bridge case
> anyway, why not:
> 
> 1) eliminate the separate pci-root controller type
> 
> 2) within <controller type='pci'>, a new <model type='pci-root-bus'/>
> will be added.
> 
> 3) a pcie-root-bus will automatically be added for q35 machinetypes, and
> pci-root-bus for any machinetype that supports a PCI bus (e.g. "pc-*")
> 
> 4) model type='pci-root-bus' will behave like pci-bridge, except that it
> will be an implicit device (nothing on qemu commandline) and it won't
> need an <address> element (neither will pcie-root-bus).

I think they should both have a domain + bus address to make it possible
to build multi-domain/multi-host bridge systems.  They do not use any
slots through.

> 5) to support multiple domains, we can simply add a "domain" attribute
> to the toplevel of controller.
> 

Or this Wouldn't even be unnecessary if we supported a 'pci-root-addr'
address type for the above with the default being domain=0, bus=0?  I
suppose it doesn't matter whether it's a separate attribute or new
address type though.  Thanks,

Alex


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]