[vfio-users] CPU Question

[Alex Williamson]

On Fri, Sep 4, 2015 at 12:17 PM, Bradley Davis <bradleydavisjr at gmail.com>
wrote:

> It appears my last message wasn't delivered by google.
>
> What is the feature called that Xeon E5s have that others do not? I'm not
> seeing references to ACS on Intel ARC. I have an Intel Xeon X5660 and i'm
> trying to see if my processor has the feature. Maybe I just got lucky with
> my motherboard having each PCIe slot in its IOMMU group.
>

It's ACS, Access Control Services.  It's not specified on ARK.  It's a PCIe
capability that let's the OS configure whether the PCIe root port is
allowed to do things like redirect a transaction before being translated by
the IOMMU or whether the root port is required to enforce source validation
to prevent a malicious device from spoofing itself as another.  For root
ports, the capability generally needs to be configurable, it's not really
enough to have a read-only implementation, so don't jump too quickly if you
see it there, the right set of control bits need to be enabled.

The Xeon 5600 series (Westmere) is relatively old, I don't think that E5
class existed back then, but it's similar to my W3520 (Nehalem), which does
support ACS on the processor root ports.  You probably see something like
this in lspci for those root ports:

        Capabilities: [150 v1] Access Control Services
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+
UpstreamFwd+ EgressCtrl- DirectTrans-
                ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+
UpstreamFwd+ EgressCtrl- DirectTrans-

Those '+' marks in the ACSCtl line are important, see the PCIe spec for a
description of each.  You'll also notice that the PCH (well, in this case
ICH) root ports do not support ACS.  We also have no kernel quirks that
enable "ACS equivalent isolation" for ICH root ports, and I don't expect to
be getting any for such an old system.  So yes, your system likely supports
ACS, but you are also getting lucky that you're plugging the cards to be
assigned into the processor root ports rather than the ICH root ports.
Most people need to do the opposite, install the assigned devices into PCH
root ports, which have quirks to enable the ACS equivalent isolation, while
the processor root ports do not have native ACS support and Intel has no
plans to tell us how to implement ACS equivalent isolation via quirks for
those root ports.

Xeon E5 systems should have native ACS support for the processor root ports
and quirks for the PCH root ports (hopefully native support there
eventually too), so we should at least have a good foundation for isolation
on those systems.  Of course if you read my blog post on IOMMU groups, you
know that isolation needs to be end-to-end, so switches and bridges
downstream can also break isolation, but the further downstream the
problem, the smaller the IOMMU group.  Thanks,

Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20150904/74f88ee2/attachment.htm>