device compatibility interface for live migration with assigned devices

[Parav Pandit]

Hi Cornelia,

> From: Cornelia Huck <cohuck at redhat.com>
> Sent: Tuesday, August 18, 2020 3:07 PM
> To: Daniel P. Berrangé <berrange at redhat.com>
> Cc: Jason Wang <jasowang at redhat.com>; Yan Zhao
> <yan.y.zhao at intel.com>; kvm at vger.kernel.org; libvir-list at redhat.com;
> qemu-devel at nongnu.org; Kirti Wankhede <kwankhede at nvidia.com>;
> eauger at redhat.com; xin-ran.wang at intel.com; corbet at lwn.net; openstack-
> discuss at lists.openstack.org; shaohe.feng at intel.com; kevin.tian at intel.com;
> Parav Pandit <parav at mellanox.com>; jian-feng.ding at intel.com;
> dgilbert at redhat.com; zhenyuw at linux.intel.com; hejie.xu at intel.com;
> bao.yumeng at zte.com.cn; Alex Williamson <alex.williamson at redhat.com>;
> eskultet at redhat.com; smooney at redhat.com; intel-gvt-
> dev at lists.freedesktop.org; Jiri Pirko <jiri at mellanox.com>;
> dinechin at redhat.com; devel at ovirt.org
> Subject: Re: device compatibility interface for live migration with assigned
> devices
> 
> On Tue, 18 Aug 2020 10:16:28 +0100
> Daniel P. Berrangé <berrange at redhat.com> wrote:
> 
> > On Tue, Aug 18, 2020 at 05:01:51PM +0800, Jason Wang wrote:
> > >    On 2020/8/18 下午4:55, Daniel P. Berrangé wrote:
> > >
> > >  On Tue, Aug 18, 2020 at 11:24:30AM +0800, Jason Wang wrote:
> > >
> > >  On 2020/8/14 下午1:16, Yan Zhao wrote:
> > >
> > >  On Thu, Aug 13, 2020 at 12:24:50PM +0800, Jason Wang wrote:
> > >
> > >  On 2020/8/10 下午3:46, Yan Zhao wrote:
> >
> > >  we actually can also retrieve the same information through sysfs,
> > > .e.g
> > >
> > >  |- [path to device]
> > >     |--- migration
> > >     |     |--- self
> > >     |     |   |---device_api
> > >     |    |   |---mdev_type
> > >     |    |   |---software_version
> > >     |    |   |---device_id
> > >     |    |   |---aggregator
> > >     |     |--- compatible
> > >     |     |   |---device_api
> > >     |    |   |---mdev_type
> > >     |    |   |---software_version
> > >     |    |   |---device_id
> > >     |    |   |---aggregator
> > >
> > >
> > >  Yes but:
> > >
> > >  - You need one file per attribute (one syscall for one attribute)
> > >  - Attribute is coupled with kobject
> 
> Is that really that bad? You have the device with an embedded kobject
> anyway, and you can just put things into an attribute group?
> 
> [Also, I think that self/compatible split in the example makes things
> needlessly complex. Shouldn't semantic versioning and matching already
> cover nearly everything? I would expect very few cases that are more
> complex than that. Maybe the aggregation stuff, but I don't think we need
> that self/compatible split for that, either.]
> 
> > >
> > >  All of above seems unnecessary.
> > >
> > >  Another point, as we discussed in another thread, it's really hard
> > > to make  sure the above API work for all types of devices and
> > > frameworks. So having a  vendor specific API looks much better.
> > >
> > >  From the POV of userspace mgmt apps doing device compat checking /
> > > migration,  we certainly do NOT want to use different vendor
> > > specific APIs. We want to  have an API that can be used / controlled in a
> standard manner across vendors.
> > >
> > >    Yes, but it could be hard. E.g vDPA will chose to use devlink (there's a
> > >    long debate on sysfs vs devlink). So if we go with sysfs, at least two
> > >    APIs needs to be supported ...
> >
> > NB, I was not questioning devlink vs sysfs directly. If devlink is
> > related to netlink, I can't say I'm enthusiastic as IMKE sysfs is
> > easier to deal with. I don't know enough about devlink to have much of an
> opinion though.
> > The key point was that I don't want the userspace APIs we need to deal
> > with to be vendor specific.
> 
> From what I've seen of devlink, it seems quite nice; but I understand why
> sysfs might be easier to deal with (especially as there's likely already a lot of
> code using it.)
> 
> I understand that some users would like devlink because it is already widely
> used for network drivers (and some others), but I don't think the majority of
> devices used with vfio are network (although certainly a lot of them are.)
> 
> >
> > What I care about is that we have a *standard* userspace API for
> > performing device compatibility checking / state migration, for use by
> > QEMU/libvirt/ OpenStack, such that we can write code without countless
> > vendor specific code paths.
> >
> > If there is vendor specific stuff on the side, that's fine as we can
> > ignore that, but the core functionality for device compat / migration
> > needs to be standardized.
> 
> To summarize:
> - choose one of sysfs or devlink
> - have a common interface, with a standardized way to add
>   vendor-specific attributes
> ?

Please refer to my previous email which has more example and details.