[libvirt] [RFC][scale] new API for querying domains stats

[Francesco Romani]

----- Original Message -----
> From: "Daniel P. Berrange" <berrange at redhat.com>
> To: "Francesco Romani" <fromani at redhat.com>
> Cc: libvir-list at redhat.com
> Sent: Tuesday, July 1, 2014 10:35:21 AM
> Subject: Re: [libvirt] [RFC][scale] new API for querying domains stats
> 

[...] 
> > We [in VDSM] currently use these APIs for our sempling:
> >   virDomainBlockInfo
> >   virDomainGetInfo
> >   virDomainGetCPUStats
> >   virDomainBlockStats
> >   virDomainBlockStatsFlags
> >   virDomainInterfaceStats
> >   virDomainGetVcpusFlags
> >   virDomainGetMetadata
> 
> Why do you need to call virDomainGetMetadata so often ? That merely contains
> a opaque data blob that can only have come from VDSM itself, so I'm surprised
> you need to call that at all frequently.

We store some QoS info in the domain metadata. Actually we can elide this API call
from the list and fix our coude to make smarter use of it.

> >   please note that we are much more concerned about thread reduction then
> >   about performance numbers. We had report of thread number becoming a
> >   real harm, while performance so far is not yet a concern
> >   (https://bugzilla.redhat.com/show_bug.cgi?id=1102147#c54)
> > 
> > * bulk APIs for querying domain stats
> > (https://bugzilla.redhat.com/show_bug.cgi?id=1113116)
> >   would be really welcome as well. It is quite independent from the
> >   previous bullet point
> >   and would help us greatly with scale.
> 
> If we did the first bullet point, we'd be adding another ~10 APIs for
> async variants. If we then did the second bullet point we'd be adding
> another ~10 APIs for bulk querying. So while you're right that they
> are independent, it would be desirable to address them both at the
> same time, so we only need to add 10 new APIs in total, not 20.

I'm fine with this approach.


> For the async API design, I could see two potential designs
> 
> 1. A custom callback to run per API
> 
>      typedef (void)(*virDomainBlockInfoCallback)(virDomainPtr dom,
>                                                  bool isError,
>                                                  virDomainBlockInfoPtr info,
>                                                  void *opaque);
> 
>     int virDomainGetBlockInfoAsync(virDomainPtr dom,
>                                    const char *disk,
>                                    virDomainBlockInfoCallback cb,
>                                    void *opaque,
>                                    unsigned int flags);
> 
> 
> 2. A standard callback and a pair of APIs
> 
>      typedef void *virDomainAsyncResult;
>      typedef (void)(*virDomainAsyncCallback)(virDomainPtr dom,
>                                              virDomainAsyncResult res);
> 
>    void virDomainGetBlockInfoAsync(virDomainPtr dom,
>                                    const char *disk,
>                                    virDomainBlockInfoCallback cb,
>                                    void *opaque,
>                                    unsigned int flags);
>    int virDomainGetBlockInfoFinish(virDomainPtr dom,
>                                   virDomainAsyncResult res,
>                                   virDomainBlockInfoPtr info);
> 
> This second approach is the way GIO works (see example in this page
> https://developer.gnome.org/gio/stable/GAsyncResult.html ). The main
> difference between them really is probably the way you get error
> reporting from the APIs. In the first example, libvirt would raise
> an error before it invoked the callback, with isError set to True.
> In the second example, the Finish() func would raise the error and
> return -1.

I need to check in deeper detail and sync up with other VDSM developers,
but I have a feel that the first approach is a bit easier for VDSM to consume.

Bests,

-- 
Francesco Romani
RedHat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani