[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [libvirt] [PATCHv10 0/4] Introduce x86 Cache Monitoring Technology (CMT)



Hi John,

The issue mentioned is generated by a -1 returned by qemuDomainGetIOThreadsMon. 

Further, it is caused by a -1 returned by virQEMUCapsGet:
 6174         if (!virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_OBJECT_IOTHREAD)) {
 6175             virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s",
 6176                            _("IOThreads not supported with this binary"));
 6177             goto endjob;
 6178         }

By checking qemuCaps->flags, it seems the bit of QEMU_CAPS_OBJECT_IOTHREAD(176) is not set. 
(gdb) call virBitmapFormat(qemuCaps->flags)
$4 = 0x7f08a4004cb0 "13,33,46,50,54-55,58,61-64,66-70,72-73,77-78,80,87-88,92-93,95-97,99,102-110,112,114,119-121,128-131,141-142,146,148-149,151-156,158-161,165,167,174-175,180,182,188,193-198,210,214,221-222,225,250,255"...

Breakpoint 1 at 0x7f088ef00880: file qemu/qemu_driver.c, line 5492.
(gdb) c
Continuing.
[Switching to Thread 0x7f08bcdcb700 (LWP 24711)]

Breakpoint 1, qemuDomainGetIOThreadsMon (driver=0x7f0874049b80, vm=0x7f08740cd840, iothreads=0x7f08bcdca8b8) at qemu/qemu_driver.c:5492
5492    {
Unprocessed breakpoint [1]
(gdb) finish
Run till exit from #0  qemuDomainGetIOThreadsMon (driver=0x7f0874049b80, vm=0x7f08740cd840, iothreads=0x7f08bcdca8b8) at qemu/qemu_driver.c:5492
0x00007f088ef0098f in qemuDomainGetStatsIOThread (driver=<optimized out>, dom=<optimized out>, record=0x7f08ac000b60, maxparams=0x7f08bcdca9e4, privflags=<optimized out>) at qemu/qemu_driver.c:20885
20885       if ((niothreads = qemuDomainGetIOThreadsMon(driver, dom, &iothreads)) < 0)
Value returned is $1 = -1
(gdb) bt
#0  0x00007f088ef0098f in qemuDomainGetStatsIOThread (driver=<optimized out>, dom=<optimized out>, record=0x7f08ac000b60, maxparams=0x7f08bcdca9e4, privflags=<optimized out>) at qemu/qemu_driver.c:20885
#1  0x00007f088eee70a9 in qemuDomainGetStats (flags=1, record=<synthetic pointer>, stats=255, dom=0x7f08740cd840, conn=0x7f08ac000c50) at qemu/qemu_driver.c:21062
#2  qemuConnectGetAllDomainStats (conn=0x7f08ac000c50, doms=<optimized out>, ndoms=<optimized out>, stats=255, retStats=0x7f08bcdcab10, flags=<optimized out>) at qemu/qemu_driver.c:21162
#3  0x00007f08cd147646 in virConnectGetAllDomainStats (conn=0x7f08ac000c50, stats=0, retStats=retStats entry=0x7f08bcdcab10, flags=0) at libvirt-domain.c:11685
#4  0x0000562e2404d2c0 in remoteDispatchConnectGetAllDomainStats (server=0x562e24ee74a0, msg=<optimized out>, ret=0x7f08ac001ec0, args=0x7f08ac001f30, rerr=0x7f08bcdcac50, client=0x562e24ef5200) at remote/remote_daemon_dispatch.c:6665
#5  remoteDispatchConnectGetAllDomainStatsHelper (server=0x562e24ee74a0, client=0x562e24ef5200, msg=<optimized out>, rerr=0x7f08bcdcac50, args=0x7f08ac001f30, ret=0x7f08ac001ec0) at remote/remote_daemon_dispatch_stubs.h:743
#6  0x00007f08cd05dad5 in virNetServerProgramDispatchCall (msg=0x562e24ef5910, client=0x562e24ef5200, server=0x562e24ee74a0, prog=0x562e24ee6650) at rpc/virnetserverprogram.c:437
#7  virNetServerProgramDispatch (prog=0x562e24ee6650, server=server entry=0x562e24ee74a0, client=0x562e24ef5200, msg=0x562e24ef5910) at rpc/virnetserverprogram.c:304
#8  0x00007f08cd063f7d in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x562e24ee74a0) at rpc/virnetserver.c:144
#9  virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x562e24ee74a0) at rpc/virnetserver.c:165
#10 0x00007f08ccf95171 in virThreadPoolWorker (opaque=opaque entry=0x562e24ee69a0) at util/virthreadpool.c:167
#11 0x00007f08ccf944f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#12 0x00007f08cb213dc5 in start_thread (arg=0x7f08bcdcb700) at pthread_create.c:308
#13 0x00007f08cab3b76d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb)

My domain configuration is:
[root dl-c200 libvirt]# virsh list
v^H Id   Name               State
----------------------------------
 1    ubuntu16.04-base   running

[root dl-c200 libvirt]# virsh dumpxml 1
<domain type='kvm' id='1'>
  <name>ubuntu16.04-base</name>
  <uuid>19d01bd1-ad54-4176-84bd-e510de08fe00</uuid>
  <memory unit='KiB'>10485760</memory>
  <currentMemory unit='KiB'>10485760</currentMemory>
  <vcpu placement='static' current='4'>8</vcpu>
  <cputune>
    <cachetune vcpus='0' id='vcpus_0'>
      <monitor level='3' vcpus='0'/>
    </cachetune>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Broadwell</model>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='xsaveopt'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/tmp/kvm-donot-delete/vm-ubuntu16.04-base'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:eb:24:d3'/>
      <source bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5901' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c157,c912</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c157,c912</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+0</label>
    <imagelabel>+0:+0</imagelabel>
  </seclabel>
</domain>

I wonder if these message is enough for you to locate the root cause, I don’t have too much time in investigating this today, I'll try to create a fix tomorrow if you need.

Huaqiang

> -----Original Message-----
> From: Wang, Huaqiang
> Sent: Tuesday, November 27, 2018 11:21 AM
> To: 'John Ferlan' <jferlan redhat com>; libvir-list redhat com
> Subject: RE: [PATCHv10 0/4] Introduce x86 Cache Monitoring Technology
> (CMT)
> 
> 
> 
> > -----Original Message-----
> > From: John Ferlan [mailto:jferlan redhat com]
> > Sent: Tuesday, November 27, 2018 10:59 AM
> > To: Wang, Huaqiang <huaqiang wang intel com>; libvir-list redhat com
> > Subject: Re: [PATCHv10 0/4] Introduce x86 Cache Monitoring Technology
> > (CMT)
> >
> >
> >
> > On 11/26/18 9:39 PM, Wang, Huaqiang wrote:
> > > Hi John,
> > >
> > > Really appreciate your hard work for the CMT series. Next I'll
> > > working on
> > the MBM.
> > >
> > > In testing the newly pushed code, I find a problem:
> > >
> > > <error message>
> > > [david dl-c200 ~]$ sudo virsh domstats
> > > error: An error occurred, but the cause is unknown </error message>
> > >
> >
> > I couldn't reproduced in a quick test here. Can you get a thread trace
> > of the failure?
> >
> > What I usually do, build libvirt, then in a terminal session at the
> > top of the git tree "./run gdb src/libvirtd"  (dbg> r)... THen in
> > another terminal session run the virsh command and when the libvirtd
> session stops do a "t a a bt"
> > (thread apply all backtrace)...
> >
> > John
> 
> I'll trace the error.
> 
> Thanks.
> Huaqiang
> 
> >
> > (done for the night)
> >
> > > seems it is caused by qemuDomainGetStatsIOThread not by the new
> > series.
> > > What I test is return 0 immediately at top of
> > > qemuDomainGetStatsIOThread, the command 'virsh domstats' reports
> > the cache statistics normally.
> > >
> > > BR
> > > Huaqiang
> > >
> > >> -----Original Message-----
> > >> From: John Ferlan [mailto:jferlan redhat com]
> > >> Sent: Tuesday, November 27, 2018 9:49 AM
> > >> To: Wang, Huaqiang <huaqiang wang intel com>; libvir-
> > list redhat com
> > >> Subject: Re: [PATCHv10 0/4] Introduce x86 Cache Monitoring
> > Technology
> > >> (CMT)
> > >>
> > >>
> > >>
> > >> On 11/26/18 12:56 PM, Wang Huaqiang wrote:
> > >>> These patches are the remaining part for the CMT enabling series,
> > >>> and most of the series have been merged.
> > >>>
> > >>> This series is addressing John's review comments and suggestions,
> > >>> which are
> > >>> https://www.redhat.com/archives/libvir-list/2018-
> > >> November/msg00907.htm
> > >>> l
> > >>> https://www.redhat.com/archives/libvir-list/2018-
> > >> November/msg00510.htm
> > >>> l
> > >>> https://www.redhat.com/archives/libvir-list/2018-
> > >> November/msg00561.htm
> > >>> l
> > >>>
> > >>> Change lists:
> > >>> Changes in v10:
> > >>> -. Add tag (virResctrlMonitorType) in
> qemuDomainGetResctrlMonData,
> > >> thus
> > >>>    qemuDomainGetResctrlMonData could be reused for MBM.
> > >>> -. Using VIR_APPEND_ELEMENT to append
> virQEMUResctrlMonDataPtr
> > >> list.
> > >>> -. Add qemuDomainFreeResctrlMonData.
> > >>> -. Add virResctrlMonitorFreeStats.
> > >>> -. Return a list of virResctrlMonitorStatsPtr instead of
> > >>>    a virResctrlMonitorStats array in virResctrlMonitorGetStats.
> > >>>
> > >>> Changes in V9:
> > >>> -. Addressing code review comments form John.
> > >>> -. Refined the names for new data structure and new functions.
> > >>> -. Merged qemuDomainGetStatsCpuResMonitorPerTag and
> > >> qemuDomainGetStatsCpuResMonitor,
> > >>>    and refined new function name based on the fact that we only
> > >>> support
> > >> cache monitor now.
> > >>> Wang Huaqiang (4):
> > >>>   util: Return a list of pointer in virResctrlMonitorGetStats
> > >>>   util: Add function to free monitor statistical data
> > >>>   qemu: Report cache occupancy (CMT) with domstats
> > >>>   docs: Updated news.xml for CMT
> > >>>
> > >>>  docs/news.xml            |  12 ++++
> > >>>  src/libvirt-domain.c     |  12 ++++
> > >>>  src/libvirt_private.syms |   1 +
> > >>>  src/qemu/qemu_driver.c   | 183
> > >> ++++++++++++++++++++++++++++++++++++++++++++++-
> > >>>  src/util/virresctrl.c    |  26 +++++--
> > >>>  src/util/virresctrl.h    |   8 ++-
> > >>>  tools/virsh.pod          |  14 ++++
> > >>>  7 files changed, 248 insertions(+), 8 deletions(-)
> > >>>
> > >>
> > >> Reviewed-by: John Ferlan <jferlan redhat com>
> > >> (series)
> > >>
> > >> and pushed,
> > >>
> > >> John


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]