[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [libvirt] memory pinning problem



Am Freitag, 6. September 2013, 12:10:04 schrieb Daniel P. Berrange:
> On Tue, Aug 27, 2013 at 09:09:25AM +0200, David Weber wrote:
> > Hi,
> > 
> > we try to use vcpu pinning on a 2 socket server with Intel Xeon E5620
> > cpus, HT enabled and 2*6*16GiB Ram but experience problems if we try to
> > start a guest on the second socket:
> > error: Failed to start domain test
> > error: internal error: process exited while connecting to monitor:
> > kvm_init_vcpu failed: Cannot allocate memory
> > 
> > Libvirt version 1.1.1
> > Linux 3.11-rc7
> > 
> > Because I coudn't find any other service which allowed a 7M file upload, I
> > put the log file and everything else which could perhabs be relevant into
> > a github repository:
> > https://github.com/David-Weber/vcpu-pinning
> > 
> > When we try to start a guest on the first node it runs fine:
> >   <vcpu placement='static' cpuset='0-3,8-11'>4</vcpu>
> >   <numatune>
> >   
> >     <memory mode='strict' nodeset='0'/>
> >   
> >   </numatune>
> > 
> > Starting it on the second node fails
> > 
> >   <vcpu placement='static' cpuset='4-7,12-15'>4</vcpu>
> >   <numatune>
> >   
> >     <memory mode='strict' nodeset='1'/>
> >   
> >   </numatune>
> > 
> > Even more strange, starting it with the CPUs of the second node and the
> > memory> 
> > of the first node works:
> >   <vcpu placement='static' cpuset='4-7,12-15'>4</vcpu>
> >   <numatune>
> >   
> >     <memory mode='strict' nodeset='0'/>
> >   
> >   </numatune>
> > 
> > The log file contains these three cases.
> > 
> > Using the placement='auto' parameter leads to the same problem. If numad
> > return the second node, the guest won't start.
> > 
> > Is this a configuration, a libvirt or a cgroup problem? :)
> 
> With mode='strict' you are telling QEMU that if it can't allocate
> memory from the requested node, it should fail. Is it possible
> that some of your Numa nodes have insufficient memory free ?
> 
> The combination of 'virsh capabilities' output and the results
> of 'virsh freecell NODENUM' for each NUMA node will give an
> indication of the allocation state.
> 
> 
> Daniel


Thank you for your response. 
There should be plenty of free memory be available.

# virsh capabilities
<capabilities>

  <host>
    <uuid>91ecf0d9-c821-c9ca-c7fa-00259064e5c6</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Westmere</model>
      <vendor>Intel</vendor>
      <topology sockets='1' cores='4' threads='2'/>
      <feature name='rdtscp'/>
      <feature name='pdpe1gb'/>
      <feature name='dca'/>
      <feature name='pcid'/>
      <feature name='pdcm'/>
      <feature name='xtpr'/>
      <feature name='tm2'/>
      <feature name='est'/>
      <feature name='smx'/>
      <feature name='vmx'/>
      <feature name='ds_cpl'/>
      <feature name='monitor'/>
      <feature name='dtes64'/>
      <feature name='pclmuldq'/>
      <feature name='pbe'/>
      <feature name='tm'/>
      <feature name='ht'/>
      <feature name='ss'/>
      <feature name='acpi'/>
      <feature name='ds'/>
      <feature name='vme'/>
    </cpu>
    <power_management>
      <suspend_mem/>
      <suspend_disk/>
      <suspend_hybrid/>
    </power_management>
    <migration_features>
      <live/>
      <uri_transports>
        <uri_transport>tcp</uri_transport>
      </uri_transports>
    </migration_features>
    <topology>
      <cells num='2'>
        <cell id='0'>
          <memory unit='KiB'>99003008</memory>
          <cpus num='8'>
            <cpu id='0' socket_id='0' core_id='0' siblings='0,8'/>
            <cpu id='1' socket_id='0' core_id='1' siblings='1,9'/>
            <cpu id='2' socket_id='0' core_id='9' siblings='2,10'/>
            <cpu id='3' socket_id='0' core_id='10' siblings='3,11'/>
            <cpu id='8' socket_id='0' core_id='0' siblings='0,8'/>
            <cpu id='9' socket_id='0' core_id='1' siblings='1,9'/>
            <cpu id='10' socket_id='0' core_id='9' siblings='2,10'/>
            <cpu id='11' socket_id='0' core_id='10' siblings='3,11'/>
          </cpus>
        </cell>
        <cell id='1'>
          <memory unit='KiB'>99088544</memory>
          <cpus num='8'>
            <cpu id='4' socket_id='1' core_id='0' siblings='4,12'/>
            <cpu id='5' socket_id='1' core_id='1' siblings='5,13'/>
            <cpu id='6' socket_id='1' core_id='9' siblings='6,14'/>
            <cpu id='7' socket_id='1' core_id='10' siblings='7,15'/>
            <cpu id='12' socket_id='1' core_id='0' siblings='4,12'/>
            <cpu id='13' socket_id='1' core_id='1' siblings='5,13'/>
            <cpu id='14' socket_id='1' core_id='9' siblings='6,14'/>
            <cpu id='15' socket_id='1' core_id='10' siblings='7,15'/>
          </cpus>
        </cell>
      </cells>
    </topology>
    <secmodel>
      <model>none</model>
      <doi>0</doi>
    </secmodel>
    <secmodel>
      <model>dac</model>
      <doi>0</doi>
    </secmodel>
  </host>

  <guest>
    <os_type>hvm</os_type>
    <arch name='i686'>
      <wordsize>32</wordsize>
      <emulator>/usr/bin/qemu-system-x86_64</emulator>
      <machine canonical='pc-i440fx-1.5' maxCpus='255'>pc</machine>
      <machine maxCpus='255'>pc-q35-1.4</machine>
      <machine canonical='pc-q35-1.5' maxCpus='255'>q35</machine>
      <machine maxCpus='1'>isapc</machine>
      <machine maxCpus='255'>pc-0.10</machine>
      <machine maxCpus='255'>pc-0.11</machine>
      <machine maxCpus='255'>pc-0.12</machine>
      <machine maxCpus='255'>pc-0.13</machine>
      <machine maxCpus='255'>pc-0.14</machine>
      <machine maxCpus='255'>pc-0.15</machine>
      <machine maxCpus='255'>pc-1.0</machine>
      <machine maxCpus='255'>pc-1.1</machine>
      <machine maxCpus='255'>pc-1.2</machine>
      <machine maxCpus='255'>pc-1.3</machine>
      <machine maxCpus='255'>pc-i440fx-1.4</machine>
      <machine maxCpus='1'>none</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/bin/qemu-kvm</emulator>
        <machine canonical='pc-i440fx-1.5' maxCpus='255'>pc</machine>
        <machine maxCpus='255'>pc-q35-1.4</machine>
        <machine canonical='pc-q35-1.5' maxCpus='255'>q35</machine>
        <machine maxCpus='1'>isapc</machine>
        <machine maxCpus='255'>pc-0.10</machine>
        <machine maxCpus='255'>pc-0.11</machine>
        <machine maxCpus='255'>pc-0.12</machine>
        <machine maxCpus='255'>pc-0.13</machine>
        <machine maxCpus='255'>pc-0.14</machine>
        <machine maxCpus='255'>pc-0.15</machine>
        <machine maxCpus='255'>pc-1.0</machine>
        <machine maxCpus='255'>pc-1.1</machine>
        <machine maxCpus='255'>pc-1.2</machine>
        <machine maxCpus='255'>pc-1.3</machine>
        <machine maxCpus='255'>pc-i440fx-1.4</machine>
        <machine maxCpus='1'>none</machine>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
      <pae/>
      <nonpae/>
    </features>
  </guest>

  <guest>
    <os_type>hvm</os_type>
    <arch name='x86_64'>
      <wordsize>64</wordsize>
      <emulator>/usr/bin/qemu-system-x86_64</emulator>
      <machine canonical='pc-i440fx-1.5' maxCpus='255'>pc</machine>
      <machine maxCpus='255'>pc-q35-1.4</machine>
      <machine canonical='pc-q35-1.5' maxCpus='255'>q35</machine>
      <machine maxCpus='1'>isapc</machine>
      <machine maxCpus='255'>pc-0.10</machine>
      <machine maxCpus='255'>pc-0.11</machine>
      <machine maxCpus='255'>pc-0.12</machine>
      <machine maxCpus='255'>pc-0.13</machine>
      <machine maxCpus='255'>pc-0.14</machine>
      <machine maxCpus='255'>pc-0.15</machine>
      <machine maxCpus='255'>pc-1.0</machine>
      <machine maxCpus='255'>pc-1.1</machine>
      <machine maxCpus='255'>pc-1.2</machine>
      <machine maxCpus='255'>pc-1.3</machine>
      <machine maxCpus='255'>pc-i440fx-1.4</machine>
      <machine maxCpus='1'>none</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/bin/qemu-kvm</emulator>
        <machine canonical='pc-i440fx-1.5' maxCpus='255'>pc</machine>
        <machine maxCpus='255'>pc-q35-1.4</machine>
        <machine canonical='pc-q35-1.5' maxCpus='255'>q35</machine>
        <machine maxCpus='1'>isapc</machine>
        <machine maxCpus='255'>pc-0.10</machine>
        <machine maxCpus='255'>pc-0.11</machine>
        <machine maxCpus='255'>pc-0.12</machine>
        <machine maxCpus='255'>pc-0.13</machine>
        <machine maxCpus='255'>pc-0.14</machine>
        <machine maxCpus='255'>pc-0.15</machine>
        <machine maxCpus='255'>pc-1.0</machine>
        <machine maxCpus='255'>pc-1.1</machine>
        <machine maxCpus='255'>pc-1.2</machine>
        <machine maxCpus='255'>pc-1.3</machine>
        <machine maxCpus='255'>pc-i440fx-1.4</machine>
        <machine maxCpus='1'>none</machine>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
    </features>
  </guest>

</capabilities>

# virsh freecell 0
0: 86071624 KiB

# virsh freecell 1
1: 75258628 KiB

# virsh edit test
<domain type='kvm'>
  <name>test</name>
  <uuid>08cdc389-78bf-450c-89f4-b4728edabdbf</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static' cpuset='4-7'>1</vcpu>
  <numatune>
    <memory mode='strict' nodeset='1'/>
  </numatune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-1.5'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x1'/>
    </controller>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' 
function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' 
function='0x0'/>
    </memballoon>
  </devices>
</domain>

 # virsh start test
error: Failed to start domain test
error: internal error: process exited while connecting to monitor: 
kvm_init_vcpu failed: Cannot allocate memory

Allocating memory on this node with numactl  works fine
# numactl --cpubind=1 --membind=1 --  dd if=/dev/zero of=/dev/null bs=2G 
count=1
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB) copied, 0.60816 s, 3.5 GB/s

David


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]