[Libvirt-cim] Query regarding Xen_ComputerSystem.RequestStateChange() and Live migration
Kaitlin Rupert
kaitlin at linux.vnet.ibm.com
Fri Oct 9 16:26:04 UTC 2009
Subba Rao, Sandeep M (STSD) wrote:
> Hi Kaitlin,
>
>> I'm not sure what you mean by "goes back to host1".
>
> So, if we switch off the VM after the live migration, the VM loses the assocation with target. Virsh shows as follows after shutdown:
>
> [root at RHEL54-host1 ~]# virsh list --all
>> Id Name State
>> ----------------------------------
>> 0 Domain-0 running
>> - liveVm shut off
>
>
>> [root at RHEL54-host2 ~]# virsh list --all
>> Id Name State
>> ----------------------------------
>> 0 Domain-0 running
>> - localVm shut off
Ah, I see what you mean. libvirt does not define the guest on the
target during a migration. So before you shutdown the guest, you'll
need to define it so that the config persists.
That said, libvirt-cim will define the guest so that when the guest is
migrated over, it's configuration persists after the guest is shutdown.
>
>> Ah, so libvirt-cim is cleaning the guest up on the source host properly.
> That is good to hear.
>
> libvirt-cim is cleaning the guest up on the source host properly, like you said looks like libvirt has some issues and we will follow up accordingly.
>
> Regarding the stop VM operation -- RequestStateChange() function.
>
> We tried the stop operation on the VM. The stop operation through libvirt-cim reported success i.e the RequestStateChange() function returned a 0, but
> the VM is still running on the host.
>
> Attaching the libvirt-cim log file.
>
> The operation was tried on the VM 'Win'. Details are as below:
>
> [root at RHEL54 ~]# cat /etc/issue
> Red Hat Enterprise Linux Server release 5.4 Beta (Tikanga)
> Kernel \r on an \m
>
> [root at RHEL54 ~]# virsh list --all
> Id Name State
> ----------------------------------
> 0 Domain-0 running
> 1 Win idle
> - HVM shut off
> - SLES shut off
> - VM shut off
>
> virsh # dominfo 1
> Id: 1
> Name: Win
> UUID: 090df819-8ac6-f4d3-784b-4309a938080a
> OS Type: hvm
> State: no state
> CPU(s): 1
> CPU time: 55.0s
> Max memory: 278528 kB
> Used memory: 270208 kB
> Autostart: disable
>
> The stop operation doesn't seem to work through xm and virsh. However, our concern is that the RequestStateChange() function returns 0 immediately.
> This is the case even when the operation succeeds.
So if xm is failing to shutdown the guest, then libvirt-cim won't be
able to shutdown the guest either.
However, I agree with you that there is a bug in libvirt-cim here. We
should check to make sure the guest is properly shutdown before
returning from RequestStateChange(). We don't currently do that.
Currently, we just return whether libvirt reports a success or a failure.
>
> Please let us know if you need any information.
>
> Regards,
> Sandeep
>
> -----Original Message-----
> From: Kaitlin Rupert [mailto:kaitlin at linux.vnet.ibm.com]
> Sent: Friday, October 09, 2009 1:41 AM
> To: Velayudham, Madan Ganesh (STSD / ESS SW)
> Cc: Deepti B Kalakeri; List for discussion and development of libvirt CIM; Subba Rao, Sandeep M (STSD)
> Subject: Re: [Libvirt-cim] Query regarding Xen_ComputerSystem.RequestStateChange() and Live migration
>
> Velayudham, Madan Ganesh (STSD / ESS SW) wrote:
>> Thanks Kaitlin,
>>
>> Here it is:
>>
>> [root at RHEL54-host1 ~]# virsh list --all
>> Id Name State
>> ----------------------------------
>> 0 Domain-0 running
>> - liveVm shut off
>
> This is why libvirt-cim reports the guest on host1. The definition of the guest still exists here.
>
>> [root at RHEL54-host2 ~]# virsh list --all
>> Id Name State
>> ----------------------------------
>> 0 Domain-0 running
>> 14 liveVm idle
>> - localVm shut off
>>
>> If I shutdown the liveVm on host2, the VM goes back to host1!! Does it indicate any problems?
>
> I'm not sure what you mean by "goes back to host1".
>
>> Please note that above mentioned was the case when we invoked the live migration through virsh command.
>
> My mistake - I didn't realize you were launching the migration through libvirt. So it sounds like libvirt doesn't remove the definition from the source system in RHEL 5.4. I believe this is the desired behavior, but you'll have to check with the libvirt team.
>
>> I enabled the logging as mentioned, and invoked live migration through libvirt-CIM - in which case - the migration completed successfully, and the VM was reported only on the target; Attached is the log.
>
> Ah, so libvirt-cim is cleaning the guest up on the source host properly.
> That is good to hear.
>
>> Now it appears to be - if we start using virsh command, then the inconsistency is seen.
>>
>> Please let us know your comments.
>>
>> We'll capture logs for the stop issue (in another setup) and send soon.
>>
>> Thanks and Regards,
>> Madan (_MaGa_)
>> Madan Ganesh Velayudham
>> Office: +91 80 2516 5333
>>
>>
>> -----Original Message-----
>> From: Kaitlin Rupert [mailto:kaitlin at linux.vnet.ibm.com]
>> Sent: Thursday, October 08, 2009 9:48 PM
>> To: Velayudham, Madan Ganesh (STSD / ESS SW)
>> Cc: Deepti B Kalakeri; List for discussion and development of libvirt CIM; Subba Rao, Sandeep M (STSD)
>> Subject: Re: [Libvirt-cim] Query regarding Xen_ComputerSystem.RequestStateChange() and Live migration
>>
>> Velayudham, Madan Ganesh (STSD / ESS SW) wrote:
>>> Thank you Kaitlin and Deepti,
>>>
>> Hi Madan,
>>
>> Can you run "virsh list --all" on both hosts. "virsh list" just shows
>> active guests, while "virsh list --all" shows active and inactive guests.
>>
>> Mu guess is that liveVM is still defined on RHEL54-host1. Can you
>> enable the provider debug and then rerun the migration? I'm wondering
>> if the providers print an error as to why the guest isn't being
>> undefined from the source system.
>>
>> To enable the provider debug, do the following:
>>
>> export CU_DEBUG="/var/log/libvirt-cim.log"
>> /etc/init.d/tog-pegasus restart
>>
>>> The VM "liveVm" on shared storage is being reported on both the hosts (source and destination), here is what we receive through "Xen_HostedDepenency" association:
>>>
>>> [root at RHEL54-host1 ~]# wbemcli ain -ac Xen_HostedDependency 'http://root:password@localhost:5988/root/virt:Xen_HostSystem.CreationClassName="Xen_HostSystem",Name="RHEL54-host1"'
>>> RHEL54-host1/root/virt:Xen_ComputerSystem.CreationClassName="Xen_ComputerSystem",Name="liveVm"
>>> RHEL54-host1/root/virt:Xen_ComputerSystem.CreationClassName="Xen_ComputerSystem",Name="Domain-0"
>>>
>>> [root at RHEL54-host1 ~]# virsh list
>>> Id Name State
>>> ----------------------------------
>>> 0 Domain-0 running
>>>
>>> [root at RHEL54-host2 ~]# wbemcli ain -ac Xen_HostedDependency 'http://root:password@localhost:5988/root/virt:Xen_HostSystem.CreationClassName="Xen_HostSystem",Name="RHEL54-host2"'
>>> RHEL54-host2/root/virt:Xen_ComputerSystem.CreationClassName="Xen_ComputerSystem",Name="localVm"
>>> RHEL54-host2/root/virt:Xen_ComputerSystem.CreationClassName="Xen_ComputerSystem",Name="Domain-0"
>>> RHEL54-host2/root/virt:Xen_ComputerSystem.CreationClassName="Xen_ComputerSystem",Name="liveVm"
>>>
>>> [root at RHEL54-host2 ~]# virsh list
>>> Id Name State
>>> ----------------------------------
>>> 0 Domain-0 running
>>> 7 liveVm idle
>>>
>>> As you notice, virsh command is listing properly, but the association is not updated properly and results in both hosts.
>>>
>>> On the RequestStateChange, we did not capture the message of stop failure. However, here is what we observe - we invoke the RequestStateChange method, it returns 0 - indicating success;
>>> But the VM will not start or stop due to insufficient resources. We assuming that RequestStatechange method to be synchronous call if it does not
>>> provide CIM_Job to track the status of start or stop. Generally starting a Windows VM take more than a minute.
>> Where is the insufficient resources printing out from? Can you paste
>> the errors you see? Can you also include the accompanying provider debug?
>>
>> Either way, libvirt-cim should not return a success in this case. So it
>> sounds like we have a bug here. My guess is that the following is
>> happening:
>>
>> 1) libvirt-cim asks libvirt to stop the guest
>> 2) libvirt returns that the operation is successful, but the operation
>> really fails
>> 3) libvirt-cim doesn't verify that the guest was really stopped /
>> started, so it doesn't catch the failure
>>
>>> Thanks again for your help,
>>>
>>> Best Regards,
>>> Madan
>>>
>>> -----Original Message-----
>>> From: Deepti B Kalakeri [mailto:deeptik at linux.vnet.ibm.com]
>>> Sent: Thursday, October 08, 2009 1:13 PM
>>> To: Velayudham, Madan Ganesh (STSD / ESS SW)
>>> Cc: List for discussion and development of libvirt CIM; Kaitlin Rupert; Deepti B Kalakeri
>>> Subject: Re: [Libvirt-cim] Query regarding Xen_ComputerSystem.RequestStateChange() and Live migration
>>>
>>>
>>>
>>> Subba Rao, Sandeep M (STSD) wrote:
>>>> Hi Kaitlin,
>>>>
>>>> Hope you are doing fine.
>>>>
>>>> We need a clarification regarding the CIM_ConcereteJob support part of Xen_ComputerSystem.RequestStateChange() method to start or stop a Virtual Machine. After we invoke the method, we were expecting a return value of 4096 (Operation Scheduled) with the reference to Job to track the status.
>>>>
>>> AFAIK, the method would not return 4096, it would return success or
>>> failure only.
>>>> Instead we are noticing that the method returns 0 - indicating success, however the Virtual machine does not stop because of memory or other errors.
>>>>
>>> Can you elaborate on what operation you are doing here ?
>>> What errors are you seeing ?
>>>> Can you clarify whether this is a known issue in libvirt-CIM? Also recommend how to track the invoked job?
>>>>
>>> Once the RqquestedStateChange() is successful you should be able to see
>>> the new state value in the Xen_ComputerSystem.EnabledState.
>>>> The OS is an RHEL 5.4 and the libvirt and xen versions are as follows:
>>>>
>>>> [root at RHEL54 ~]# cat /etc/issue
>>>> Red Hat Enterprise Linux Server release 5.4 Beta (Tikanga)
>>>> Kernel \r on an \m
>>>>
>>>> [root at RHEL54 ~]# uname -a
>>>> Linux RHEL54 2.6.18-160.el5xen #1 SMP Mon Jul 27 18:01:58 EDT 2009 i686 i686 i386 GNU/Linux
>>>> [root at RHEL54 ~]# rpm -qa | grep libvirt
>>>> libvirt-0.6.3-17.el5
>>>> libvirt-cim-0.5.5-2.el5
>>>> libvirt-python-0.6.3-17.el5
>>>> [root at RHEL54 ~]# rpm -qa | grep xen
>>>> kernel-xen-devel-2.6.18-160.el5
>>>> kernel-xen-2.6.18-160.el5
>>>> xen-3.0.3-92.el5
>>>> xen-libs-3.0.3-92.el5
>>>> [root at RHEL54 ~]# rpm -qa | grep sblim
>>>> sblim-cmpi-base-1.5.5-34.el5
>>>> [root at RHEL54 ~]#
>>>>
>>>> We also need clarification for live migration. We notice that Xen_ComputerSystem instance for the Virtual machine which is live migrated is being reported on the source and target host. Can you clarify us how to determine the actual host where the VM is associated? We were hoping that HostedDependency association will be helpful, but did not work.
>>>>
>>>>
>>> Once the live migration of the guest using libvirt-cim is successful you
>>> should not see any information of the VM on the host.
>>> I dont think there is any means from which you can track information
>>> regarding the source from which the guest was migrated using libivrt-cim
>>> providers.
>>> Xen_HostedDependency will give information of the VM's the particular
>>> Server hosts.
>>>> Appreciate any help in this regard.
>>>>
>>>> Sandeep
>>>>
>>>> _______________________________________________
>>>> Libvirt-cim mailing list
>>>> Libvirt-cim at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/libvirt-cim
>>>>
>>
>
>
--
Kaitlin Rupert
IBM Linux Technology Center
kaitlin at linux.vnet.ibm.com
More information about the Libvirt-cim
mailing list