[Libvirt-cim] KVM test report on Fedora 9

[Kaitlin Rupert]

>> Also, there was some discussion on the mailing list about modifying 
>> the negative test cases so that they only check the provider return 
>> codes. I think it'll be awhile before we can add implementation 
>> specific return codes to the providers.  Since the CIM return codes 
>> aren't specific enough to indicate exactly what kind of error 
>> occurred, I'm inclined to continue checking the return messages in the 
>> test cases for now.
>>
>> Thoughts?
> I agree with you on checking both the return codes and the messages. But 
> I thought branching the test cases for different changeset of providers 
> is a little risky. I am in the mood that we're going to maintain massive 
> if-else branches on this if the provider message strings change too fast.

Completely agree - they've already been a headache to maintain as it is.

> 
> An optimistic view would be that even though we need to maintain a 
> little bit too many of branches at first. But as the providers get more 
> stable, these frequent changes are less likely to happen.
> 
> Ok, my third view is a little unrealistic. We can develop a fifth test 
> case return code, named 'conditional pass', specifically for the rc 
> matches, string doesn't match issue. :=)

While not a bad idea, maintaining yet another return code can be a pain. 
  Especially if it doesn't get set back to pass/fail when need be. 
Heidi was working on updating these message, and is most of the way done.

 From your F9 release providers test run, it looks like 1 or possibly 2 
tests cases that encounter this issue.  I'm inclined to have these tests 
branch for now.  If I see a trend that more error messages are changing, 
then add something like an additional return code.

Either way, I'd say leave these test cases as a lower priority to fix 
for now.  I'd rather focus on ensuring the more complex tests pass on 
KVM.  =)

-- 
Kaitlin Rupert
IBM Linux Technology Center
kaitlin at linux.vnet.ibm.com