[fedora-virt] f10 x86_64 xen VM guests fail to boot on f8 host (guest setting NX bit in L1 PTE?)

[Valtteri Kiviniemi]

Hi,

I can confirm this. With NX enabled in BIOS the domU boots fine (tested 
with 2.6.28).

If I disable NX in BIOS the domU will crash.

- Valtteri Kiviniemi

Virtualization kirjoitti:
> Hi Ian,
> 
> Indeed nx is on one and not the other! However, that doesn't help...
> 
> Broken CPU:
> /proc/cpuinfo:
> model name      :                   Intel(R) Xeon(TM) CPU 3.00GHz
> flags           : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc up pni monitor ds_cpl cid cx16 xtpr
> 
> Good CPU:
> model name      : AMD Athlon(tm) 64 X2 Dual Core Processor 3600+
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy 3dnowprefetch
> 
> So, ran it again with noexec=0,
> 
>   [ Minimal BASH-like line editing is supported.
>     ESC at any time cancels.  ENTER at any time accepts your changes. ]
> 
> 
>   > kernel /vmlinuz-2.6.27.9-159.fc10.x86_64 ro root=LABEL=/ selinux=0 noipv6 nomodeset noexec=off
> 
> Results:
> 
> [ root at office64 xen ]# /usr/lib64/xen/bin/xenctx -s System.map-2.6.27.9-159.fc10.x86_64 119
> rip: ffffffff8100b8a2 set_page_prot+0x6d
> rsp: ffffffff81575f08
> rax: ffffffea   rbx: 000016e4   rcx: 00000055   rdx: 00000000
> rsi: 800000014a293061   rdi: ffffffff816e4000   rbp: ffffffff81575f68
>  r8: 0000000f    r9: ffffffff817ee350   r10: ffffffff817ee550   r11: 00000010
> r12: ffffffff816e4000   r13: 800000014a293061   r14: 8000000000000161   r15: 00002c00
>  cs: 0000e033    ds: 00000000    fs: 00000000    gs: 00000000
> 
> Stack:
>  0000000000000055 0000000000000010 ffffffff8100b8a2 000000010000e030
>  0000000000010082 ffffffff81575f48 000000000000e02b ffffffff8100b89e
>  0000000000000200 ffffffff816e7000 0000000000000800 0000000000000016
>  ffffffff81575ff8 ffffffff815a5c60 0000000000002c00 0000000000000000
> 
> Code:
> df 54 1d 00 4c 89 e7 4c 89 ee 31 d2 e8 22 d9 ff ff 85 c0 74 04 <0f> 0b eb fe 5b 41 5c 41 5d 41 5e
> 
> Call Trace:
>   [<ffffffff8100b8a2>] set_page_prot+0x6d <--
>   [<ffffffff8100b8a2>] set_page_prot+0x6d
>   [<ffffffff8100b89e>] set_page_prot+0x69
>   [<ffffffff815a5c60>] xen_start_kernel+0x5dd
> 
> Battling with bugzilla trying to get a new account. It doesn't like me :-(
> 
> Might have to leave it up to Jon to do the bugzilla thing.
> 
> Cheers
> Phill.
> 
> On Tue, 2009-01-20 at 17:03 +0000, Ian Campbell wrote:
>> (resending with original xen-devel thread participants on CC, please
>> reply to this subthread, I'll forward you guys Mark's original mail in a
>> second)
>>
>> On Tue, 2009-01-20 at 10:27 +0000, Mark McLoughlin wrote:
>>>        if ( unlikely(l1e_get_flags(nl1e) & L1_DISALLOW_MASK) )
>>>         {
>>>             MEM_LOG("Bad L1 flags %x",
>>>                     l1e_get_flags(nl1e) & L1_DISALLOW_MASK);
>>>             return 0;
>>>         }
>>> ...
>>> }
>>>
>>> the PTE flags are 800000 which corresponds to:
>>>
>>> #define _PAGE_NX_BIT (1U<<23)
>> At least in xen-unstable (and I think for much longer) L1_DISALLOW_MASK
>> contains _PAGE_NX_BIT dynamically depending on the processor
>> capabilities.
>>
>>         #define _PAGE_NX     (cpu_has_nx ? _PAGE_NX_BIT : 0)
>>         ...
>>         /*
>>          * Disallow unused flag bits plus PAT/PSE, PCD, PWT and GLOBAL.
>>          * Permit the NX bit if the hardware supports it.
>>          */
>>         #define BASE_DISALLOW_MASK (0xFFFFF198U & ~_PAGE_NX)
>>         
>>         #define L1_DISALLOW_MASK (BASE_DISALLOW_MASK | _PAGE_GNTTAB)
>>
>> Does the hardware support NX? What does /proc/cpuinfo in dom0 think?
>>
>> The guest kernel should be setting up __supported_pte_mask appropriately
>> to match the hardware and hence shouldn't be using NX if it isn't
>> available. There's a command line option to force NX, can you try
>> noexec=off on the guest command line.
>>
>> My guess would be that the guest is getting a wrong EFER from
>> somewhere...
>>
>> Ian.
>>
>>
>