flexmmap bug on x86-64

[Utz Lehmann]

Hi

I found a bug with flexmmap on x86-64 (kernel-2.6.8-1.521).

A 32bit process only get the new vm layout when it's started from a 32bit
process. When it's started from a 64bit process it's get the legacy layout:

A 32bit cat started from a 32bit shell:

> cat32 /proc/self/maps 
00111000-00126000 r-xp 00000000 03:01 342737                             /lib/ld-2.3.3.so
00126000-00127000 r-xp 00014000 03:01 342737                             /lib/ld-2.3.3.so
00127000-00128000 rwxp 00015000 03:01 342737                             /lib/ld-2.3.3.so
00128000-0012e000 r-xp 00da2000 03:01 1066271                            /usr/lib/locale/locale-archive
0012e000-0012f000 r-xp 02153000 03:01 1066271                            /usr/lib/locale/locale-archive
00136000-00137000 rwxp 00136000 00:00 0 
00137000-0024c000 r-xp 00000000 03:01 342750                             /lib/tls/libc-2.3.3.so
0024c000-0024e000 r-xp 00115000 03:01 342750                             /lib/tls/libc-2.3.3.so
0024e000-00250000 rwxp 00117000 03:01 342750                             /lib/tls/libc-2.3.3.so
00250000-00252000 rwxp 00250000 00:00 0 
00252000-00452000 r-xp 00000000 03:01 1066271                            /usr/lib/locale/locale-archive
00452000-00486000 r-xp 00da9000 03:01 1066271                            /usr/lib/locale/locale-archive
08048000-0804c000 r-xp 00000000 03:01 146973                             /bin/cat32
0804c000-0804d000 rwxp 00003000 03:01 146973                             /bin/cat32
0804d000-0806e000 rwxp 0804d000 00:00 0 
ffffc000-ffffe000 rwxp ffffc000 00:00 0 
ffffe000-fffff000 ---p 00000000 00:00 0 

The same 32bit cat started from a 64bit shell:

> cat32 /proc/self/maps 
08048000-0804c000 r-xp 00000000 03:01 146973                             /bin/cat32
0804c000-0804d000 rwxp 00003000 03:01 146973                             /bin/cat32
0804d000-0806e000 rwxp 0804d000 00:00 0 
55555000-5556a000 r-xp 00000000 03:01 342737                             /lib/ld-2.3.3.so
5556a000-5556b000 r-xp 00014000 03:01 342737                             /lib/ld-2.3.3.so
5556b000-5556c000 rwxp 00015000 03:01 342737                             /lib/ld-2.3.3.so
5557a000-5557b000 rwxp 5557a000 00:00 0 
5557b000-55690000 r-xp 00000000 03:01 342750                             /lib/tls/libc-2.3.3.so
55690000-55692000 r-xp 00115000 03:01 342750                             /lib/tls/libc-2.3.3.so
55692000-55694000 rwxp 00117000 03:01 342750                             /lib/tls/libc-2.3.3.so
55694000-55696000 rwxp 55694000 00:00 0 
55696000-55896000 r-xp 00000000 03:01 1066271                            /usr/lib/locale/locale-archive
55896000-5589c000 r-xp 00da2000 03:01 1066271                            /usr/lib/locale/locale-archive
5589c000-558d0000 r-xp 00da9000 03:01 1066271                            /usr/lib/locale/locale-archive
558d0000-558d1000 r-xp 02153000 03:01 1066271                            /usr/lib/locale/locale-archive
ffffc000-ffffe000 rwxp ffffc000 00:00 0 
ffffe000-fffff000 ---p 00000000 00:00 0 


I think when arch_pick_mmap_layout() is called in fs/exec.c::exec_mmap() the
TIF_IA32 flag is not setuped yet for the new process. So it's really the
flag from the parent.

Adding a additional arch_pick_mmap_layout() in fs/binfmt_elf.c works for me
(only tested on x86-64):


diff -Nrup linux-2.6.8-1.521/fs/binfmt_elf.c linux-2.6.8-1.521-fix-flexmm1/fs/binfmt_elf.c
--- linux-2.6.8-1.521/fs/binfmt_elf.c	2004-08-16 14:58:43.000000000 +0200
+++ linux-2.6.8-1.521-fix-flexmm1/fs/binfmt_elf.c	2004-08-18 16:28:27.000000000 +0200
@@ -769,6 +769,8 @@ static int load_elf_binary(struct linux_
 	if (elf_read_implies_exec(elf_ex, have_pt_gnu_stack))
 		current->personality |= READ_IMPLIES_EXEC;
 
+	arch_pick_mmap_layout(current->mm);
+
 	/* Do this so that we can load the interpreter, if need be.  We will
 	   change some of these later */
 	current->mm->rss = 0;


utz