[Crash-utility] "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash

Worth, Kevin kevin.worth at hp.com
Thu Oct 2 00:03:18 UTC 2008 

Tried running crash on a running kernel... seems that 4.0-3.7 doesn't like my kernel. When I run crash 4.0-7.2 on a live system, it appears that it has no problems with vmalloc'd module memory.

crash 4.0-3.7
...
GNU gdb 6.1
...
This GDB was configured as "i686-pc-linux-gnu"...

crash: /boot/System.map-2.6.20-17.39-custom2 and /dev/mem do not match!

Usage:
  crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]

Enter "crash -h" for details.


crash 4.0-7.2
...
GNU gdb 6.1
...
This GDB was configured as "i686-pc-linux-gnu"...

      KERNEL: vmlinux-2.6.20-17.39-custom2
    DUMPFILE: /dev/mem
        CPUS: 2
        DATE: Wed Oct  1 16:31:39 2008
      UPTIME: 04:57:53
LOAD AVERAGE: 0.10, 0.09, 0.09
       TASKS: 95
    NODENAME: ProCurve-TMS-zl-Module
     RELEASE: 2.6.20-17.39-custom2
     VERSION: #3 SMP Wed Sep 24 10:11:03 PDT 2008
     MACHINE: i686  (2200 Mhz)
      MEMORY: 5 GB
         PID: 15801
     COMMAND: "crash"
        TASK: 47bd6030  [THREAD_INFO: 4a8a8000]
         CPU: 1
       STATE: TASK_RUNNING (ACTIVE)
crash>

Since that seems ok (and I don't encounter the error) I'll run crash with -d7 on the dump file to hopefully expose what is wrong with either the dump or with crash.

I've attached the output of crash with -d7... not sure how the mailing like handles file attachments, but if needed I can paste the text. (or if there is something specific I should look for let me know and I can paste just that section).

-Kevin


-----Original Message-----
From: crash-utility-bounces at redhat.com [mailto:crash-utility-bounces at redhat.com] On Behalf Of Dave Anderson
Sent: Wednesday, October 01, 2008 12:44 PM
To: Discussion list for crash utility usage, maintenance and development
Subject: Re: [Crash-utility] "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash

Worth, Kevin wrote:
> Hello kexec and crash mailing lists,
>
>
>
> Sorry to spam whoever's code this ISN'T an issue with, but I really am
> unsure of whether is a kdump or a crash issue. I am running an Ubuntu
> 7.04 with a 2.6.20 kernel (includes Ubuntus patches- source at
> http://packages.ubuntu.com/feisty/linux-source-2.6.20 ) and a modified
> VMSPLIT/PAGE_OFFSET value (see bottom for details) on an i386 machine
> with 4GB of memory. At first I thought this could be an issue with
> makedumpfile stripping out things it shouldn't, but I've found that
> setting up my initrd script so that it simply performs "cp /proc/vmcore
> /var/crash/vmcore" results in the same issue.
>
>
>
> I've tried this with both crash 4.0-6.3 and 4.0-7.2 and get the same
> result. Unfortunately I'm locked at kernel 2.6.20 for other reasons, or
> else I would try that.
>
>
>
> If anyone can offer suggestions of what to try, please let me know. If
> this is something that has already been resolved elsewhere, sorry to
> waste time, and if someone can point me to what resolved it, perhaps I
> can look at backporting the fix myself. Thanks for your time.
>
>
>
> crash-4.0-7.2$ ./crash ~/vmcore ~/targetfiles/vmlinux-2.6.20-17.39-custom2
>
>
>
> crash 4.0-7.2
>
> <snip>Copyright notices...</snip>
>
> GNU gdb 6.1
>
> <snip>Copyright notices...</snip>
>
> This GDB was configured as "i686-pc-linux-gnu"...
>
>
>
> please wait... (gathering module symbol data)
>
> WARNING: cannot access vmalloc'd module memory
>
>
>
>       KERNEL: /home/worthk/targetfiles/vmlinux-2.6.20-17.39-custom2
>
>     DUMPFILE: /home/worthk/vmcore
>
>         CPUS: 2
>
>         DATE: Wed Oct  1 12:30:50 2008
>
>       UPTIME: 00:35:11
>
> LOAD AVERAGE: 0.07, 0.09, 0.08
>
>        TASKS: 94
>
>     NODENAME: test-module
>
>      RELEASE: 2.6.20-17.39-custom2
>
>      VERSION: #3 SMP Wed Sep 24 10:11:03 PDT 2008
>
>      MACHINE: i686  (2200 Mhz)
>
>       MEMORY: 5 GB
>
> <6>SysRq : Trigger a crashdump"
>
>          PID: 4304
>
>      COMMAND: "bash"
>
>         TASK: 5d7e9030  [THREAD_INFO: f4b70000]
>
>          CPU: 0
>
>        STATE: TASK_RUNNING (SYSRQ)
>
>
>
> crash> mod -s test
>
> mod: cannot access vmalloc'd module memory
>
>
>
>
>
> My kernel config is a bit outside the norm, in that the VMSPLIT value
> has been modified to give 3GB of memory the kernelspace and 1GB of
> memory to userspace. Below is a diff between the default Ubuntu
> "generic" config and mine:

To be honest with you I'm surprised it comes up at all...

If you do a "crash -d7 vmlinux vmcore", amongst the reams of
debug data you will see the readmem() that failed just prior
to the "WARNING: cannot access vmalloc'd module memory".  And
that will probably be the very first access of a vmalloc'd
virtual memory address.  Probably it's best to enter
"crash -d7 vmlinux vmcore > /tmp/junk", and then enter "q"
to silently kill the session.

For that matter, once you come up, I'm guessing that user
virtual address translation will fail as well.  Come up
as you did above, do a "vm" command on the "bash" task,
and then a "vtop" on a user virtual address.

Like this example:

crash> vm
PID: 25479  TASK: f6f2aaa0  CPU: 3   COMMAND: "bash"
    MM       PGD      RSS    TOTAL_VM
f6e3d740  f745c980  1560k    4608k
   VMA       START      END    FLAGS  FILE
f6c115f4    110000    112000     75  /lib/
f7212f94    112000    113000 100071  /lib/
f78cd0cc    113000    114000 100073  /lib/
f6954d84    584000    585000 8000075
f7241bcc    5b1000    5b4000     75  /lib/libtermcap.so.2.0.8.#prelink#.YYRDOu
f7212a14    5b4000    5b5000 100073  /lib/libtermcap.so.2.0.8.#prelink#.YYRDOu
f6e1a64c    61a000    623000     75  /lib/libnss_files-2.5.so
f73f738c    623000    624000 100071  /lib/libnss_files-2.5.so
f6eb79bc    624000    625000 100073  /lib/libnss_files-2.5.so
f7212f3c    719000    733000    875  /lib/
f721238c    733000    734000 100871  /lib/
f72e2b1c    734000    735000 100873  /lib/
f73f7964    ab5000    bf2000     75  /lib/
f6c11ee4    bf2000    bf4000 100071  /lib/
f73f7ee4    bf4000    bf5000 100073  /lib/
f73f7f3c    bf5000    bf8000 100073
f721217c   8048000   80f5000   1875  /bin/
f6cab90c   80f5000   80fa000 101873  /bin/
f724143c   80fa000   80ff000 100073
f68cf7ac   8574000   85aa000 100073
f6d354ec  b7d81000  b7f81000     71  /usr/lib/locale/locale-archive
f7594d84  b7f81000  b7f84000 100073
f6f24124  b7f84000  b7f85000 100073
f72e2d2c  b7f85000  b7f8c000     d1  /usr/lib/gconv/gconv-modules.cache
f72418b4  bfa8a000  bfa9f000 100173
crash> vtop 584000
VIRTUAL   PHYSICAL
584000    37cd6000

PAGE DIRECTORY: f745c980
   PGD: f745c980 => 369e0001
   PMD: 369e0010 => 11b7c1067
   PTE: 11b7c1c20 => 37cd6025
  PAGE: 37cd6000

   PTE     PHYSICAL  FLAGS
37cd6025  37cd6000  (PRESENT|USER|ACCESSED)

   VMA       START      END    FLAGS  FILE
f6954d84    584000    585000 8000075

   PAGE     PHYSICAL   MAPPING    INDEX CNT FLAGS
c16f9ac0   37cd6000         0         0 48 80000004
crash>

Does the vtop command fall apart somewhere?

BTW, if you haven't done it already, you should also
take the dumpfile out of the picture, and just run crash
on the live system.  If by some stretch of the imagination
*that* works, then you might have to point the finger back
at kdump operation.

In any case, at least you've got a situation where crash
can at least deal with unity-mapped addresses.  With those
addresses it doesn't have to do any kind of page-table
walk-throughs.

I'm guessing that there's something in x86.c's x86_init() function
in the PRE_GDB section that's not correct for your setup.
There is support for Red Hat's older "hugemem" 4G/4G split
kernels, where both the kernel and user space have 4G
(over-lapping) virtual address regions, and so there may be
some confusion there with yours.

For starters, bring up the session as you did above,
and enter "help -v" and "help -m".  They're debug
options that dump a couple internal crash data structures
which may shed some light.

Dave

--
Crash-utility mailing list
Crash-utility at redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
-------------- next part --------------
A non-text attachment was scrubbed...
Name: crash.log
Type: application/octet-stream
Size: 99695 bytes
Desc: crash.log
URL: <http://listman.redhat.com/archives/crash-utility/attachments/20081002/070f274e/attachment.obj>

More information about the Crash-utility mailing list