[Crash-utility] "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash

[Dave Anderson]

>         if ((readcnt != cnt) && BITS32() && !readcnt && !errno &&
>             IS_VMALLOC_ADDR(addr))
>                 readcnt = read_dev_kmem(addr, bufptr, cnt);
> 
> You'll have to debug crash's memory.c:read_dev_mem() function to determine
> whether it followed that path, and successfully read_dev_kmem() above to get
> the legitimate module data.  That would be a possible (the only?)
> explanation for what you're seeing.

I should note that without having to rebuild a debug version of
crash, you can tell whether any /dev/kmem reads have been attempted.
The crash utility's read_dev_kmem() looks like this:

  /*
   *  When /dev/mem won't allow access, try /dev/kmem.
   */
  static ssize_t
  read_dev_kmem(ulong vaddr, char *bufptr, long cnt)
  {
          ssize_t readcnt;

          if (pc->kfd < 0) {
                  if ((pc->kfd = open("/dev/kmem", O_RDONLY)) < 0)
                          return 0;
          }

          if (lseek(pc->kfd, vaddr, SEEK_SET) == -1)
                  return 0;

          readcnt = read(pc->kfd, bufptr, cnt);
          if (readcnt != cnt)
                  readcnt = 0;

          return readcnt;
  }

The global file descriptor pc->kfd is initialized to -1, and
so when run on RHEL systems it will remain equal to -1 because
/dev/kmem doesn't exist.  Presuming that it does exist on your
system, pc->kfd will get a file descriptor -- and will keep
it open for the remainder of the crash session.  You can look
at the value of pc->kfd by entering the debug command "help -p".

Here on on x86 running on RHEL5, pc->kfd is still -1:

crash> help -p
     program_name: crash
     program_path: crash
  program_version: 4.0-7.2
      gdb_version: 6.1
      program_pid: 7165
           prompt: "crash> "
            flags: 3200000210c07 
(RUNTIME|LIVE_SYSTEM|TTY|HASH|SCROLL|GDB_INIT|READLINE|MEMMOD|DATADEBUG|
 FINDKERNEL)
         namelist: /usr/lib/debug/lib/modules/2.6.18-118.el5.perfmon2/vmlinux
         dumpfile: (null)
      live_memsrc: /dev/crash
       system_map: (null)
   namelist_debug: (null)
   debuginfo_file: (null)
    memory_module: crash
    memory_device: /dev/crash
     machine_type: X86
     editing_mode: vi
              nfd: -1
              mfd: 4
              kfd: -1
   ...
            
If your "help -p" shows a legitimate pc->kfd file descriptor, then
in all probability, that's why you can read module data via its
virtual address -- while at the same time not being able to
access it via its physical address (rd -p).

Dave