[Crash-utility] "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash

[Dave Anderson]

----- "Kevin Worth" <kevin.worth at hp.com> wrote:

> Hi Dave,
> 
> I tried changing the PAGE_OFFSET definition in kexec-tools. Didn't
> seem to affect it- crash still fails to load the vmalloc'ed memory. If
> that seems like it absolves kexec-tools of any sins then perhaps we
> can drop the kexec-ml off the CC list.
> 
> Your statement "Theoretically, anything at and above 0xb8000000 should
> fail." was accurate, which I saw on my live system (with no dump
> involved). Hoping this provides some insight.

Right -- but when you did the "module 0xf9102280", it read legitimate
data -- but the translated vmalloc address of 0xf9102280 shows a physical
address of 119b76280, i.e. well beyond the physical limit of 0xb8000000:
 
crash> module 0xf9102280
> struct module {
>   state = MODULE_STATE_LIVE,
>   list = {
>     next = 0xf9073d84,
>     prev = 0x403c63a4
>   },
>   name = "custom_lkm" 
> ...
> crash> vtop 0xf9102280
> VIRTUAL   PHYSICAL
> f9102280  119b76280
> ...
> crash> rd -p 119b76000 30
> rd: read error: physical address: 119b76000  type: "32-bit PHYSADDR"
> ...

That being the case, I just remembered something that I had completely 
forgotten about -- because of yet another Red Hat imposed restriction.
On RHEL systems, we have the restricted /dev/mem, but in addition to
that /dev/kmem has been completely removed:

  $ ls -l /dev/mem /dev/kmem
  ls: /dev/kmem: No such file or directory
  crw-r----- 1 root kmem 1, 1 Oct  6 09:17 /dev/mem
  $ 

However, the crash utility, if it realizes that it cannot access a physical
address because it's bigger than the high_memory limit, does this in
read_dev_mem():

        /*
         *  /dev/mem disallows anything >= __pa(high_memory)
         *
         *  However it will allow 64-bit lseeks to anywhere, and when followed
         *  by pulling a 32-bit address from the 64-bit file position, it
         *  quietly returns faulty data from the (wrapped-around) address.
         */
        if (vt->high_memory && (paddr >= (physaddr_t)(VTOP(vt->high_memory)))) {
                readcnt = 0;
                errno = 0;
                goto try_dev_kmem;
        }

So the vmalloc read of 0xf9102280, whose physical address is 0x119b76280 is
greater than the VTOP of 0xf8000000, or b8000000, will goto try_dev_kmem.

First, do you have a /dev/kmem on your system?  If so, the read attempt
continues like so if the passed-in address was a vmalloc address:

        if ((readcnt != cnt) && BITS32() && !readcnt && !errno &&
            IS_VMALLOC_ADDR(addr))
                readcnt = read_dev_kmem(addr, bufptr, cnt);

You'll have to debug crash's memory.c:read_dev_mem() function to determine
whether it followed that path, and successfully read_dev_kmem() above to get
the legitimate module data.  That would be a possible (the only?) explanation
for what you're seeing.

Now, that all being said, debugging the above offers nothing towards the
debugging of the kdump/dumpfile issue. 

Dave