[Crash-utility] "crash: cannot allocate any more memory!" for 2.6.27.4
Dheeraj Sangamkar
dheerajrs at gmail.com
Fri Dec 5 06:51:44 UTC 2008
Thanks Dave. The kretprobe worked for me.
Dheeraj
On Thu, Dec 4, 2008 at 8:14 PM, Dave Anderson <anderson at redhat.com> wrote:
>
> ----- "Dheeraj Sangamkar" <dheerajrs at gmail.com> wrote:
>
> > On the same machine, I am unable to live-debug the running kernel.
> > Here is the error I get. Please let me know if there is something I
> > can do to get crash working.
> >
> > /dheeraj/crash/crash-4.0-7.4 # crash -d 255
> > /boot/System.map-2.6.27.4-2-default
> > /root/dheeraj/linux-2.6.27.4-2.1/vmlinux
> >
> > crash 4.0-7.4
> > Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
> > Copyright (C) 2004, 2005, 2006 IBM Corporation
> > Copyright (C) 1999-2006 Hewlett-Packard Co
> > Copyright (C) 2005, 2006 Fujitsu Limited
> > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> > Copyright (C) 2005 NEC Corporation
> > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> > This program is free software, covered by the GNU General Public
> > License,
> > and you are welcome to change it and/or distribute copies of it under
> > certain conditions. Enter "help copying" to see the conditions.
> > This program has absolutely no warranty. Enter "help warranty" for
> > details.
> >
> > crash: diskdump / compressed kdump: dump does not have panic dump
> > header
> > get_live_memory_source: /dev/mem
> > crash: pv_init_ops exists: ARCH_PVOPS
> > _text: ffffffff80200000 Kernel code: 200000 -> phys_base: 0
> >
> > gdb /root/dheeraj/linux-2.6.27.4-2.1/vmlinux
> > GNU gdb 6.1
> > Copyright 2004 Free Software Foundation, Inc.
> > GDB is free software, covered by the GNU General Public License, and
> > you are
> > welcome to change it and/or distribute copies of it under certain
> > conditions.
> > Type "show copying" to see the conditions.
> > There is absolutely no warranty for GDB. Type "show warranty" for
> > details.
> > This GDB was configured as "x86_64-unknown-linux-gnu"...
> > GETBUF(248 -> 0)
> > GETBUF(1500 -> 1)
> >
> > FREEBUF(1)
> > FREEBUF(0)
> > <readmem: ffffffff804be660, KVADDR, "kernel_config_data", 32768,
> > (ROE), 1ae5040>
> > /dev/mem: Operation not permitted
> > crash: read(/dev/mem, 4be660, 2464): 4294967295 (ffffffff)
> > crash: read error: kernel virtual address: ffffffff804be660 type:
> > "kernel_config_data"
> > WARNING: cannot read kernel_config_data
> > GETBUF(248 -> 0)
> > FREEBUF(0)
> > GETBUF(16 -> 0)
> > <readmem: ffffffff80a14800, KVADDR, "cpu_possible_map", 16, (ROE),
> > a49160>
> > /dev/mem: Operation not permitted
> > crash: read(/dev/mem, a14800, 16): 4294967295 (ffffffff)
> > crash: read error: kernel virtual address: ffffffff80a14800 type:
> > "cpu_possible_map"
> > WARNING: cannot read cpu_possible_map
> > <readmem: ffffffff80872710, KVADDR, "cpu_present_map", 16, (ROE),
> > a49160>
> > /dev/mem: Operation not permitted
> > crash: read(/dev/mem, 872710, 16): 4294967295 (ffffffff)
> > crash: read error: kernel virtual address: ffffffff80872710 type:
> > "cpu_present_map"
> > WARNING: cannot read cpu_present_map
> > <readmem: ffffffff80846c90, KVADDR, "cpu_online_map", 16, (ROE),
> > a49160>
> > /dev/mem: Operation not permitted
> > crash: read(/dev/mem, 846c90, 16): 4294967295 (ffffffff)
> > crash: read error: kernel virtual address: ffffffff80846c90 type:
> > "cpu_online_map"
> > WARNING: cannot read cpu_online_map
> > FREEBUF(0)
> > <readmem: ffffffff80a74400, KVADDR, "xtime", 16, (FOE), a0d2f0>
> > /dev/mem: Operation not permitted
> > crash: read(/dev/mem, a74400, 16): 4294967295 (ffffffff)
> > crash: read error: kernel virtual address: ffffffff80a74400 type:
> > "xtime"
>
> Live system analysis is impossible if /dev/mem won't allow you
> to read physical memory. Each read attempt above results in an
> "Operation not permitted" (EPERM) error.
>
> They fail because of a pain-in-the-ass CONFIG_STRICT_DEVMEM configuration
> option that was recently added to upstream kernels. When the crash
> utility makes a read() from /dev/mem, it ends up in the kernel's
> drivers/char/mem.c driver, and proceeds like so:
>
> read_mem()
> range_is_allowed()
> devmem_is_allowed()
>
> Here is the call in read_mem() that returns the EPERM:
>
> if (!range_is_allowed(p >> PAGE_SHIFT, count))
> return -EPERM;
>
> Here are the two versions of range_is_allowed() in drivers/char/mem.c,
> which depend upon CONFIG_STRICT_DEVMEM:
>
> #ifdef CONFIG_STRICT_DEVMEM
> static inline int range_is_allowed(unsigned long pfn, unsigned long size)
> {
> u64 from = ((u64)pfn) << PAGE_SHIFT;
> u64 to = from + size;
> u64 cursor = from;
>
> while (cursor < to) {
> if (!devmem_is_allowed(pfn)) {
> printk(KERN_INFO
> "Program %s tried to access /dev/mem between %Lx->%Lx.\n",
> current->comm, from, to);
> return 0;
> }
> cursor += PAGE_SIZE;
> pfn++;
> }
> return 1;
> }
> #else
> static inline int range_is_allowed(unsigned long pfn, unsigned long size)
> {
> return 1;
> }
> #endif
>
> And the call to devmem_is_allowed() above ends up here, where only the
> first 256 pages (1MB) of physical memory is accessible:
>
> int devmem_is_allowed(unsigned long pagenr)
> {
> if (pagenr <= 256)
> return 1;
> if (!page_is_ram(pagenr))
> return 1;
> return 0;
> }
>
> So that makes /dev/mem useless for the crash utility...
>
> Bernhard Walle made a valiant attempt to make this a run-time configurable
> rather than a compile-time config option,
>
> http://lkml.org/lkml/2008/11/16/117
>
> But the LKML thread went off into the weeds arguing whether the
> CONFIG_STRICT_DEVMEM
> imposition itself should be ripped out, rather than putting lipstick on a
> pig.
> And so it got nowhere...
>
> This same 256-page restriction has been in place in RHEL4 and RHEL5 kernels
> as well, because the same guy who managed to push it upstream formerly
> worked
> at Red Hat, and convinced the powers-that-be here that /dev/mem was a
> security
> hole. As a result, I had to write a whole new physical memory-access
> device
> driver (/dev/crash) that is a read-only "misc" driver that the crash
> utility
> automatically loads when run on a live system. But it's only available in
> the Red Hat RHEL4 and RHEL5 distribution, although it can be ported easily
> enough.
>
> So you've got 3 options:
>
> 1. Rebuild your kernel with CONFIG_STRICT_DEVMEM turned off.
>
> 2. Port the RHEL /dev/crash driver to your kernel. I did suggest that
> in another crash-utility thread here:
>
>
> https://www.redhat.com/archives/crash-utility/2008-October/msg00057.html
>
> 3. Write a kprobe module that forces devmem_is_allowed() to return 1
> always:
>
> http://www.redhat.com/archives/crash-utility/2008-March/msg00036.html
>
> If you've got the power, obviously #1 is the best option. But if you
> cannot
> change kernels, probably the kprobe option #3 is easier than porting the
> RHEL
> /dev/crash driver. Check Documentation/kprobes.txt file and the
> samples/kprobes
> directory in your kernel build tree, and then use the RHEL kprobe example
> I put in the thread referenced in #3 above. If that won't work (and I
> don't
> know why it wouldn't), then you can always port the RHEL /dev/crash driver
> that's attached to the thread reference in #2 above.
>
> Dave
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
--
Simplicity of character is the natural result of profound thought.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/crash-utility/attachments/20081205/c6e303e7/attachment.htm>
More information about the Crash-utility
mailing list