[Crash-utility] Crash fails with the error: "crash: read error: kernel virtual address:..."
Adhiraj Joshi
adhiraj at linsyssoft.com
Fri Feb 27 09:56:35 UTC 2009
Hi Dave,
Thanks for the quick reply! I followed approach 1 of disabling
CONFIG_STRICT_DEVMEM and the problem disappeared. Now a new problem :-) On
invoking crash, I get the following warning:
*WARNING: cannot access vmalloc'd module memory*
Because of this I am not able to run commands like "mod":
crash> mod
mod: cannot access vmalloc'd module memory
crash>
Should the approach 2 or 3 that you mentioned below solve this problem? I
have upgraded crash to the latest version (4.0-7.7).
Thanks and regards,
Adhiraj.
On Wed, Feb 25, 2009 at 7:45 PM, Dave Anderson <anderson at redhat.com> wrote:
>
> ----- "Adhiraj Joshi" <adhiraj at linsyssoft.com> wrote:
>
> > Hi All,
> >
> > On my x86 machine with fedora 10, I get an error which says: "crash:
> > read error: kernel virtual address:...". The running kernel is not the
> > Fedora 10 kernel, I have installed a latest vanilla kernel (2.6.27.12)
> > on my machine.
> >
> > On googling, I found that someone has reported a similar problem but
> > which was on Fedora kernel and x86_64 arch. Here is the link to that bug:
> > https://bugzilla.redhat.com/show_bug.cgi?id=237383
> >
> > In my case, this is the output from the crash command:
> >
> > [root at maveric ~]# crash /lib/modules/`uname -r`/build/vmlinux
> > crash 4.0-7
> > Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
> > Copyright (C) 2004, 2005, 2006 IBM Corporation
> > Copyright (C) 1999-2006 Hewlett-Packard Co
> > Copyright (C) 2005, 2006 Fujitsu Limited
> > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> > Copyright (C) 2005 NEC Corporation
> > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> > This program is free software, covered by the GNU General Public License,
> > and you are welcome to change it and/or distribute copies of it under
> > certain conditions. Enter "help copying" to see the conditions.
> > This program has absolutely no warranty. Enter "help warranty" for
> > details.
> >
> > GNU gdb 6.1
> > Copyright 2004 Free Software Foundation, Inc.
> > GDB is free software, covered by the GNU General Public License, and you
> are
> > welcome to change it and/or distribute copies of it under certain
> conditions.
> > Type "show copying" to see the conditions.
> > There is absolutely no warranty for GDB. Type "show warranty" for
> details.
> > This GDB was configured as "i686-pc-linux-gnu"...
> >
> > crash: read error: kernel virtual address: c08b0ae8 type:
> "cpu_possible_map"
> > WARNING: cannot read cpu_possible_map
> > crash: read error: kernel virtual address: c0801a94 type:
> "cpu_present_map"
> > WARNING: cannot read cpu_present_map
> > crash: read error: kernel virtual address: c080155c type:
> "cpu_online_map"
> > WARNING: cannot read cpu_online_map
> > crash: read error: kernel virtual address: c08dd700 type: "xtime"
> > [root at maveric ~]#
> >
> > This is the "uname -a" output:
> > "Linux maveric 2.6.27.12-adhi #1 SMP Fri Feb 20 22:13:54 IST 2009 i686
> > i686 i386 GNU/Linux"
> >
> > Does anyone has any idea on this?
>
> Your kernel is configured with CONFIG_STRICT_DEVMEM, which disallows
> the reading of /dev/mem above 1MB physical. That makes it useless for
> the crash utility:
>
> https://www.redhat.com/archives/crash-utility/2009-February/msg00003.html
>
> Fedora and RHEL kernels contain the /dev/crash driver, which gets
> automatically
> installed upon crash invocation on the live system. That driver, found in
> the kernel file "drivers/char/crash.c", is a replacement for the
> (restricted)
> /dev/mem driver.
>
> For your kernel, you've got 3 options:
>
> (1) Rebuild your kernel without the CONFIG_STRICT_DEVMEM restriction.
> (2) Port the Fedora /dev/crash driver (./drivers/char/crash.c) to your
> kernel.
> (3) Write a kretprobe module that tinkers with the return value of the
> kernel's devmem_is_allowed() function such that it always returns 1.
>
> And w/respect to each option:
>
> (1) Obviously this is the course of least resistance.
>
> (2) Porting the RHEL/Fedora /dev/crash driver is possible, but since
> page_is_ram()
> is not EXPORT_GPL()'d in 2.6.27 upstream kernels, then its usage by the
> driver
> would have to be removed. And if you were to make it EXPORT_GPL(), then
> it
> makes far more sense to just remove the CONFIG_STRICT_DEVMEM instead.
>
> (3) I'll append some information re: writing a kretprobe module, but like
> porting the /dev/crash driver, it may require rebuilding your kernel
> anyway.
> And if that's the case, just remove the CONFIG_STRICT_DEVMEM
> restriction.
>
> Also, you should upgrade from 4.0-7. There have been several fixes
> for 2.6.27 kernels since that version.
>
> Dave
>
>
> ----------------------------------------------------------------------------
>
> Writing a kretprobe for devmem_is_allowed():
>
> There are are currently three types of probes: kprobes, jprobes, and
> kretprobes (also called return probes), but for this purpose, a
> kretprobe is required because it allows the return value of the
> devmem_is_allowed() function to be modified such that it will
> always return 1.
>
> The kernel documentation contains both directions for building
> a kretprobe module and example module files for each kprobe type.
> Pre-2.6.25 kernel trees put sample cut-and-pastable module files
> contained within the "./Documentation/kprobes.txt" file itself.
> 2.6.25 and later kernel trees locate them in the "./samples/kprobes"
> directory.
>
> Note that pre-2.6.25 kernels must have been configured with both
> CONFIG_KPROBES, CONFIG_KALLSYMS and CONFIG_MODULES turned on.
> Additionally, 2.6.25 and later kernels also need CONFIG_KRETPROBES,
> In any case, if the target kernel configs preclude the installation
> of the module, then it probably makes more sense to rebuild the
> kernel with CONFIG_STRICT_DEVMEM turned off, and avoid this kprobe
> approach entirely.
>
> Take the "kretprobe-example.c" cut-and-pastable file from the
> the "./Documentation/kprobes.txt" file, or the standalone
> "./samples/kprobes/kretprobe_example.c" file, whichever is
> appropriate, and modify the kretprobe_init() and ret_handler()
> functions like so:
>
> static int __init kretprobe_init(void)
> {
> int ret;
>
> - my_kretprobe.kp.symbol_name = func_name;
> + my_kretprobe.kp.symbol_name = "devmem_is_allowed";
> ret = register_kretprobe(&my_kretprobe);
> if (ret < 0) {
> printk(KERN_INFO "register_kretprobe failed, returned
> %d\n",
> ret);
> return -1;
> }
> printk(KERN_INFO "Planted return probe at %s: %p\n",
> my_kretprobe.kp.symbol_name, my_kretprobe.kp.addr);
> return 0;
> }
>
> and make the ret_handler() function to simply do this:
>
> static int ret_handler(struct kretprobe_instance *ri, struct pt_regs
> *regs)
> {
> regs->ax = 1;
> return 0;
> }
>
> Earlier kernel versions may have different pt_regs structure member
> names for the return register, i.e., "regs->eax" or "regs->rax" for
> the x86 and x86_64 architectures respectively.
>
> Build the kretprobe module as directed in ./Documentation/kprobes.txt"
> and insmod it, after which crash will run just fine with /dev/mem.
>
>
>
>
>
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/crash-utility/attachments/20090227/9e8988ca/attachment.htm>
More information about the Crash-utility
mailing list