[Crash-utility] Pages excluded in dump RHEL 6.2 - set to ffffffffffffffff
Oberman, Laurence (GSE MSE)
Laurence.Oberman at hp.com
Wed Jun 20 15:18:38 UTC 2012
Dave,
You had it exactly right:
Using the generic System.map-2.6.32-220.7.1.el6.x86_64 works fine. The System.map from my build was missing symbols.
Thanks!
Laurence
crash 6.0.7
Copyright (C) 2002-2012 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
SYSTEM MAP: System.map-2.6.32-220.7.1.el6.x86_64
DEBUG KERNEL: vmlinux (2.6.32-220.7.1.el6)
DUMPFILE: va102.2012-06-14.vmcore [PARTIAL DUMP]
CPUS: 24
DATE: Thu Jun 14 05:24:17 2012
UPTIME: 11:13:23
LOAD AVERAGE: 215.00, 196.78, 135.81
TASKS: 1778
NODENAME: va1splunkindex02.va1.swtor.int
RELEASE: 2.6.32-220.7.1.el6.x86_64
VERSION: #1 SMP Tue Mar 6 13:54:51 EST 2012
MACHINE: x86_64 (2799 Mhz)
MEMORY: 48 GB
PANIC: "Oops: 0002 [#1] SMP " (check log for details)
PID: 2428
COMMAND: "bash"
TASK: ffff88063874ab40 [THREAD_INFO: ffff8806dcd1a000]
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
-----Original Message-----
From: crash-utility-bounces at redhat.com [mailto:crash-utility-bounces at redhat.com] On Behalf Of Dave Anderson
Sent: Wednesday, June 20, 2012 10:42 AM
To: Discussion list for crash utility usage, maintenance and development
Subject: Re: [Crash-utility] Pages excluded in dump RHEL 6.2 - set to ffffffffffffffff
----- Original Message -----
> Hi Dave,
>
> OK, makes sense. I still spend 95% of my time in the RHEL 5 kernels so
> it's just lack of enough time in the RHEL 6 kernels.
> I did not have the debuginfo so I built a vmkernel with debug from
> source and used that. I was able to get what I wanted out of the dump,
> specifically needed to look at the net_dev structures.
>
> Is there a write up somewhere on changes with crash for the 2.6.32.x
> kernels that you can share.
No, not specifically for the 2.6.32 or any other kernel version.
The crash utility is constantly updated to handle the shifting sands of the underlying Linux kernel. But it's also kept backwards-compatible, so the crash sources are full of checks for kernel differences such as the cpu maps to mask changes.
But I'm still not clear why you are reading the ffffffffffffffff's at those symbolic locations if the System.map's symbol values (which over-write the ones in the vmlinux file) match what's in the dumped kernel? I believe that the vmlinux file that you built -- even with the correct System.map -- is different enough from the crashed kernel so that there are inequities that cannot be overcome with just a System.map.
Dave
> Thanks
> Laurence
>
>
> -----Original Message-----
> From: crash-utility-bounces at redhat.com
> [mailto:crash-utility-bounces at redhat.com] On Behalf Of Dave Anderson
> Sent: Wednesday, June 20, 2012 9:22 AM
> To: Discussion list for crash utility usage, maintenance and
> development
> Subject: Re: [Crash-utility] Pages excluded in dump RHEL 6.2 - set to
> ffffffffffffffff
>
>
>
> ----- Original Message -----
> > I have a dump from RHEL 6.2 with all these masks set to all f's, so
> > obviously crash excludes them.
> >
> > crash: page excluded: kernel virtual address: ffffffffffffffff
> > type: "possible"
> > WARNING: cannot read cpu_possible_map
> > crash: page excluded: kernel virtual address: ffffffffffffffff
> > type: "present"
> > WARNING: cannot read cpu_present_map
> > crash: page excluded: kernel virtual address: ffffffffffffffff
> > type: "online"
> > WARNING: cannot read cpu_online_map
> >
> > typedef struct cpumask { DECLARE_BITMAP(bits, NR_CPUS); } cpumask_t;
> >
> > #ifndef CONFIG_DISABLE_OBSOLETE_CPUMASK_FUNCTIONS
> > /* These strip const, as traditionally they weren't const. */
> > #define cpu_possible_map (*(cpumask_t *)cpu_possible_mask)
> > #define cpu_online_map (*(cpumask_t *)cpu_online_mask)
> > #define cpu_present_map (*(cpumask_t *)cpu_present_mask)
> > #define cpu_active_map (*(cpumask_t *)cpu_active_mask)
> >
> > Are these deprecated now and is that why they are fixed at all f's.
>
> No, that's just a common error message for accessing the maps or
> masks.
>
> When reading them, crash checks first whether there is the older
> kernel symbol named "cpu_xxx_map", and if it doesn't exists, then it
> checks for the symbol named "cpu_xxx_mask". It then tried to read the
> memory that those symbols point to, which all contained values of
> ffffffffffffffff. So clearly there's some kind of mis-match between
> the symbol values -- modified by using a System.map file -- and what's
> in the vmcore file.
>
> (FWIW, the dumpfile couldn't find the page for bogus address, and
> presumed that it was "excluded" since it's a filtered dump, so that's
> a non-issue...)
>
> First question: it appears that you are analyzing a RHEL6 kernel, so
> why are you using a "System.map" argument? Did you rebuild the kernel
> but when you did that, you didn't save the rebuilt vmlinux file?
>
> Dave
>
> >
> > [root at centos62 crash]# crash System.map vmlinux
> > va102.2012-06-14.vmcore
> >
> > crash 6.0.7
> > Copyright (C) 2002-2012 Red Hat, Inc.
> > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C)
> > 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012
> > Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan
> > K.K.
> > Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002,
> > 2007 Silicon Graphics, Inc.
> > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> > This program is free software, covered by the GNU General Public
> > License, and you are welcome to change it and/or distribute copies
> > of it under certain conditions. Enter "help copying" to see the
> > conditions.
> > This program has absolutely no warranty. Enter "help warranty" for
> > details.
> >
> > GNU gdb (GDB) 7.3.1
> > Copyright (C) 2011 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later
> > <http://gnu.org/licenses/gpl.html>
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law. Type "show
> > copying"
> > and "show warranty" for details.
> > This GDB was configured as "x86_64-unknown-linux-gnu"...
> >
> > crash: page excluded: kernel virtual address: ffffffffffffffff
> > type: "possible"
> > WARNING: cannot read cpu_possible_map
> > crash: page excluded: kernel virtual address: ffffffffffffffff
> > type: "present"
> > WARNING: cannot read cpu_present_map
> > crash: page excluded: kernel virtual address: ffffffffffffffff
> > type: "online"
> > WARNING: cannot read cpu_online_map
> > crash: page excluded: kernel virtual address: ffffffffffffffff
> > type: "cpu_present_map"
> > crash: page excluded: kernel virtual address: ffffffffffffffff
> > type: "cpu_present_map"
> > crash: page excluded: kernel virtual address: ffffffffffffffff
> > type: "cpu_online_map"
> > SYSTEM MAP: System.map
> > DEBUG KERNEL: vmlinux (2.6.32-220.7.1.el6)
> > DUMPFILE: va102.2012-06-14.vmcore [PARTIAL DUMP]
> > CPUS: 32
> > DATE: Thu Jun 14 05:24:17 2012
> > UPTIME: 11:13:23
> > LOAD AVERAGE: 215.00, 196.78, 135.81
> > TASKS: 1778
> > NODENAME: va1splunkindex02.va1.swtor.int
> > RELEASE: 2.6.32-220.7.1.el6.x86_64
> > VERSION: #1 SMP Tue Mar 6 13:54:51 EST 2012
> > MACHINE: x86_64 (2799 Mhz)
> > MEMORY: 48 GB
> > PANIC: "Oops: 0002 [#1] SMP " (check log for details)
> > PID: 2428
> > COMMAND: "bash"
> > TASK: ffff88063874ab40 [THREAD_INFO: ffff8806dcd1a000]
> > CPU: 0
> > STATE: TASK_RUNNING (PANIC)
> >
> > Thanks
> > Laurence
> >
> > --
> > Crash-utility mailing list
> > Crash-utility at redhat.com
> > https://www.redhat.com/mailman/listinfo/crash-utility
> >
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
--
Crash-utility mailing list
Crash-utility at redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
More information about the Crash-utility
mailing list