[Crash-utility] crash and sles 9 GUEST dumps
Daniel Li
dli at virtualiron.com
Mon Aug 13 15:27:41 UTC 2007
After finding out how to get crash working with native sles 9 LKCD
format dumps -- namely, build and use a debug vmlinux with appropriate
flags to feed to crash -- I started looking into using crash on kernel
dumps created for sles 9 guest domains.
As compared to the LKCD format of native sles 9 dumps, those dumps are
created using the new non-standard ELF format with section headers
instead of program headers, which is the case with the xenctrl library
now. Such formats are working for RHAS4U4 64bit guests, while I had to
make minor modification to make it work for RHAS4U4 32bit guests as
well. However, when it comes to sles 9 guests, crash seems to be having
problems locating the stacks for each thread, with the exception of the
CURRENT thread. (see below)
It may well be that the stack pointers were not saved properly for sles
9 guests by the Xen library in the dump. I'll take a look into the dump
and the xen library code to see if that is the case... Or is this the
case of crash not looking in the right places for those stack pointers?
Thanks,
Daniel
/dumps/new/sles/64bit$ /home/dli/bin/crash vmlinux-2.6.5-7.244-smp
vmlinux.dbg DUMP10.1.230.112
crash 4.0-4.5
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
WARNING: could not find MAGIC_START!
please wait... (gathering task table data)
crash: invalid kernel virtual address: 13 type: "fill_thread_info"
crash: invalid kernel virtual address: f type: "fill_thread_info"
crash: invalid kernel virtual address: 6 type: "fill_thread_info"
crash: invalid kernel virtual address: c type: "fill_thread_info"
crash: invalid kernel virtual address: 13 type: "fill_thread_info"
crash: invalid kernel virtual address: c type: "fill_thread_info"
crash: invalid kernel virtual address: 18 type: "fill_thread_info"
crash: invalid kernel virtual address: f type: "fill_thread_info"
crash: invalid kernel virtual address: 13 type: "fill_thread_info"
crash: invalid kernel virtual address: 13 type: "fill_thread_info"
crash: invalid kernel virtual address: 12 type: "fill_thread_info"
crash: invalid kernel virtual address: 11 type: "fill_thread_info"
crash: invalid kernel virtual address: 15 type: "fill_thread_info"
crash: invalid kernel virtual address: f type: "fill_thread_info"
crash: invalid kernel virtual address: 12 type: "fill_thread_info"
crash: invalid kernel virtual address: 6e type: "fill_thread_info"
crash: invalid kernel virtual address: 22 type: "fill_thread_info"
crash: invalid kernel virtual address: 13 type: "fill_thread_info"
crash: invalid kernel virtual address: f type: "fill_thread_info"
crash: invalid kernel virtual address: c type: "fill_thread_info"
crash: invalid kernel virtual address: c type: "fill_thread_info"
crash: invalid kernel virtual address: 13 type: "fill_thread_info"
crash: invalid kernel virtual address: f type: "fill_thread_info"
crash: invalid kernel virtual address: f type: "fill_thread_info"
crash: invalid kernel virtual address: 11 type: "fill_thread_info"
crash: invalid kernel virtual address: 10 type: "fill_thread_info"
crash: invalid kernel virtual address: c type: "fill_thread_info"
crash: invalid kernel virtual address: 14 type: "fill_thread_info"
crash: invalid kernel virtual address: 13 type: "fill_thread_info"
crash: invalid kernel virtual address: 18 type: "fill_thread_info"
crash: invalid kernel virtual address: f type: "fill_thread_info"
crash: invalid kernel virtual address: e type: "fill_thread_info"
crash: invalid kernel virtual address: f type: "fill_thread_info"
please wait... (determining panic task)
bt: invalid kernel virtual address: 13 type: "stack contents"
bt: read of stack at 13 failed
bt: invalid kernel virtual address: f type: "stack contents"
bt: read of stack at f failed
bt: invalid kernel virtual address: 6 type: "stack contents"
bt: read of stack at 6 failed
bt: invalid kernel virtual address: c type: "stack contents"
bt: read of stack at c failed
bt: invalid kernel virtual address: 13 type: "stack contents"
bt: read of stack at 13 failed
bt: invalid kernel virtual address: c type: "stack contents"
bt: read of stack at c failed
bt: invalid kernel virtual address: 18 type: "stack contents"
bt: read of stack at 18 failed
bt: invalid kernel virtual address: f type: "stack contents"
bt: read of stack at f failed
bt: invalid kernel virtual address: 13 type: "stack contents"
bt: read of stack at 13 failed
bt: invalid kernel virtual address: 13 type: "stack contents"
bt: read of stack at 13 failed
bt: invalid kernel virtual address: 12 type: "stack contents"
bt: read of stack at 12 failed
bt: invalid kernel virtual address: 11 type: "stack contents"
bt: read of stack at 11 failed
bt: invalid kernel virtual address: 15 type: "stack contents"
bt: read of stack at 15 failed
bt: invalid kernel virtual address: f type: "stack contents"
bt: read of stack at f failed
bt: invalid kernel virtual address: 12 type: "stack contents"
bt: read of stack at 12 failed
bt: invalid kernel virtual address: 6e type: "stack contents"
bt: read of stack at 6e failed
bt: invalid kernel virtual address: 22 type: "stack contents"
bt: read of stack at 22 failed
bt: invalid kernel virtual address: 13 type: "stack contents"
bt: read of stack at 13 failed
bt: invalid kernel virtual address: f type: "stack contents"
bt: read of stack at f failed
bt: invalid kernel virtual address: c type: "stack contents"
bt: read of stack at c failed
bt: invalid kernel virtual address: c type: "stack contents"
bt: read of stack at c failed
bt: invalid kernel virtual address: 13 type: "stack contents"
bt: read of stack at 13 failed
bt: invalid kernel virtual address: f type: "stack contents"
bt: read of stack at f failed
bt: invalid kernel virtual address: f type: "stack contents"
bt: read of stack at f failed
bt: invalid kernel virtual address: 11 type: "stack contents"
bt: read of stack at 11 failed
bt: invalid kernel virtual address: 10 type: "stack contents"
bt: read of stack at 10 failed
bt: invalid kernel virtual address: c type: "stack contents"
bt: read of stack at c failed
bt: invalid kernel virtual address: 14 type: "stack contents"
bt: read of stack at 14 failed
bt: invalid kernel virtual address: 13 type: "stack contents"
bt: read of stack at 13 failed
bt: invalid kernel virtual address: 18 type: "stack contents"
bt: read of stack at 18 failed
bt: invalid kernel virtual address: f type: "stack contents"
bt: read of stack at f failed
bt: invalid kernel virtual address: e type: "stack contents"
bt: read of stack at e failed
bt: invalid kernel virtual address: f type: "stack contents"
bt: read of stack at f failed
KERNEL: vmlinux-2.6.5-7.244-smp
DEBUG KERNEL: vmlinux.dbg (2.6.5-7.244-default)
DUMPFILE: DUMP10.1.230.112
CPUS: 1
DATE: Thu Jul 26 14:34:46 2007
UPTIME: 213503982284 days, 21:34:00
LOAD AVERAGE: 0.01, 0.12, 0.07
TASKS: 34
NODENAME: linux
RELEASE: 2.6.5-7.244-smp
VERSION: #1 SMP Mon Dec 12 18:32:25 UTC 2005
MACHINE: x86_64 (2793 Mhz)
MEMORY: 1015808 GB
PANIC: ""
PID: 0
COMMAND: "
"
TASK: ffffffff803d2800 (1 of 2) [THREAD_INFO: ffffffff80590000]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
WARNING: panic task not found
crash> bt
PID: 0 TASK: ffffffff803d2800 CPU: 0 COMMAND: "<80>^L"
#0 [ffffffff80591ef0] schedule at ffffffff801394e4
#1 [ffffffff80591f98] default_idle at ffffffff8010f1c0
#2 [ffffffff80591fc8] cpu_idle at ffffffff8010f65a
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
> 0 0 0 ffffffff803d2800 RU 0.0 4399648058624
4389578663200 [<80>^L]
0 0 0 ffffffff803d2808 ?? 0.0 0 0 [swapper]
1 0 0 10017e1f2c8 ?? 0.1 640 304 init
2 -1 0 10017e1e9a8 ?? 0.0 0 0 [migration/0]
3 -1 0 10017e1e088 ?? 0.0 0 0 [ksoftirqd/0]
4 -1 0 10001b712d8 ?? 0.0 0 0 [events/0]
5 -1 0 10001b709b8 ?? 0.0 0 0 [khelper]
6 -1 0 10001b70098 ?? 0.0 0 0 [kacpid]
25 -1 0 10017dd72e8 ?? 0.0 0 0 [kblockd/0]
47 -1 0 10017dd69c8 ?? 0.0 0 0 [pdflush]
48 -1 0 10017dd60a8 ?? 0.0 0 0 [pdflush]
49 -1 0 100178272f8 ?? 0.0 0 0 [kswapd0]
50 -1 0 100178269d8 ?? 0.0 0 0 [aio/0]
1295 -1 0 100178260b8 ?? 0.0 0 0 [kseriod]
2077 -1 0 10017897308 ?? 0.0 0 0 [reiserfs/0]
2744 -1 0 10014de9488 ?? 0.0 0 0 [khubd]
3077 -1 0 10015aa13c8 ?? 0.2 2560 608 hwscand
3693 -1 0 100164e1348 ?? 0.2 3568 816 syslogd
3696 -1 0 10015180208 ?? 0.3 2744 1112 klogd
3721 -1 0 10015b0eab8 ?? 0.2 3536 628 resmgrd
3722 -1 0 10015e6e1c8 ?? 0.2 4564 640 portmap
3803 -1 0 10015d49368 ?? 0.6 20036 2340 master
3814 -1 0 10015daea58 ?? 0.6 20100 2312 pickup
3815 -1 0 10016c5a0d8 ?? 0.6 20144 2364 qmgr
3861 -1 0 10016ca2a08 ?? 0.7 26800 2932 sshd
4022 -1 0 10014c42b48 ?? 0.2 6804 924 cron
4057 -1 0 100178960c8 ?? 0.2 2484 612 agetty
4058 -1 0 10016c5b318 ?? 0.5 21864 1772 login
4059 -1 0 10016ca3328 ?? 0.2 7012 936 mingetty
4060 -1 0 10015fb5398 ?? 0.2 7012 936 mingetty
4061 -1 0 10014cc6238 ?? 0.2 7012 936 mingetty
4062 -1 0 10015b0f3d8 ?? 0.2 7012 936 mingetty
4063 -1 0 100151e7458 ?? 0.2 7012 936 mingetty
4152 -1 0 10016a180f8 ?? 0.8 12716 2992 bash
crash> bt 10015180208
PID: 3696 TASK: 10015180208 CPU: 0 COMMAND: "klogd"
*bt: invalid kernel virtual address: 12 type: "stack contents"*
bt: read of stack at 12 failed
crash> task 10015180208
PID: 3696 TASK: 10015180208 CPU: 0 COMMAND: "klogd"
struct task_struct {
*state = 1099873050624,*
* thread_info = 0x12,*
usage = {
counter = 320
},
flags = 0,
ptrace = 502511173631,
lock_depth = 120,
prio = 0,
static_prio = 1048832,
run_list = {
next = 0x200200,
prev = 0x0
},
array = 0x50fe72e6,
sleep_avg = 1,
interactive_credit = 67616128664,
timestamp = 67616128664,
last_ran = 0,
activated = 0,
policy = 18446744073709551615,
cpus_allowed = 18446744073709551615,
time_slice = 150,
first_time_slice = 0,
tasks = {
next = 0x10015b0eb48,
prev = 0x100164e13d8
},
ptrace_children = {
next = 0x100151802a8,
prev = 0x100151802a8
},
ptrace_list = {
next = 0x100151802b8,
prev = 0x100151802b8
},
mm = 0x1001546c500,
active_mm = 0x1001546c500,
binfmt = 0xffffffff803e70c0,
exit_state = 0,
exit_code = 0,
exit_signal = 17,
pdeath_signal = 0,
personality = 0,
did_exec = 0,
pid = 3696,
tgid = 3696,
real_parent = 0x10017e1f2c0,
parent = 0x10017e1f2c0,
children = {
next = 0x10015180320,
prev = 0x10015180320
},
sibling = {
next = 0x10015b0ebe0,
prev = 0x100164e1470
},
group_leader = 0x10015180200,
pids = {{
pid_chain = {
next = 0x10015180370,
prev = 0x10015180370
},
pidptr = 0x10015180360,
pid = {
nr = 3696,
count = {
counter = 1
},
task = 0x10015180200,
task_list = {
next = 0x10015180348,
prev = 0x10015180348
},
hash_chain = {
next = 0x10017827470,
prev = 0x10016ca2b80
}
}
}, {
pid_chain = {
next = 0x100151803b8,
prev = 0x100151803b8
},
pidptr = 0x100151803a8,
pid = {
nr = 3696,
count = {
counter = 1
},
task = 0x10015180200,
task_list = {
next = 0x10015180390,
prev = 0x10015180390
},
hash_chain = {
next = 0x100178274b8,
prev = 0x10016ca2bc8
}
}
}, {
pid_chain = {
next = 0x10015180400,
prev = 0x10015180400
},
pidptr = 0x100151803f0,
pid = {
nr = 3696,
count = {
counter = 1
},
task = 0x10015180200,
task_list = {
next = 0x100151803d8,
prev = 0x100151803d8
},
hash_chain = {
next = 0x10001949240,
prev = 0x10016ca2c10
}
}
}, {
pid_chain = {
next = 0x10015180448,
prev = 0x10015180448
},
pidptr = 0x10015180438,
pid = {
nr = 3696,
count = {
counter = 1
},
task = 0x10015180200,
task_list = {
next = 0x10015180420,
prev = 0x10015180420
},
hash_chain = {
next = 0x10001949340,
prev = 0x10016ca2c58
}
}
}},
wait_chldexit = {
lock = {
lock = 1
},
task_list = {
next = 0x10015180470,
prev = 0x10015180470
}
},
vfork_done = 0x0,
set_child_tid = 0x2a95894b90,
clear_child_tid = 0x2a95894b90,
rt_priority = 0,
it_real_value = 0,
it_prof_value = 0,
it_virt_value = 0,
it_real_incr = 0,
it_prof_incr = 0,
it_virt_incr = 0,
real_timer = {
entry = {
next = 0x100100,
prev = 0x200200
},
expires = 29143,
lock = {
lock = 1
},
magic = 1267182958,
function = 0xffffffff80141b50 <it_real_fn>,
data = 1099865522688,
base = 0x0
},
utime = 0,
stime = 4,
cutime = 0,
cstime = 0,
nvcsw = 13,
nivcsw = 2,
cnvcsw = 0,
cnivcsw = 0,
start_time = 53888910424,
min_flt = 105,
maj_flt = 0,
cmin_flt = 0,
cmaj_flt = 0,
uid = 0,
euid = 0,
suid = 0,
fsuid = 0,
gid = 0,
egid = 0,
sgid = 0,
fsgid = 0,
group_info = 0xffffffff803e2a00,
cap_effective = 4294967039,
cap_inheritable = 0,
cap_permitted = 4294967039,
keep_capabilities = 0,
user = 0xffffffff803e29a0,
rlim = {{
rlim_cur = 18446744073709551615,
rlim_max = 18446744073709551615
}, {
rlim_cur = 18446744073709551615,
rlim_max = 18446744073709551615
}, {
rlim_cur = 18446744073709551615,
rlim_max = 18446744073709551615
}, {
rlim_cur = 8388608,
rlim_max = 18446744073709551615
}, {
rlim_cur = 0,
rlim_max = 18446744073709551615
}, {
rlim_cur = 18446744073709551615,
rlim_max = 18446744073709551615
}, {
rlim_cur = 3071,
rlim_max = 3071
}, {
rlim_cur = 1024,
rlim_max = 1024
}, {
rlim_cur = 18446744073709551615,
rlim_max = 18446744073709551615
}, {
rlim_cur = 18446744073709551615,
rlim_max = 18446744073709551615
}, {
rlim_cur = 18446744073709551615,
rlim_max = 18446744073709551615
}, {
rlim_cur = 1024,
rlim_max = 1024
}, {
rlim_cur = 819200,
rlim_max = 819200
}},
used_math = 0,
rcvd_sigterm = 0,
oomkilladj = 0,
comm = "klogd\000roc\000\000\000\000\000\000",
link_count = 0,
total_link_count = 0,
sysvsem = {
undo_list = 0x0
},
thread = {
rsp0 = 1099873058120,
rsp = 548682070920,
userrsp = 182897429248,
fs = 0,
gs = 0,
es = 0,
ds = 0,
fsindex = 0,
gsindex = 0,
debugreg0 = 0,
debugreg1 = 0,
debugreg2 = 0,
debugreg3 = 0,
debugreg6 = 0,
debugreg7 = 0,
cr2 = 0,
trap_no = 0,
error_code = 0,
i387 = {
fxsave = {
cwd = 0,
swd = 0,
twd = 0,
fop = 0,
rip = 0,
rdp = 281470681751424,
mxcsr = 0,
mxcsr_mask = 0,
st_space = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0},
xmm_space = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0
, 0, 0, 0},
padding = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0}
}
},
ioperm = 0,
io_bitmap_ptr = 0x0,
tls_array = {0, 0, 0}
},
fs = 0x10014c7a180,
files = 0x10001a114c0,
namespace = 0x100154fb900,
signal = 0x10015184600,
sighand = 0x0,
blocked = {
sig = {0}
},
real_blocked = {
sig = {1099865524632}
},
pending = {
list = {
next = 0x10015180998,
prev = 0x0
},
signal = {
sig = {0}
}
},
sas_ss_sp = 0,
sas_ss_size = 0,
notifier = 0,
notifier_data = 0x0,
notifier_mask = 0x0,
security = 0x600000005,
parent_exec_id = 1,
self_exec_id = 1,
alloc_lock = {
lock = 1
},
proc_lock = {
lock = 0
},
switch_lock = {
lock = 0
},
journal_info = 0x0,
reclaim_state = 0x10015469180,
proc_dentry = 0x0,
backing_dev_info = 0x10015b40940,
io_context = 0x0,
ptrace_message = 0,
last_siginfo = 0x0,
io_wait = 0xac9,
rchar = 2292,
wchar = 3,
syscr = 32,
syscw = 475,
acct_rss_mem1 = 2743,
acct_vm_mem1 = 4,
acct_stimexpd = 4294967297,
ckrm_tsklock = {
lock = 0
},
ckrm_celock = {
lock = 0
},
ce_data = 0xffffffff804f3f20,
taskclass = 0x100164e1bc8,
taskclass_link = {
next = 0x10015b0f338,
prev = 0xffffffff80537940
},
cpu_class = 0x0,
demand_stat = {
run = 0,
total = 61218488692,
last_sleep = 32000000,
recalc_interval = 0,
cpu_demand = 105133020
},
delays = {
waitcpu_total = 3647587,
runcpu_total = 23603870,
iowait_total = 0,
mem_iowait_total = 4294967311,
runs = 0,
num_iowaits = 0,
num_memwaits = 0,
splpar_total = 1431654400
},
map_base = 0,
mempolicy = 0x0,
il_next = 0,
audit = 0x10
}
crash>
More information about the Crash-utility
mailing list