[Crash-utility] Unable to switch stack frames while using crash
Dave Anderson
anderson at redhat.com
Fri Jun 24 13:40:24 UTC 2011
---- Original Message -----
>>
>> BTW, are you sure about that?
>>
>> Presuming that the "tty" pointer is ffff8802cbd54800 as you've shown below,
>> and therefore tty->read_buf is 0xffff8802cbfe6000 and tty->read_tail is 0,
>> then the statement above would be simply be reading tty->read_buf[0], or
>> virtual address 0xffff8802cbfe6000. But the oops shows it faulting on a
>> virtual address of "5":
Well, as it turns out, you have every reason to be sure about that...
Anyway, I don't understand why line numbers are not available with
your kernel:
> crash> dis -rl n_tty_read+0x58c
> dis: line numbers are not available
... [ cut ] ...
> 0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
> 0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
> 0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
> 0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
> 0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx
But nonetheless, there is only on movsbl instruction in n_tty_read(),
and looking at a RHEL6 kernel, you were correct in your original
determination of the faulting instruction:
crash> dis n_tty_read | grep movsbl
0xffffffff812f88c9 <n_tty_read+0x2c9>: movsbl (%rdx,%rax,1),%ebx
crash> dis -rl 0xffffffff812f88c9 | tail
... [ cut ] ...
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/drivers/char/n_tty.c: 1821
0xffffffff812f88c9 <n_tty_read+0x2c9>: movsbl (%rdx,%rax,1),%ebx
crash>
which is here:
1814 if (tty->icanon) {
1815 /* N.B. avoid overrun if nr == 0 */
1816 while (nr && tty->read_cnt) {
1817 int eol;
1818
1819 eol = test_and_clear_bit(tty->read_tail,
1820 tty->read_flags);
1821 c = tty->read_buf[tty->read_tail];
The tty_struct offsets are these:
crash> tty_struct -o
struct tty_struct {
... [ cut ]...
[0x250] char *read_buf;
[0x258] int read_head;
[0x25c] int read_tail;
...
And you can see in the previous instructions the tty->read_buf (0x250)
and tty->read_tail (0x25c) offsets being added to the tty_struct
base address that is located in R13:
> 0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
> 0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
> 0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
> 0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
> 0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx
And as you originally reported, the tty_struct address in R13
is ffff8802cbd54800:
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#3 [ffff88031ce75b50] no_context at ffffffff8102d801
#4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#5 [ffff88031ce75c70] bad_area at ffffffff8102da41
#6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
[exception RIP: n_tty_read+1420]
RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#8 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#9 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#10 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#11 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
But for whatever reason -- and I cannot explain it -- after these
instructions were done:
> 0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
> 0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
the RDX register ended up with 0000000000000005, and the RAX register with
a 0000000000000000, leading to the:
BUG: unable to handle kernel NULL pointer dereference at 0000000000000005
But when you display the tty_struct at ffff8802cbd54800, you see the
read_buf and read_tail with seemingly legitimate values:
crash> tty_struct ffff8802cbd54800
struct tty_struct { ...
magic = 21505,
driver = 0xffff88031b54ea00,
ops = 0xffffffff8130f650,
name = "pts9\000\...",
driver_data = 0xffff88029c8a9668,
icanon = 1 '\001',
read_buf = 0xffff8802cbfe6000 "",
read_head = 0,
read_tail = 0,
read_cnt = 0,
...
So everything in your analysis was correct, but how it is possible
that the RDX and RAX registers to have ended up with 0 and 5 is hard
to explain. And for that matter, since tty->read_cnt is 0 above,
your original question as to how that code path was taken to
begin with is also valid.
So I don't know -- anybody on the list ever seen anything like this?
Stumped,
Dave
----- Original Message -----
> Hi Dave,
>
> Thank you so much for your help.
>
> Below is the output of dis -rl n_tty_read+0x58c
>
> crash> dis -rl n_tty_read+0x58c
> dis: line numbers are not available
> 0xffffffff811efe27 <n_tty_read>: push %rbp
> 0xffffffff811efe28 <n_tty_read+1>: mov %gs:0xb500,%rax
> 0xffffffff811efe31 <n_tty_read+10>: mov %rsp,%rbp
> 0xffffffff811efe34 <n_tty_read+13>: push %r15
> 0xffffffff811efe36 <n_tty_read+15>: push %r14
> 0xffffffff811efe38 <n_tty_read+17>: push %r13
> 0xffffffff811efe3a <n_tty_read+19>: mov %rdi,%r13
> 0xffffffff811efe3d <n_tty_read+22>: lea -0x70(%rbp),%rdi
> 0xffffffff811efe41 <n_tty_read+26>: push %r12
> 0xffffffff811efe43 <n_tty_read+28>: push %rbx
> 0xffffffff811efe44 <n_tty_read+29>: lea 0x490(%r13),%rbx
> 0xffffffff811efe4b <n_tty_read+36>: sub $0xe8,%rsp
> 0xffffffff811efe52 <n_tty_read+43>: mov %rax,-0x98(%rbp)
> 0xffffffff811efe59 <n_tty_read+50>: mov %rcx,-0x78(%rbp)
> 0xffffffff811efe5d <n_tty_read+54>: xor %eax,%eax
> 0xffffffff811efe5f <n_tty_read+56>: mov $0xa,%ecx
> 0xffffffff811efe64 <n_tty_read+61>: mov %rdx,-0xd8(%rbp)
> 0xffffffff811efe6b <n_tty_read+68>: mov %rsi,-0xd0(%rbp)
> 0xffffffff811efe72 <n_tty_read+75>: mov %rdx,-0x40(%rbp)
> 0xffffffff811efe76 <n_tty_read+79>: rep stos %eax,%es:(%rdi)
> 0xffffffff811efe78 <n_tty_read+81>: lea 0x1c0(%r13),%rax
> 0xffffffff811efe7f <n_tty_read+88>: lea 0x1c8(%r13),%rcx
> 0xffffffff811efe86 <n_tty_read+95>: mov %rbx,-0xc0(%rbp)
> 0xffffffff811efe8d <n_tty_read+102>: lea 0xd8(%r13),%rbx
> 0xffffffff811efe94 <n_tty_read+109>: movq
> $0xffffffff81045f84,-0x60(%rbp)
> 0xffffffff811efe9c <n_tty_read+117>: movq $0x0,-0xa8(%rbp)
> 0xffffffff811efea7 <n_tty_read+128>: mov -0x98(%rbp),%rdx
> 0xffffffff811efeae <n_tty_read+135>: mov %rax,-0xc8(%rbp)
> 0xffffffff811efeb5 <n_tty_read+142>: mov -0x98(%rbp),%rax
> 0xffffffff811efebc <n_tty_read+149>: mov %rcx,-0x90(%rbp)
> 0xffffffff811efec3 <n_tty_read+156>: lea 0x51c(%r13),%rcx
> 0xffffffff811efeca <n_tty_read+163>: mov %rbx,-0x80(%rbp)
> 0xffffffff811efece <n_tty_read+167>: mov %rdx,-0x68(%rbp)
> 0xffffffff811efed2 <n_tty_read+171>: lea 0x268(%r13),%rdx
> 0xffffffff811efed9 <n_tty_read+178>: mov %rcx,-0xb8(%rbp)
> 0xffffffff811efee0 <n_tty_read+185>: mov %rax,-0xf8(%rbp)
> 0xffffffff811efee7 <n_tty_read+192>: mov %rax,-0x100(%rbp)
> 0xffffffff811efeee <n_tty_read+199>: mov %rdx,-0x88(%rbp)
> 0xffffffff811efef5 <n_tty_read+206>: mov %rax,-0x108(%rbp)
> 0xffffffff811efefc <n_tty_read+213>: mov %rax,-0x110(%rbp)
> 0xffffffff811eff03 <n_tty_read+220>: cmpq $0x0,0x250(%r13)
> 0xffffffff811eff0b <n_tty_read+228>: jne 0xffffffff811eff11
> <n_tty_read+234>
> 0xffffffff811eff0d <n_tty_read+230>: ud2a
> 0xffffffff811eff0f <n_tty_read+232>: jmp 0xffffffff811eff0f
> <n_tty_read+232>
> 0xffffffff811eff11 <n_tty_read+234>: mov -0xd0(%rbp),%rdx
> 0xffffffff811eff18 <n_tty_read+241>: mov 0x20(%rdx),%rax
> 0xffffffff811eff1c <n_tty_read+245>: cmpq
> $0xffffffff811ed61f,0x18(%rax)
> 0xffffffff811eff24 <n_tty_read+253>: je 0xffffffff811effef
> <n_tty_read+456>
> 0xffffffff811eff2a <n_tty_read+259>: mov -0xf8(%rbp),%rcx
> 0xffffffff811eff31 <n_tty_read+266>: mov 0x478(%rcx),%rax
> 0xffffffff811eff38 <n_tty_read+273>: cmp %r13,0x180(%rax)
> 0xffffffff811eff3f <n_tty_read+280>: jne 0xffffffff811effef
> <n_tty_read+456>
> 0xffffffff811eff45 <n_tty_read+286>: mov 0xc8(%r13),%rdx
> 0xffffffff811eff4c <n_tty_read+293>: test %rdx,%rdx
> 0xffffffff811eff4f <n_tty_read+296>: jne 0xffffffff811eff64
> <n_tty_read+317>
> 0xffffffff811eff51 <n_tty_read+298>: mov $0xffffffff8139c972,%rdi
> 0xffffffff811eff58 <n_tty_read+305>: xor %eax,%eax
> 0xffffffff811eff5a <n_tty_read+307>: callq 0xffffffff812d4abf
> <printk>
> 0xffffffff811eff5f <n_tty_read+312>: jmpq 0xffffffff811effef
> <n_tty_read+456>
> 0xffffffff811eff64 <n_tty_read+317>: mov -0xf8(%rbp),%rbx
> 0xffffffff811eff6b <n_tty_read+324>: mov 0x1e0(%rbx),%rax
> 0xffffffff811eff72 <n_tty_read+331>: cmp %rdx,0x238(%rax)
> 0xffffffff811eff79 <n_tty_read+338>: je 0xffffffff811effef
> <n_tty_read+456>
> 0xffffffff811eff7b <n_tty_read+340>: mov -0x98(%rbp),%rax
> 0xffffffff811eff82 <n_tty_read+347>: testb $0x10,0x48a(%rax)
> 0xffffffff811eff89 <n_tty_read+354>: jne 0xffffffff811f0611
> <n_tty_read+2026>
> 0xffffffff811eff8f <n_tty_read+360>: mov 0x480(%rax),%rax
> 0xffffffff811eff96 <n_tty_read+367>: cmpq $0x1,0x288(%rax)
> 0xffffffff811eff9e <n_tty_read+375>: jne 0xffffffff811f0604
> <n_tty_read+2013>
> 0xffffffff811effa4 <n_tty_read+381>: jmpq 0xffffffff811f0611
> <n_tty_read+2026>
> 0xffffffff811effa9 <n_tty_read+386>: mov -0x98(%rbp),%rcx
> 0xffffffff811effb0 <n_tty_read+393>: mov $0x1,%edx
> 0xffffffff811effb5 <n_tty_read+398>: mov $0x15,%esi
> 0xffffffff811effba <n_tty_read+403>: mov 0x1e0(%rcx),%rax
> 0xffffffff811effc1 <n_tty_read+410>: mov 0x238(%rax),%rdi
> 0xffffffff811effc8 <n_tty_read+417>: callq 0xffffffff8105953a
> <kill_pgrp>
> 0xffffffff811effcd <n_tty_read+422>: mov %gs:0xb508,%rdx
> 0xffffffff811effd6 <n_tty_read+431>: lea -0x1fc8(%rdx),%rax
> 0xffffffff811effdd <n_tty_read+438>: lock orb $0x4,-0x1fc8(%rdx)
> 0xffffffff811effe5 <n_tty_read+446>: mov $0xfffffe00,%eax
> 0xffffffff811effea <n_tty_read+451>: jmpq 0xffffffff811f0616
> <n_tty_read+2031>
> 0xffffffff811effef <n_tty_read+456>: testb $0x10,0x21c(%r13)
> 0xffffffff811efff7 <n_tty_read+464>: je 0xffffffff811f000f
> <n_tty_read+488>
> 0xffffffff811efff9 <n_tty_read+466>: movl $0x0,-0xb0(%rbp)
> 0xffffffff811f0003 <n_tty_read+476>: movl $0x0,-0xac(%rbp)
> 0xffffffff811f000d <n_tty_read+486>: jmp 0xffffffff811f0076
> <n_tty_read+591>
> 0xffffffff811f000f <n_tty_read+488>: mov 0x70(%r13),%rdx
> 0xffffffff811f0013 <n_tty_read+492>: movzbl 0x16(%rdx),%eax
> 0xffffffff811f0017 <n_tty_read+496>: imul $0x19,%eax,%eax
> 0xffffffff811f001a <n_tty_read+499>: mov %eax,-0xac(%rbp)
> 0xffffffff811f0020 <n_tty_read+505>: movzbl 0x17(%rdx),%edx
> 0xffffffff811f0024 <n_tty_read+509>: test %edx,%edx
> 0xffffffff811f0026 <n_tty_read+511>: mov %edx,-0xb0(%rbp)
> 0xffffffff811f002c <n_tty_read+517>: je 0xffffffff811f0082
> <n_tty_read+603>
> 0xffffffff811f002e <n_tty_read+519>: test %eax,%eax
> 0xffffffff811f0030 <n_tty_read+521>: je 0xffffffff811f003e
> <n_tty_read+535>
> 0xffffffff811f0032 <n_tty_read+523>: movw $0x1,0x21e(%r13)
> 0xffffffff811f003c <n_tty_read+533>: jmp 0xffffffff811f0076
> <n_tty_read+591>
> 0xffffffff811f003e <n_tty_read+535>: mov -0x90(%rbp),%rbx
> 0xffffffff811f0045 <n_tty_read+542>: cmp %rbx,0x1c8(%r13)
> 0xffffffff811f004c <n_tty_read+549>: je 0xffffffff811f0068
> <n_tty_read+577>
> 0xffffffff811f004e <n_tty_read+551>: movzwl 0x21e(%r13),%eax
> 0xffffffff811f0056 <n_tty_read+559>: mov $0x7fffffffffffffff,%r15
> 0xffffffff811f0060 <n_tty_read+569>: cmp -0xb0(%rbp),%eax
> 0xffffffff811f0066 <n_tty_read+575>: jle 0xffffffff811f00b7
> <n_tty_read+656>
> 0xffffffff811f0068 <n_tty_read+577>: mov -0xb0(%rbp),%eax
> 0xffffffff811f006e <n_tty_read+583>: mov %ax,0x21e(%r13)
> 0xffffffff811f0076 <n_tty_read+591>: mov $0x7fffffffffffffff,%r15
> 0xffffffff811f0080 <n_tty_read+601>: jmp 0xffffffff811f00b7
> <n_tty_read+656>
> 0xffffffff811f0082 <n_tty_read+603>: movslq -0xac(%rbp),%r15
> 0xffffffff811f0089 <n_tty_read+610>: cmpl $0x0,-0xac(%rbp)
> 0xffffffff811f0090 <n_tty_read+617>: mov $0x0,%eax
> 0xffffffff811f0095 <n_tty_read+622>: movw $0x1,0x21e(%r13)
> 0xffffffff811f009f <n_tty_read+632>: movl $0x1,-0xb0(%rbp)
> 0xffffffff811f00a9 <n_tty_read+642>: movl $0x0,-0xac(%rbp)
> 0xffffffff811f00b3 <n_tty_read+652>: cmove %rax,%r15
> 0xffffffff811f00b7 <n_tty_read+656>: mov -0xd0(%rbp),%rdx
> 0xffffffff811f00be <n_tty_read+663>: testb $0x8,0x39(%rdx)
> 0xffffffff811f00c2 <n_tty_read+667>: je 0xffffffff811f00e4
> <n_tty_read+701>
> 0xffffffff811f00c4 <n_tty_read+669>: mov -0xc0(%rbp),%rdi
> 0xffffffff811f00cb <n_tty_read+676>: callq 0xffffffff812d5ec7
> <mutex_trylock>
> 0xffffffff811f00d0 <n_tty_read+681>: test %eax,%eax
> 0xffffffff811f00d2 <n_tty_read+683>: jne 0xffffffff811f0104
> <n_tty_read+733>
> 0xffffffff811f00d4 <n_tty_read+685>: movq
> $0xfffffffffffffff5,-0xa8(%rbp)
> 0xffffffff811f00df <n_tty_read+696>: jmpq 0xffffffff811f05eb
> <n_tty_read+1988>
> 0xffffffff811f00e4 <n_tty_read+701>: mov -0xc0(%rbp),%rdi
> 0xffffffff811f00eb <n_tty_read+708>: callq 0xffffffff812d6358
> <mutex_lock_interruptible>
> 0xffffffff811f00f0 <n_tty_read+713>: test %eax,%eax
> 0xffffffff811f00f2 <n_tty_read+715>: je 0xffffffff811f0104
> <n_tty_read+733>
> 0xffffffff811f00f4 <n_tty_read+717>: movq
> $0xfffffffffffffe00,-0xa8(%rbp)
> 0xffffffff811f00ff <n_tty_read+728>: jmpq 0xffffffff811f05eb
> <n_tty_read+1988>
> 0xffffffff811f0104 <n_tty_read+733>: mov 0xec(%r13),%al
> 0xffffffff811f010b <n_tty_read+740>: mov -0xc8(%rbp),%rdi
> 0xffffffff811f0112 <n_tty_read+747>: lea -0x70(%rbp),%rsi
> 0xffffffff811f0116 <n_tty_read+751>: shr $0x3,%al
> 0xffffffff811f0119 <n_tty_read+754>: mov %eax,%ecx
> 0xffffffff811f011b <n_tty_read+756>: and $0x1,%ecx
> 0xffffffff811f011e <n_tty_read+759>: mov %ecx,-0x9c(%rbp)
> 0xffffffff811f0124 <n_tty_read+765>: callq 0xffffffff8106201b
> <add_wait_queue>
> 0xffffffff811f0129 <n_tty_read+770>: movslq -0xb0(%rbp),%rbx
> 0xffffffff811f0130 <n_tty_read+777>: movslq -0xac(%rbp),%rax
> 0xffffffff811f0137 <n_tty_read+784>: mov -0xd8(%rbp),%rdx
> 0xffffffff811f013e <n_tty_read+791>: inc %rdx
> 0xffffffff811f0141 <n_tty_read+794>: mov %rbx,-0xe0(%rbp)
> 0xffffffff811f0148 <n_tty_read+801>: mov %rax,-0xe8(%rbp)
> 0xffffffff811f014f <n_tty_read+808>: mov %rdx,-0xf0(%rbp)
> 0xffffffff811f0156 <n_tty_read+815>: jmpq 0xffffffff811f0522
> <n_tty_read+1787>
> 0xffffffff811f015b <n_tty_read+820>: cmpl $0x0,-0x9c(%rbp)
> 0xffffffff811f0162 <n_tty_read+827>: je 0xffffffff811f01ef
> <n_tty_read+968>
> 0xffffffff811f0168 <n_tty_read+833>: mov 0xf8(%r13),%rax
> 0xffffffff811f016f <n_tty_read+840>: cmpb $0x0,0xed(%rax)
> 0xffffffff811f0176 <n_tty_read+847>: je 0xffffffff811f01ef
> <n_tty_read+968>
> 0xffffffff811f0178 <n_tty_read+849>: mov -0xd8(%rbp),%rcx
> 0xffffffff811f017f <n_tty_read+856>: cmp %rcx,-0x40(%rbp)
> 0xffffffff811f0183 <n_tty_read+860>: jne 0xffffffff811f052d
> <n_tty_read+1798>
> 0xffffffff811f0189 <n_tty_read+866>: lea 0x68(%rax),%rdi
> 0xffffffff811f018d <n_tty_read+870>: callq 0xffffffff812d6fb8
> <_spin_lock_irqsave>
> 0xffffffff811f0192 <n_tty_read+875>: mov 0xf8(%r13),%rdi
> 0xffffffff811f0199 <n_tty_read+882>: mov %rax,%rsi
> 0xffffffff811f019c <n_tty_read+885>: mov 0xed(%rdi),%bl
> 0xffffffff811f01a2 <n_tty_read+891>: movb $0x0,0xed(%rdi)
> 0xffffffff811f01a9 <n_tty_read+898>: add $0x68,%rdi
> 0xffffffff811f01ad <n_tty_read+902>: callq 0xffffffff812d70c1
> <_spin_unlock_irqrestore>
> 0xffffffff811f01b2 <n_tty_read+907>: mov -0x40(%rbp),%r12
> 0xffffffff811f01b6 <n_tty_read+911>: lea -0x31(%rbp),%rsi
> 0xffffffff811f01ba <n_tty_read+915>: mov $0x1,%edx
> 0xffffffff811f01bf <n_tty_read+920>: mov %r13,%rdi
> 0xffffffff811f01c2 <n_tty_read+923>: mov %bl,-0x31(%rbp)
> 0xffffffff811f01c5 <n_tty_read+926>: lea 0x1(%r12),%rax
> 0xffffffff811f01ca <n_tty_read+931>: mov %rax,-0x40(%rbp)
> 0xffffffff811f01ce <n_tty_read+935>: callq 0xffffffff812008ac
> <tty_audit_add_data>
> 0xffffffff811f01d3 <n_tty_read+940>: mov -0x31(%rbp),%al
> 0xffffffff811f01d6 <n_tty_read+943>: mov %r12,%rcx
> 0xffffffff811f01d9 <n_tty_read+946>: callq 0xffffffff811949a0
> <__put_user_1>
> 0xffffffff811f01de <n_tty_read+951>: test %eax,%eax
> 0xffffffff811f01e0 <n_tty_read+953>: jne 0xffffffff811f043d
> <n_tty_read+1558>
> 0xffffffff811f01e6 <n_tty_read+959>: decq -0x78(%rbp)
> 0xffffffff811f01ea <n_tty_read+963>: jmpq 0xffffffff811f052d
> <n_tty_read+1798>
> 0xffffffff811f01ef <n_tty_read+968>: mov $0x1,%eax
> 0xffffffff811f01f4 <n_tty_read+973>: mov -0x100(%rbp),%rbx
> 0xffffffff811f01fb <n_tty_read+980>: xchg %rax,(%rbx)
> 0xffffffff811f01fe <n_tty_read+983>: mov -0x40(%rbp),%rcx
> 0xffffffff811f0202 <n_tty_read+987>: mov -0xd8(%rbp),%rax
> 0xffffffff811f0209 <n_tty_read+994>: mov -0xe0(%rbp),%rbx
> 0xffffffff811f0210 <n_tty_read+1001>: sub %rcx,%rax
> 0xffffffff811f0213 <n_tty_read+1004>: lea (%rax,%rbx,1),%rdx
> 0xffffffff811f0217 <n_tty_read+1008>: movzwl 0x21e(%r13),%eax
> 0xffffffff811f021f <n_tty_read+1016>: cmp %rax,%rdx
> 0xffffffff811f0222 <n_tty_read+1019>: jge 0xffffffff811f0240
> <n_tty_read+1049>
> 0xffffffff811f0224 <n_tty_read+1021>: test %rdx,%rdx
> 0xffffffff811f0227 <n_tty_read+1024>: jle 0xffffffff811f0240
> <n_tty_read+1049>
> 0xffffffff811f0229 <n_tty_read+1026>: mov -0xd8(%rbp),%eax
> 0xffffffff811f022f <n_tty_read+1032>: sub %cx,%ax
> 0xffffffff811f0232 <n_tty_read+1035>: add -0xb0(%rbp),%eax
> 0xffffffff811f0238 <n_tty_read+1041>: mov %ax,0x21e(%r13)
> 0xffffffff811f0240 <n_tty_read+1049>: mov %r13,%rdi
> 0xffffffff811f0243 <n_tty_read+1052>: callq 0xffffffff811f37f3
> <tty_flush_to_ldisc>
> 0xffffffff811f0248 <n_tty_read+1057>: testb $0x10,0x21c(%r13)
> 0xffffffff811f0250 <n_tty_read+1065>: je 0xffffffff811f0261
> <n_tty_read+1082>
> 0xffffffff811f0252 <n_tty_read+1067>: cmpl $0x0,0x478(%r13)
> 0xffffffff811f025a <n_tty_read+1075>: jne 0xffffffff811f026f
> <n_tty_read+1096>
> 0xffffffff811f025c <n_tty_read+1077>: jmpq 0xffffffff811f0621
> <n_tty_read+2042>
> 0xffffffff811f0261 <n_tty_read+1082>: cmpl $0x0,0x260(%r13)
> 0xffffffff811f0269 <n_tty_read+1090>: jle 0xffffffff811f0621
> <n_tty_read+2042>
> 0xffffffff811f026f <n_tty_read+1096>: mov -0x110(%rbp),%rax
> 0xffffffff811f0276 <n_tty_read+1103>: movq $0x0,(%rax)
> 0xffffffff811f027d <n_tty_read+1110>: cmpl $0x0,-0x9c(%rbp)
> 0xffffffff811f0284 <n_tty_read+1117>: mov -0x40(%rbp),%rax
> 0xffffffff811f0288 <n_tty_read+1121>: je 0xffffffff811f0376
> <n_tty_read+1359>
> 0xffffffff811f028e <n_tty_read+1127>: cmp -0xd8(%rbp),%rax
> 0xffffffff811f0295 <n_tty_read+1134>: jne 0xffffffff811f0376
> <n_tty_read+1359>
> 0xffffffff811f029b <n_tty_read+1140>: jmpq 0xffffffff811f033b
> <n_tty_read+1300>
> 0xffffffff811f02a0 <n_tty_read+1145>: mov -0xd0(%rbp),%rdi
> 0xffffffff811f02a7 <n_tty_read+1152>: callq 0xffffffff811eb980
> <tty_hung_up_p>
> 0xffffffff811f02ac <n_tty_read+1157>: test %eax,%eax
> 0xffffffff811f02ae <n_tty_read+1159>: jne 0xffffffff811f052d
> <n_tty_read+1798>
> 0xffffffff811f02b4 <n_tty_read+1165>: test %r15,%r15
> 0xffffffff811f02b7 <n_tty_read+1168>: je 0xffffffff811f052d
> <n_tty_read+1798>
> 0xffffffff811f02bd <n_tty_read+1174>: mov -0xd0(%rbp),%rdx
> 0xffffffff811f02c4 <n_tty_read+1181>: testb $0x8,0x39(%rdx)
> 0xffffffff811f02c8 <n_tty_read+1185>: je 0xffffffff811f02da
> <n_tty_read+1203>
> 0xffffffff811f02ca <n_tty_read+1187>: movq
> $0xfffffffffffffff5,-0xa8(%rbp)
> 0xffffffff811f02d5 <n_tty_read+1198>: jmpq 0xffffffff811f052d
> <n_tty_read+1798>
> 0xffffffff811f02da <n_tty_read+1203>: mov -0x108(%rbp),%rcx
> 0xffffffff811f02e1 <n_tty_read+1210>: mov 0x8(%rcx),%rax
> 0xffffffff811f02e5 <n_tty_read+1214>: testb $0x4,0x10(%rax)
> 0xffffffff811f02e9 <n_tty_read+1218>: je 0xffffffff811f02fb
> <n_tty_read+1236>
> 0xffffffff811f02eb <n_tty_read+1220>: movq
> $0xfffffffffffffe00,-0xa8(%rbp)
> 0xffffffff811f02f6 <n_tty_read+1231>: jmpq 0xffffffff811f052d
> <n_tty_read+1798>
> 0xffffffff811f02fb <n_tty_read+1236>: mov $0xfff,%eax
> 0xffffffff811f0300 <n_tty_read+1241>: sub 0x260(%r13),%eax
> 0xffffffff811f0307 <n_tty_read+1248>: test %eax,%eax
> 0xffffffff811f0309 <n_tty_read+1250>: jg 0xffffffff811f0324
> <n_tty_read+1277>
> 0xffffffff811f030b <n_tty_read+1252>: xor %eax,%eax
> 0xffffffff811f030d <n_tty_read+1254>: testb $0x10,0x21c(%r13)
> 0xffffffff811f0315 <n_tty_read+1262>: je 0xffffffff811f0324
> <n_tty_read+1277>
> 0xffffffff811f0317 <n_tty_read+1264>: xor %eax,%eax
> 0xffffffff811f0319 <n_tty_read+1266>: cmpl $0x0,0x478(%r13)
> 0xffffffff811f0321 <n_tty_read+1274>: sete %al
> 0xffffffff811f0324 <n_tty_read+1277>: mov %r15,%rdi
> 0xffffffff811f0327 <n_tty_read+1280>: mov %eax,0xf0(%r13)
> 0xffffffff811f032e <n_tty_read+1287>: callq 0xffffffff812d5a02
> <schedule_timeout>
> 0xffffffff811f0333 <n_tty_read+1292>: mov %rax,%r15
> 0xffffffff811f0336 <n_tty_read+1295>: jmpq 0xffffffff811f0522
> <n_tty_read+1787>
> 0xffffffff811f033b <n_tty_read+1300>: mov -0xf0(%rbp),%rbx
> 0xffffffff811f0342 <n_tty_read+1307>: lea -0x31(%rbp),%rsi
> 0xffffffff811f0346 <n_tty_read+1311>: mov $0x1,%edx
> 0xffffffff811f034b <n_tty_read+1316>: mov %r13,%rdi
> 0xffffffff811f034e <n_tty_read+1319>: movb $0x0,-0x31(%rbp)
> 0xffffffff811f0352 <n_tty_read+1323>: mov %rbx,-0x40(%rbp)
> 0xffffffff811f0356 <n_tty_read+1327>: callq 0xffffffff812008ac
> <tty_audit_add_data>
> 0xffffffff811f035b <n_tty_read+1332>: mov -0x31(%rbp),%al
> 0xffffffff811f035e <n_tty_read+1335>: mov -0xd8(%rbp),%rcx
> 0xffffffff811f0365 <n_tty_read+1342>: callq 0xffffffff811949a0
> <__put_user_1>
> 0xffffffff811f036a <n_tty_read+1347>: test %eax,%eax
> 0xffffffff811f036c <n_tty_read+1349>: jne 0xffffffff811f043d
> <n_tty_read+1558>
> 0xffffffff811f0372 <n_tty_read+1355>: decq -0x78(%rbp)
> 0xffffffff811f0376 <n_tty_read+1359>: testb $0x10,0x21c(%r13)
> 0xffffffff811f037e <n_tty_read+1367>: jne 0xffffffff811f0456
> <n_tty_read+1583>
> 0xffffffff811f0384 <n_tty_read+1373>: jmpq 0xffffffff811f047a
> <n_tty_read+1619>
> 0xffffffff811f0389 <n_tty_read+1378>: mov 0x25c(%r13),%eax
> 0xffffffff811f0390 <n_tty_read+1385>: mov -0x88(%rbp),%rbx
> 0xffffffff811f0397 <n_tty_read+1392>: lock btr %eax,(%rbx)
> 0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
> 0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
> 0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
> 0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
> 0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx
>
> Below is the output of bt -a command in crash
>
> bt -a
> PID: 0 TASK: ffffffff814204b0 CPU: 0 COMMAND: "swapper"
> #0 [ffff880033007e80] crash_nmi_callback at ffffffff8101fbc9
> #1 [ffff880033007e90] notifier_call_chain at ffffffff81065893
> #2 [ffff880033007ed0] atomic_notifier_call_chain at ffffffff810658dd
> #3 [ffff880033007ee0] notify_die at ffffffff8106597f
> #4 [ffff880033007f10] do_nmi at ffffffff8100dc5d
> #5 [ffff880033007f50] nmi at ffffffff812d76b0
> [exception RIP: mwait_idle+163]
> RIP: ffffffff81013029 RSP: ffffffff813e3eb8 RFLAGS: 00000246
> RAX: 0000000000000000 RBX: ffffffff813e3fd8 RCX: 0000000000000000
> RDX: 0000000000000000 RSI: ffffffff813e3fd8 RDI: ffffffff81522308
> RBP: ffffffff813e3ec8 R8: 0000000000000000 R9: ffff88003306e290
> R10: 0000000000012d80 R11: 0000000000000000 R12: ffffffff8147e368
> R13: ffffffff814ccb30 R14: ffffffff814cdfa0 R15: ffffffff813e3fa8
> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
> --- <NMI exception stack> ---
> #6 [ffffffff813e3eb8] mwait_idle at ffffffff81013029
> #7 [ffffffff813e3ed0] cpu_idle at ffffffff8100af21
>
> PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
> #0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
> #1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
> #2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
> #3 [ffff88031ce75b50] no_context at ffffffff8102d801
> #4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
> #5 [ffff88031ce75c70] bad_area at ffffffff8102da41
> #6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
> #7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
> [exception RIP: n_tty_read+1420]
> RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
> RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
> RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
> RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
> R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
> R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
> #8 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
> #9 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
> #10 [ffff88031ce75f40] sys_read at ffffffff810ebe48
> #11 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
> RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
> RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
> RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
> RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
> R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
> R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
> ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
>
> PID: 0 TASK: ffff88031e0e3540 CPU: 2 COMMAND: "swapper"
> #0 [ffff880033047e80] crash_nmi_callback at ffffffff8101fbc9
> #1 [ffff880033047e90] notifier_call_chain at ffffffff81065893
> #2 [ffff880033047ed0] atomic_notifier_call_chain at ffffffff810658dd
> #3 [ffff880033047ee0] notify_die at ffffffff8106597f
> #4 [ffff880033047f10] do_nmi at ffffffff8100dc5d
> #5 [ffff880033047f50] nmi at ffffffff812d76b0
> [exception RIP: mwait_idle+163]
> RIP: ffffffff81013029 RSP: ffff88031e0e5ef8 RFLAGS: 00000246
> RAX: 0000000000000000 RBX: ffff88031e0e5fd8 RCX: 0000000000000000
> RDX: 0000000000000000 RSI: ffff88031e0e5fd8 RDI: ffffffff81522308
> RBP: ffff88031e0e5f08 R8: 0000000000000000 R9: ffff88003302e290
> R10: 0000000000012d80 R11: 0000000000000000 R12: ffffffff8147e368
> R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
> --- <NMI exception stack> ---
> #6 [ffff88031e0e5ef8] mwait_idle at ffffffff81013029
> #7 [ffff88031e0e5f10] cpu_idle at ffffffff8100af21
>
> PID: 0 TASK: ffff88031e113580 CPU: 3 COMMAND: "swapper"
> #0 [ffff880033067e80] crash_nmi_callback at ffffffff8101fbc9
> #1 [ffff880033067e90] notifier_call_chain at ffffffff81065893
> #2 [ffff880033067ed0] atomic_notifier_call_chain at ffffffff810658dd
> #3 [ffff880033067ee0] notify_die at ffffffff8106597f
> #4 [ffff880033067f10] do_nmi at ffffffff8100dc5d
> #5 [ffff880033067f50] nmi at ffffffff812d76b0
> [exception RIP: mwait_idle+163]
> RIP: ffffffff81013029 RSP: ffff88031e115ef8 RFLAGS: 00000246
> RAX: 0000000000000000 RBX: ffff88031e115fd8 RCX: 0000000000000000
> RDX: 0000000000000000 RSI: ffff88031e115fd8 RDI: ffffffff81522308
> RBP: ffff88031e115f08 R8: 0000000000000000 R9: 0000000000000000
> R10: 0000000000000800 R11: 0000000000000000 R12: ffffffff8147e368
> R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
> --- <NMI exception stack> ---
> #6 [ffff88031e115ef8] mwait_idle at ffffffff81013029
> #7 [ffff88031e115f10] cpu_idle at ffffffff8100af21
>
> Please let me know if you need any other details.
>
> Thanks and Regards
> Shashidhara
>
>
> -----Original Message-----
> From: crash-utility-bounces at redhat.com
> [mailto:crash-utility-bounces at redhat.com] On Behalf Of Dave Anderson
> Sent: Thursday, June 23, 2011 9:35 PM
> To: Discussion list for crash utility usage,maintenance and
> development
> Subject: Re: [Crash-utility] Unable to switch stack frames while using
> crash
>
>
>
> ----- Original Message -----
>
> > BTW, are you sure about that?
> >
> > Presuming that the "tty" pointer is ffff8802cbd54800 as you've shown
> below,
> > and therefore tty->read_buf is 0xffff8802cbfe6000 and tty->read_tail
> is 0,
> > then the statement above would be simply be reading
> > tty->read_buf[0],
> or
> > virtual address 0xffff8802cbfe6000. But the oops shows it faulting
> > on
> a
> > virtual address of "5":
> >
> > BUG: unable to handle kernel NULL pointer dereference at
> 0000000000000005
>
> Just for my own sanity, can you either attach the
> "drivers/char/n_tty.c"
>
> from *your* specific kernel, or get the source-code/line-number data
> from
> the embedded gdb module?
>
> If you don't have the n_tty.c file readily available, you can get the
> source-code/line-number data of a particular function by doing
> something
> like this:
>
> Get the line number of the beginning of n_tty_read(), which in my
> kernel
> is at 1698 -- your's will probably be different:
>
> crash> gdb list n_tty_read
> 1695 * This code must be sure never to sleep through a hangup.
> 1696 */
> 1697
> 1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file
> *file,
> 1699 unsigned char __user *buf, size_t nr)
> 1700 {
> 1701 unsigned char __user *b = buf;
> 1702 DECLARE_WAITQUEUE(wait, current);
> 1703 int c;
> 1704 int minimum, time;
> crash>
>
> Then get the line number of the next function in the file, which is
> n_tty_write():
>
> crash> gdb list n_tty_write
> 1918 * lock themselves)
> 1919 */
> 1920
> 1921 static ssize_t n_tty_write(struct tty_struct *tty, struct file
> *file,
> 1922 const unsigned char *buf, size_t nr)
> 1923 {
> 1924 const unsigned char *b = buf;
> 1925 DECLARE_WAITQUEUE(wait, current);
> 1926 int c;
> 1927 ssize_t retval = 0;
>
> And then dump the whole n_tty_read() function (plus some extra stuff):
>
> crash> gdb list 1698,1920
> 1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file
> *file,
> 1699 unsigned char __user *buf, size_t nr)
> 1700 {
> 1701 unsigned char __user *b = buf;
> 1702 DECLARE_WAITQUEUE(wait, current);
> 1703 int c;
> 1704 int minimum, time;
> 1705 ssize_t retval = 0;
> 1706 ssize_t size;
> 1707 long timeout;
> 1708 unsigned long flags;
> 1709 int packet;
> 1710
> 1711 do_it_again:
> 1712
> 1713 BUG_ON(!tty->read_buf);
> 1714
> 1715 c = job_control(tty, file);
> 1716 if (c < 0)
> 1717 return c;
> 1718
> 1719 minimum = time = 0;
> 1720 timeout = MAX_SCHEDULE_TIMEOUT;
> 1721 if (!tty->icanon) {
> 1722 time = (HZ / 10) * TIME_CHAR(tty);
> 1723 minimum = MIN_CHAR(tty);
> ...
>
> And lastly, since the crash occurred at
>
> IP: [<ffffffff811f03b3>] n_tty_read+0x58c/0x818
>
> Do this:
>
> crash> dis -rl n_tty_read+0x58c
> ...
>
> And then post all of that data.
>
> Dave
>
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
> Information transmitted by this e-mail is proprietary to MphasiS, its
> associated companies and/ or its customers and is intended
> for use only by the individual or entity to which it is addressed, and
> may contain information that is privileged, confidential or
> exempt from disclosure under applicable law. If you are not the
> intended recipient or it appears that this mail has been forwarded
> to you without proper authority, you are notified that any use or
> dissemination of this information in any manner is strictly
> prohibited. In such cases, please notify us immediately at
> mailmaster at mphasis.com and delete this mail from your records.
>
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
More information about the Crash-utility
mailing list