[Crash-utility] Crash issue when loading vmcore

[Dave Anderson]

----- "Paul-Kenji Cahier Furuya" <pkc at f1-photo.com> wrote:

> On 08/23/2010 23:25, Dave Anderson wrote:
> > cat /proc/kallsyms
> 
> I have attached allsymbols from the live system.
> 
> As for the debuginfo, that's exactly what I was doing(using the vmlinux 
> with the -g debuginfo from the root of the kernel tree)

OK, so the /proc/kallsyms shows a set of symbols with a more typical 
virtual address range, where the kernel's PAGE_OFFSET unity-map
value is c0000000:

  c0101000 T _stext
  c0101010 T do_one_initcall
  c0101180 t init_post
  c01012c0 T name_to_dev_t
  c0101500 T thread_saved_pc
  c0101510 T prepare_to_copy
  c0101590 T get_wchan
  c0101640 T __switch_to
  c01017f0 T start_thread
  c0101840 T copy_thread
  c0101990 T release_thread
  ...

And that seemingly matches the symbol values captured
in the VMCOREINFO section of the vmcore's ELF header:
  
  # grep SYMBOL crashd8.txt 
                           SYMBOL(init_uts_ns)=c06f9120
                           SYMBOL(node_online_map)=c0730644
                           SYMBOL(swapper_pg_dir)=c06e4000
                           SYMBOL(_stext)=c0101000
                           SYMBOL(vmlist)=c07d3540
                           SYMBOL(mem_map)=c07d3500
                           SYMBOL(contig_page_data)=c072ce80
                           SYMBOL(log_buf)=c06fc83c
                           SYMBOL(log_end)=c07bb7ec
                           SYMBOL(log_buf_len)=c06fc838
                           SYMBOL(logged_chars)=c07c38a0
  #

Unfortunately the /proc/kallsyms output only contains text 
symbols, and "_stext" is the only text symbol in the VMCOREINFO
list value that can be cross-checked between the two.  (I'm not
sure why your /proc/kallsyms doesn't contain data symbols?) 
In any case, at least "_stext" does match at c0101000.  

What I should have asked is for is the /boot/System.map-<version>
on that system, which will contain data symbols that *could* be
matched against the "SYMBOL(xxx)" values above.

In any case, since the symbol list from the vmlinux file that
you are using looks like this (from the "sym -l" output): 
  
  c1000000 (T) _text
  c1000000 (T) startup_32
  c1000054 (t) default_entry
  c1001000 (T) _stext
  c1001010 (T) do_one_initcall
  c1001180 (t) init_post
  c10012c0 (T) name_to_dev_t
  c1001500 (T) thread_saved_pc
  c1001510 (T) prepare_to_copy
  c1001590 (T) get_wchan
  c1001640 (T) __switch_to
  c10017f0 (T) start_thread
  ...

which, BTW, you can also see by doing this:

  # nm -Bn vmlinux

it's clearly not the same vmlinux that was running when
the system crashed.

By any chance did you build that vmlinux after-the-fact
in an attempt to "match" the kernel that was running?

In any case, what you can try would be this.  Get the original 
/boot/System-map-<version> file from the target system, and, 
presuming the symbol values in that file match the VMCOREINFO's
SYMBOL(xxx) values, as well as matching the symbols in the
/proc/kallsyms output, and do this:

  # crash System.map-<version> vmlinux vmcore.201008231930

Using a System.map file on the command line is pretty much an 
act of desperation, but it may work.  The crash utility will 
take the symbols in the System.map file, go into a back door 
into the embedded gdb module, and over-write the symbols from 
the vmlinux file.  But there's no guarantee that it will work,
because there may be differences in the definition of crucial
data structures that may cause crash to go off into the weeds.

Other than that, there's nothing much else that I can suggest.

Dave