[Crash-utility] EXT: Re: EXT: Re: EXT: Re: crash start in CentOS 8 - Solved
Agrain Patrick
patrick.agrain at al-enterprise.com
Mon Jul 20 13:22:52 UTC 2020
Hello,
Using the kernel-debuginfo packages is the solution.
The provided vmlinux has the required ' with debug_info, not stripped' tags.
'crash' is starting well.
Thanks to all for your time.
Best regards,
Patrick Agrain
-----Message d'origine-----
De : Masayoshi Mizuma [mailto:msys.mizuma at gmail.com]
Envoyé : lundi 20 juillet 2020 14:50
À : Agrain Patrick <patrick.agrain at al-enterprise.com>
Cc : Discussion list for crash utility usage, maintenance and development <crash-utility at redhat.com>
Objet : EXT: Re: [Crash-utility] EXT: Re: EXT: Re: crash start in CentOS 8
** External email - Please consider with caution **
Hello,
On 7/20/20 8:04 AM, Agrain Patrick wrote:
> Hello all,
>
> I've upgraded my Centos 8 VM to the latest kernel (4.18.0-193.6.3.el8_2.x86_64).
> I have then installed the kernel-debug packages (and dependencies), as suggested below.
>
> Now, I have following kernels in /boot:
> - vmlinuz-4.18.0-193.6.3.el8_2.x86_64
> - vmlinuz-4.18.0-193.6.3.el8_2.x86_64+debug
This is a kernel-debug pacakge, not kernel-debuginfo package.
I think you can get the kernel-debuginfo here:
http://debuginfo.centos.org/8/x86_64/Packages/
Probably you need to install kernel-debuginfo-common-x86_64 package as well.
After the debuginfo package is installed, you'll see the vmlinux as follows:
/usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux
Then:
crash /usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux vmcore
Could you try it?
Thanks,
Masa
> And a vmlinuz file in /lib/modules/4.18.0-193.6.3.el8_2.x86_64+debug
>
> Then, I reboot the VM on this latest kernel and extracted the vmlinux files with the extract-vmlinux script from a CentOS 4.18.0 kernel source tree.
> A last, the 'echo c > /proc/sysrq-trigger' creates the /var/crash/vmcore file.
>
> Exporting the vmcore, vmlinux-4.18.0-193.6.3.el8_2.x86_64+debug and vmlinux (from /lib/modules) on another CentOS machine and executing:
> /usr/bin/crash -d 15 vmlinux-4.18.0-193.6.3.el8_2.x86_64+debug vmcore
> returns:
> crash: vmlinux-4.18.0-193.6.3.el8_2.x86_64+debug: no .gnu_debuglink
> section
>
> Note: same result when using the exported vmlinux from /lib/modules directory.
>
> Still not driving me nut... but not far away...
>
> To be continued...
> Thanks.
> Best regards,
> Patrick
>
>
> -----Message d'origine-----
> De : crash-utility-bounces at redhat.com
> [mailto:crash-utility-bounces at redhat.com] De la part de Kazuhito Hagio
> Envoyé : jeudi 9 juillet 2020 07:00 À : Discussion list for crash
> utility usage, maintenance and development <crash-utility at redhat.com>
> Objet : EXT: Re: [Crash-utility] EXT: Re: crash start in CentOS 8
>
>
> ** External email - Please consider with caution **
>
>
> Hi Patrick,
>
>> OSRELEASE=4.18.0-147.el8.x86_64
>
> Hmm, I think you might boot the virtual machine with the original (unmodified) CentOS 8 kernel, because the address of init_uts_ns looks to be its one:
>
>> SYMBOL(init_uts_ns)=ffffffffb5a12540
>> KERNELOFFSET=33800000
>
> I happen to have a CentOS 8 machine..
>
> # grep 'D init_uts_ns' /boot/System.map-4.18.0-147.el8.x86_64
> ffffffff82212540 D init_uts_ns
> # printf "%16llx\n" $(( 0xffffffff82212540 + 0x33800000 ))
> ffffffffb5a12540
>
> while your vmcore.debuginfo has the different address:
>
>> <readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390, (ROE),
>> d71b7c>
>
> So, which kernel do you actually want to use on the virtual machine?
> (1) The original CentOS 8 kernel
> (2) The kernel you built
>
> If it's (1), you can just install the kernel-debuginfo package corresponding to the original CentOS 8 kernel into the machine on which you want to analyze vmcores and execute:
> # crash /usr/lib/debug/lib/modules/4.18.0-147.el8.x86_64/vmlinux
> vmcore
>
> If it's (2), you need to boot the VM with the kernel and make a crash.
>
> Could you check this first?
>
> Thanks,
> Kazu
>
>
> On Mon, Jul 6, 2020 at 10:03 PM Agrain Patrick <patrick.agrain at al-enterprise.com> wrote:
>>
>> Hello all,
>>
>> Back on this topic:
>> I compiled a kernel with debuginfo on the Virtual Machine with CentOS8.
>> I made a crash on this same VM.
>> I transferred the vmcore and vmlinux.debuginfo files on another CentOS8 baremetal machine.
>>
>> The command '/usr/bin/crash -d 15 vmlinux.debuginfo vmcore' produces following logs, but still failing to get the crash> prompt.
>> Still KASLR ?
>>
>> Thanks in advance for any information.
>> Best regards,
>> Patrick Agrain
>>
>> -- Logs:
>> crash 7.2.6-2.el8
>> Copyright (C) 2002-2019 Red Hat, Inc.
>> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C)
>> 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012
>> Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
>> Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002,
>> 2007 Silicon Graphics, Inc.
>> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
>> This program is free software, covered by the GNU General Public
>> License, and you are welcome to change it and/or distribute copies of
>> it under certain conditions. Enter "help copying" to see the conditions.
>> This program has absolutely no warranty. Enter "help warranty" for details.
>>
>> compressed kdump: header->utsname.machine: x86_64 compressed kdump:
>> memory bitmap offset: 2000
>> diskdump_data:
>> filename: vmcore
>> flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
>> dfd: 3
>> ofp: 0
>> machine_type: 62 (EM_X86_64)
>>
>> header: 212b5d0
>> signature: "KDUMP "
>> header_version: 6
>> utsname:
>> sysname: Linux
>> nodename: localhost.localdomain
>> release: 4.18.0-147.el8.x86_64
>> version: #1 SMP Wed Dec 4 21:51:45 UTC 2019
>> machine: x86_64
>> domainname: (none)
>> timestamp:
>> tv_sec: 5ee72d23
>> tv_usec: 0
>> status: 2 (DUMP_DH_COMPRESSED_LZO)
>> block_size: 4096
>> sub_hdr_size: 1
>> bitmap_blocks: 32
>> max_mapnr: 524288
>> total_ram_blocks: 0
>> device_blocks: 0
>> written_blocks: 0
>> current_cpu: 0
>> nr_cpus: 1
>> tasks[nr_cpus]: 0
>>
>> sub_header: 0 (n/a)
>>
>> sub_header_kdump: 212c5e0
>> phys_base: 7c00000
>> dump_level: 31 (0x1f) (DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
>> split: 0
>> start_pfn: (unused)
>> end_pfn: (unused)
>> offset_vmcoreinfo: 4580 (0x11e4)
>> size_vmcoreinfo: 2025 (0x7e9)
>> OSRELEASE=4.18.0-147.el8.x86_64
>> PAGESIZE=4096
>> SYMBOL(init_uts_ns)=ffffffffb5a12540
>> SYMBOL(node_online_map)=ffffffffb5c06d60
>> SYMBOL(swapper_pg_dir)=ffffffffb5a0a000
>> SYMBOL(_stext)=ffffffffb4800000
>> SYMBOL(vmap_area_list)=ffffffffb5ad2a90
>> SYMBOL(mem_section)=ffff89df7ffd2000
>> LENGTH(mem_section)=2048
>> SIZE(mem_section)=16
>> OFFSET(mem_section.section_mem_map)=0
>> SIZE(page)=64
>> SIZE(pglist_data)=171968
>> SIZE(zone)=1472
>> SIZE(free_area)=88
>> SIZE(list_head)=16
>> SIZE(nodemask_t)=128
>> OFFSET(page.flags)=0
>> OFFSET(page._refcount)=52
>> OFFSET(page.mapping)=24
>> OFFSET(page.lru)=8
>> OFFSET(page._mapcount)=48
>> OFFSET(page.private)=40
>> OFFSET(page.compound_dtor)=16
>> OFFSET(page.compound_order)=17
>> OFFSET(page.compound_head)=8
>> OFFSET(pglist_data.node_zones)=0
>> OFFSET(pglist_data.nr_zones)=171232
>> OFFSET(pglist_data.node_start_pfn)=171240
>> OFFSET(pglist_data.node_spanned_pages)=171256
>> OFFSET(pglist_data.node_id)=171264
>> OFFSET(zone.free_area)=192
>> OFFSET(zone.vm_stat)=1296
>> OFFSET(zone.spanned_pages)=112
>> OFFSET(free_area.free_list)=0
>> OFFSET(list_head.next)=0
>> OFFSET(list_head.prev)=8
>> OFFSET(vmap_area.va_start)=0
>> OFFSET(vmap_area.list)=48
>> LENGTH(zone.free_area)=11
>> SYMBOL(log_buf)=ffffffffb5a5a280
>> SYMBOL(log_buf_len)=ffffffffb5a5a27c
>> SYMBOL(log_first_idx)=ffffffffb633a770
>> SYMBOL(clear_idx)=ffffffffb633a744
>> SYMBOL(log_next_idx)=ffffffffb633a760
>> SIZE(printk_log)=16
>> OFFSET(printk_log.ts_nsec)=0
>> OFFSET(printk_log.len)=8
>> OFFSET(printk_log.text_len)=10
>> OFFSET(printk_log.dict_len)=12
>> LENGTH(free_area.free_list)=5
>> NUMBER(NR_FREE_PAGES)=0
>> NUMBER(PG_lru)=5
>> NUMBER(PG_private)=12
>> NUMBER(PG_swapcache)=9
>> NUMBER(PG_swapbacked)=18
>> NUMBER(PG_slab)=8
>> NUMBER(PG_hwpoison)=22
>> NUMBER(PG_head_mask)=32768
>> NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129
>> NUMBER(HUGETLB_PAGE_DTOR)=2
>> NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257
>> NUMBER(phys_base)=130023424
>> SYMBOL(init_top_pgt)=ffffffffb5a0a000
>> NUMBER(pgtable_l5_enabled)=0
>> SYMBOL(node_data)=ffffffffb5c02580
>> LENGTH(node_data)=1024
>> KERNELOFFSET=33800000
>> NUMBER(KERNEL_IMAGE_SIZE)=1073741824
>> NUMBER(sme_mask)=0
>> CRASHTIME=1592208675
>> offset_note: 4200 (0x1068)
>> size_note: 2408 (0x968)
>> notes_buf: 2129460
>> num_vmcoredd_notes: 0
>> num_prstatus_notes: 1
>> notes[0]: 2129460 (NT_PRSTATUS)
>> si.signo: 0 si.code: 0 si.errno: 0
>> cursig: 0 sigpend: 0 sighold: 0
>> pid: 2846 ppid: 0 pgrp: 0 sid:0
>> utime: 0.000000 stime: 0.000000
>> cutime: 0.000000 cstime: 0.000000
>> ORIG_RAX: ffffffffffffffff fpvalid: 0
>> R15: 0000000000000000 R14: 00005616bb1208f0
>> R13: ffffffffb5b36fc0 R12: 0000000000000000
>> RBP: 0000000000000004 RBX: 0000000000000063
>> R11: 6873617263206120 R10: 7265676769725420
>> R9: 54203a2071527379 R8: 000000000000056a
>> RAX: ffffffffb4d12d10 RCX: 0000000000000006
>> RDX: 0000000000000000 RSI: 0000000000000086
>> RDI: 0000000000000063 RIP: ffffffffb4d12d22
>> RFLAGS: 0000000000010246 RSP: ffffb03903147e80
>> FS_BASE: 00007f5d6e06f740
>> GS_BASE: 0000000000000000
>> CS: 0010 SS: 0018 DS: 0000
>> ES: 0000 FS: 0000 GS: 0000
>> snapshot_task: 0
>> num_qemu_notes: 0
>> NOTE offsets: 1068 (NT_PRSTATUS)
>> offset_eraseinfo: 0 (0x0)
>> size_eraseinfo: 0 (0x0)
>> start_pfn_64: (unused)
>> end_pfn_64: (unused)
>> max_mapnr_64: 524288 (0x80000)
>>
>> data_offset: 22000
>> block_size: 4096
>> block_shift: 12
>> bitmap: 212f190
>> bitmap_len: 131072
>> max_mapnr: 524288 (0x80000)
>> dumpable_bitmap: 7f937ad93010
>> byte: 0
>> bit: 0
>> compressed_page: 212d5f0
>> curbufptr: 0
>>
>> page_cache_hdr[0]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 217f1d0
>> pg_hit_count: 0
>> page_cache_hdr[1]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 21801d0
>> pg_hit_count: 0
>> page_cache_hdr[2]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 21811d0
>> pg_hit_count: 0
>> page_cache_hdr[3]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 21821d0
>> pg_hit_count: 0
>> page_cache_hdr[4]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 21831d0
>> pg_hit_count: 0
>> page_cache_hdr[5]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 21841d0
>> pg_hit_count: 0
>> page_cache_hdr[6]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 21851d0
>> pg_hit_count: 0
>> page_cache_hdr[7]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 21861d0
>> pg_hit_count: 0
>> page_cache_hdr[8]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 21871d0
>> pg_hit_count: 0
>> page_cache_hdr[9]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 21881d0
>> pg_hit_count: 0
>> page_cache_hdr[10]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 21891d0
>> pg_hit_count: 0
>> page_cache_hdr[11]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 218a1d0
>> pg_hit_count: 0
>> page_cache_hdr[12]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 218b1d0
>> pg_hit_count: 0
>> page_cache_hdr[13]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 218c1d0
>> pg_hit_count: 0
>> page_cache_hdr[14]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 218d1d0
>> pg_hit_count: 0
>> page_cache_hdr[15]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 218e1d0
>> pg_hit_count: 0
>>
>> page_cache_buf: 217f1d0
>> evict_index: 0
>> evictions: 0
>> accesses: 0
>> cached_reads: 0
>> valid_pages: 212a3b0
>> readmem: read_diskdump()
>> KASLR:
>> _stext from vmlinux.debuginfo: ffffffff81000000
>> _stext from vmcoreinfo: ffffffffb4800000
>> relocate: 33800000 (824MB)
>> crash: pv_init_ops exists: ARCH_PVOPS
>> VMCOREINFO: NUMBER(phys_base): 130023424 -> 7c00000 gdb
>> vmlinux.debuginfo GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software
>> Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-unknown-linux-gnu"...
>> GETBUF(328 -> 0)
>> GETBUF(1500 -> 1)
>>
>> WARNING: kernel relocated [824MB]: patching 109676 gdb minimal_symbol
>> values
>>
>> FREEBUF(1)
>> FREEBUF(0)
>> <readmem: ffffffffb70df958, KVADDR, "page_offset_base", 8, (FOE|Q),
>> d8b948>
>> <read_diskdump: addr: ffffffffb70df958 paddr: 3ecdf958 cnt: 8>
>> read_diskdump: paddr/pfn: 3ecdf958/3ecdf -> cache physical page:
>> 3ecdf000
>> GETBUF(328 -> 0)
>> FREEBUF(0)
>> GETBUF(1024 -> 0)
>> <readmem: ffffffffb7f0c5e0, KVADDR, "possible", 1024, (ROE), fcd120>
>> <read_diskdump: addr: ffffffffb7f0c5e0 paddr: 3fb0c5e0 cnt: 1024>
>> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c5e0/3fb0c
>> crash: page excluded: kernel virtual address: ffffffffb7f0c5e0 type: "possible"
>> WARNING: cannot read cpu_possible_map
>> <readmem: ffffffffb7f0bde0, KVADDR, "present", 1024, (ROE), fcd120>
>> <read_diskdump: addr: ffffffffb7f0bde0 paddr: 3fb0bde0 cnt: 544>
>> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0bde0/3fb0b
>> crash: page excluded: kernel virtual address: ffffffffb7f0bde0 type: "present"
>> WARNING: cannot read cpu_present_map
>> <readmem: ffffffffb7f0c1e0, KVADDR, "online", 1024, (ROE), fcd120>
>> <read_diskdump: addr: ffffffffb7f0c1e0 paddr: 3fb0c1e0 cnt: 1024>
>> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c1e0/3fb0c
>> crash: page excluded: kernel virtual address: ffffffffb7f0c1e0 type: "online"
>> WARNING: cannot read cpu_online_map
>> <readmem: ffffffffb7f0b9e0, KVADDR, "active", 1024, (ROE), fcd120>
>> <read_diskdump: addr: ffffffffb7f0b9e0 paddr: 3fb0b9e0 cnt: 1024>
>> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0b9e0/3fb0b
>> crash: page excluded: kernel virtual address: ffffffffb7f0b9e0 type: "active"
>> WARNING: cannot read cpu_active_map
>> FREEBUF(0)
>> <readmem: ffffffffb731a5b0, KVADDR, "pv_init_ops", 8, (ROE),
>> 7ffd3655d640>
>> <read_diskdump: addr: ffffffffb731a5b0 paddr: 3ef1a5b0 cnt: 8>
>> read_diskdump: paddr/pfn: 3ef1a5b0/3ef1a -> cache physical page:
>> 3ef1a000
>> GETBUF(328 -> 0)
>> FREEBUF(0)
>> GETBUF(328 -> 0)
>> FREEBUF(0)
>> <readmem: ffffffffb9bad1d0, KVADDR, "shadow_timekeeper xtime_sec", 8,
>> (ROE), 7ffd3655d5f0>
>> <read_diskdump: addr: ffffffffb9bad1d0 paddr: 417ad1d0 cnt: 8>
>> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 417ad1d0/417ad
>> crash: page excluded: kernel virtual address: ffffffffb9bad1d0 type: "shadow_timekeeper xtime_sec"
>> xtime timespec.tv_sec: 900f27: Mon Apr 20 07:31:03 1970
>> <readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390, (ROE),
>> d71b7c>
>> <read_diskdump: addr: ffffffffb7221d64 paddr: 3ee21d64 cnt: 390>
>> read_diskdump: paddr/pfn: 3ee21d64/3ee21 -> cache physical page:
>> 3ee21000
>> utsname:
>> sysname:
>> nodename:
>> release:
>> version:
>> machine:
>> domainname:
>> base kernel version: 0.0.0
>> <readmem: ffffffffb6a016c0, KVADDR, "accessible check", 8, (ROE|Q),
>> 7ffd3655af30>
>> <read_diskdump: addr: ffffffffb6a016c0 paddr: 3e6016c0 cnt: 8>
>> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3e6016c0/3e601
>> crash: page excluded: kernel virtual address: ffffffffb6a016c0 type: "accessible check"
>> crash: vmlinux.debuginfo and vmcore do not match!
>>
>> Usage:
>>
>> crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form)
>> crash [OPTION]... [NAMELIST] (live system form)
>>
>> Enter "crash -h" for details.
>>
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
More information about the Crash-utility
mailing list