[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Linux-cluster] Kernel panic: GFS: Assertion failed on line 1227 of file rgrp.c



Were getting consistent kernel panics with most of our GFS nodes all pointing to the same line and file:

Kernel panic: GFS: Assertion failed on line 1227 of file rgrp.c

The longest uptime we've seen was a couple weeks, but for the most part the nodes will only stay up for a few days. Once one goes down, a couple more follow, not immediately, but within the hour.

Configured are 5 nodes, three lock_gulm servers and two clients. Currently all are running the same version kernel, GFS and GFS modules as list below.

- GFS-modules-smp-6.0.2.27-0
- GFS-6.0.2.27-0

- 2.4.21-37.ELsmp

- Scientific Linux Release 303 (Fermi), RHES 3, update 3 really.
- This is a new installation.


From /var/log/messages:

Jan 2 12:46:48 fnd0374 kernel: ce1bbbac f8bc7b72 00000246 00001000 db122da8 f8bf4000 db122da8 f8bc7d70 Jan 2 12:46:48 fnd0374 kernel: 00000246 000001f0 00000000 55b6bd08 f60fdd8c 00000005 00000001 ffffffff Jan 2 12:46:48 fnd0374 kernel: f8be16d0 f8be8c8f f8be8bc4 000004cb 00000016 f60f7c00 00000006 f8bf4000 Jan 2 12:46:48 fnd0374 kernel: Call Trace: [<f8bc7b72>] gfs_asserti [gfs] 0x32 (0xce1bbbb0)
Jan  2 12:46:48 fnd0374 kernel: [<f8bc7d70>] gmalloc [gfs] 0x20 (0xce1bbbc8)
Jan 2 12:46:48 fnd0374 kernel: [<f8be16d0>] blkalloc_internal [gfs] 0x130 (0xce1bbbec) Jan 2 12:46:48 fnd0374 kernel: [<f8be8c8f>] .rodata.str1.1 [gfs] 0x1da3 (0xce1bbbf0) Jan 2 12:46:48 fnd0374 kernel: [<f8be8bc4>] .rodata.str1.1 [gfs] 0x1cd8 (0xce1bbbf4) Jan 2 12:46:48 fnd0374 kernel: [<f8be1b8b>] gfs_blkalloc [gfs] 0x7b (0xce1bbc20) Jan 2 12:46:48 fnd0374 kernel: [<f8bbb90c>] get_datablock [gfs] 0xfc (0xce1bbc4c) Jan 2 12:46:48 fnd0374 kernel: [<f8bbbc43>] gfs_block_map [gfs] 0x333 (0xce1bbc70) Jan 2 12:46:48 fnd0374 kernel: [<c0149093>] find_or_create_page [kernel] 0x63 (0xce1bbc9c) Jan 2 12:46:48 fnd0374 kernel: [<f8bac08c>] gfs_dgetblk [gfs] 0x3c (0xce1bbcec) Jan 2 12:46:48 fnd0374 kernel: [<f8bb5239>] get_block [gfs] 0xb9 (0xce1bbd28) Jan 2 12:46:48 fnd0374 kernel: [<c016814b>] __block_prepare_write [kernel] 0x1ab (0xce1bbd64) Jan 2 12:46:48 fnd0374 kernel: [<c0168b09>] block_prepare_write [kernel] 0x39 (0xce1bbda8) Jan 2 12:46:48 fnd0374 kernel: [<f8bb5180>] get_block [gfs] 0x0 (0xce1bbdbc) Jan 2 12:46:48 fnd0374 kernel: [<f8bb58fc>] gfs_prepare_write [gfs] 0x12c (0xce1bbdc8) Jan 2 12:46:48 fnd0374 kernel: [<f8bb5180>] get_block [gfs] 0x0 (0xce1bbdd8) Jan 2 12:46:48 fnd0374 kernel: [<c014c053>] do_generic_file_write [kernel] 0x1e3 (0xce1bbdf4) Jan 2 12:46:48 fnd0374 kernel: [<f8bafbab>] do_do_write [gfs] 0x2ab (0xce1bbe48) Jan 2 12:46:48 fnd0374 kernel: [<f8baffeb>] do_write [gfs] 0x18b (0xce1bbe94) Jan 2 12:46:48 fnd0374 kernel: [<f8badf1e>] gfs_walk_vma [gfs] 0x12e (0xce1bbed0) Jan 2 12:46:48 fnd0374 kernel: [<c0225936>] sock_read [kernel] 0x96 (0xce1bbf50) Jan 2 12:46:48 fnd0374 kernel: [<f8bb00c1>] gfs_write [gfs] 0x91 (0xce1bbf6c)
Jan  2 12:46:48 fnd0374 kernel: [<f8bafe60>] do_write [gfs] 0x0 (0xce1bbf80)
Jan 2 12:46:48 fnd0374 kernel: [<c0164b27>] sys_write [kernel] 0x97 (0xce1bbf94)
Jan  2 12:46:48 fnd0374 kernel:
Jan 2 12:46:48 fnd0374 kernel: Kernel panic: GFS: Assertion failed on line 1227 of file rgrp.c
Jan  2 12:46:48 fnd0374 kernel: GFS: assertion: "x <= length"
Jan  2 12:46:48 fnd0374 kernel: GFS: time = 1136227608
Jan  2 12:46:48 fnd0374 kernel: GFS: fsid=d0recon:d0.4: RG = 71028427


Appreciate any help.
Thanks,
Paul

--
===========================================================================
Paul Tader  <ptader fnal gov>
Fermi National Accelerator Lab; PO Box 500 Batavia, IL 60510-0500


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]