[Linux-cachefs] 5.2.0-rc1-afs-next-9f4a9105 : kernel BUG at fs/fscache/operation.c:74

[Ian Wienand]

Hello,

We are running a 5.2.0-rc1 kernel from the afs-next branch @ 9f4a9105
and hit the following.

The host is serving files from various AFS volumes via Apache.  It is
all read-only traffic.  We have cachefilesd configured to a dedicated
ext4 filesystem on a partition from a LVM volume.

We have had the host disappear a number of times without trace.  It is
a VM and just went into "shutdown" state on the provider side.  It did
the same this time, but this was the first time we managed to grab a
oops (via netconsole to a remote host).  If the prior shutdowns were
due to this is unclear, but likely.  The host has been up for several
days at times, but only about 12 hours before we hit this.  So it
seems racy.

Happy for suggestions!

Thanks,

-i

---

[74690.011888] FS-Cache:
[74690.011917] FS-Cache: Assertion failed
[74690.011929] FS-Cache: 4 == 5 is false
[74690.011964] ------------[ cut here ]------------
[74690.011982] kernel BUG at fs/fscache/operation.c:74!
[74690.012004] invalid opcode: 0000 [#1] SMP PTI
[74690.012017] CPU: 2 PID: 21 Comm: ksoftirqd/2 Not tainted 5.2.0-rc1-afs-next-9f4a9105 #2
[74690.012035] Hardware name: Xen HVM domU, BIOS 4.1.5 11/28/2013
[74690.012059] RIP: 0010:fscache_enqueue_operation+0x1f1/0x210 [fscache]
[74690.012075] Code: c7 78 52 9e c0 e8 22 c6 51 e9 48 c7 c7 86 52 9e c0 e8 16 c6 51 e9 8b 73 40 ba 05 00 00 00 48 c7 c7 a8 42 9e c0 e8 02 c6 51 e9 <0f> 0b 48 c7 c7 f0 42 9e c0 e8 f4 c5 51 e9 0f 0b 0f 1f 44 00 00 66
[74690.012114] RSP: 0018:ffffa93f00d7bb78 EFLAGS: 00010086
[74690.012127] RAX: 0000000000000019 RBX: ffff8fedcef5f900 RCX: 0000000000000006
[74690.012144] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff8fefc7497380
[74690.012160] RBP: ffffa93f00d7bb90 R08: 000000000000027b R09: 0000000000000000
[74690.012175] R10: ffffa93f00d7bc88 R11: ffff8fefbd898000 R12: ffff8fedcef5f900
[74690.012189] R13: ffff8fedc815b020 R14: ffffffffab408100 R15: 0000000000000000
[74690.012238] CR2: 00007f724441a000 CR3: 000000003e420000 CR4: 00000000000006e0
[74690.012255] Call Trace:
[74690.012271]  cachefiles_read_waiter+0xd5/0x130 [cachefiles]
[74690.012287]  __wake_up_common+0x73/0x130
[74690.012209] FS:  0000000000000000(0000) GS:ffff8fefc7480000(0000) knlGS:0000000000000000
[74690.012225] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[74690.012306]  __wake_up_locked_key_bookmark+0x1b/0x20
[74690.012327]  wake_up_page_bit+0xab/0x100
[74690.012339]  unlock_page+0x26/0x30
[74690.012772]  mpage_end_io+0x74/0x100
[74690.013141]  bio_endio+0xf0/0x170
[74690.013505]  dec_pending+0x10e/0x210
[74690.013919]  clone_endio+0x90/0x180
[74690.014268]  bio_endio+0xf0/0x170
[74690.014616]  blk_update_request+0x7b/0x300
[74690.014970]  blk_mq_end_request+0x20/0x130
[74690.015319]  blkif_complete_rq+0x15/0x20
[74690.015669]  blk_done_softirq+0x92/0xc0
[74690.016018]  __do_softirq+0xe4/0x2f3
[74690.016365]  run_ksoftirqd+0x2b/0x40
[74690.016708]  smpboot_thread_fn+0xfc/0x170
[74690.017051]  kthread+0x121/0x140
[74690.017388]  ? sort_range+0x30/0x30
[74690.017726]  ? kthread_park+0x90/0x90
[74690.018069]  ret_from_fork+0x35/0x40
[74690.018410] Modules linked in: netconsole kafs fcrypt pcbc rxrpc cachefiles fscache binfmt_misc ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6_tables ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter ppdev joydev intel_rapl sb_edac intel_rapl_perf input_leds serio_raw parport_pc parport mac_hid sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core xenfs xen_privcmd iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel cirrus drm_kms_helper aes_x86_64 crypto_simd syscopyarea cryptd sysfillrect glue_helper sysimgblt fb_sys_fops psmouse drm i2c_piix4 pata_acpi floppy
[74690.021073] ---[ end trace 8a69254422d3d600 ]---
[74690.021493] RIP: 0010:fscache_enqueue_operation+0x1f1/0x210 [fscache]
[74690.021913] Code: c7 78 52 9e c0 e8 22 c6 51 e9 48 c7 c7 86 52 9e c0 e8 16 c6 51 e9 8b 73 40 ba 05 00 00 00 48 c7 c7 a8 42 9e c0 e8 02 c6 51 e9 <0f> 0b 48 c7 c7 f0 42 9e c0 e8 f4 c5 51 e9 0f 0b 0f 1f 44 00 00 66
[74690.022784] RSP: 0018:ffffa93f00d7bb78 EFLAGS: 00010086
[74690.023224] RAX: 0000000000000019 RBX: ffff8fedcef5f900 RCX: 0000000000000006
[74690.023664] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff8fefc7497380
[74690.024103] RBP: ffffa93f00d7bb90 R08: 000000000000027b R09: 0000000000000000
[74690.024535] R10: ffffa93f00d7bc88 R11: ffff8fefbd898000 R12: ffff8fedcef5f900
[74690.024955] R13: ffff8fedc815b020 R14: ffffffffab408100 R15: 0000000000000000
[74690.025483] FS:  0000000000000000(0000) GS:ffff8fefc7480000(0000) knlGS:0000000000000000
[74690.026142] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[74690.026815] CR2: 00007f724441a000 CR3: 000000003e420000 CR4: 00000000000006e0
[74690.027510] Kernel panic - not syncing: Fatal exception in interrupt
[74690.028393] Kernel Offset: 0x28e00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)