I think I've deadlocked a GFS2 cluster I've been testing with. It's a three node cluster with shared storage exported via GNBD from a separate host. I was running an "rm -Rf /mnt/*; rsync -av /etc/ /mnt/foo/" script in an infinite loop on each node just to see what'd happen, now everything's locked up :) There's no traffic to my GNBD server so I don't think one node has got an indefinite lock.
No operations on the file system on any node works, the processes that are hung all look stuck in system calls: nothing can be killed. Any tips on how to resolve this without rebooting the whole cluster? Is there any debugging information I can get that'd help diagnose what caused the problem?