[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] GFS Problem: invalid metadata block



Matt Eagleson wrote:
Hello,

I have been evaluating a GFS cluster as an NFS solution and have unfortunately run in to a serious problem which I cannot explain. Both of the GFS filesystems I am exporting became corrupt and unusable.

The system is Redhat AS4 with 2.6.9-42.0.2.ELsmp. I cannot find anything unusual on the host or the SAN at the time of this error. Nobody was logged in to the nodes.

Can anyone help me understand what is happening here?

Here are the logs:

Hi Matt,

These errors indicate file system corruption on your SAN. The "bh =" is the
block number where the error was detected.  Two of the errors were found
in GFS resource group data ("RG"), which are areas on disk that indicate which blocks on the SAN are allocated and which aren't. (Not to be confused with the Resource Groups in rgmanager, which is something completely different.) The third error is usually reserved for the quota file inode. Corruption in the RG information is extremely rare, and may indicate a hardware problem with your SAN. The fact that both nodes detected problems in different areas is an indication that the problem might be in the SAN itself rather than the motherboards, fibre channel cards or memory of the nodes, although that's
still not guaranteed.  Many things can cause data corruption.

I recommend you:

1. Verify the hardware is working properly in all respects. One way you can do this is to make a backup of the raw data to another device and verify the copy against
   the original without GFS or any of the cluster software in the mix.
For example, unmount the file system from all nodes in the cluster, then do
   something like "dd if=/dev/my_vg/lvol0 of=/mnt/backup/sanbackup" then:
   "diff /dev/my_vg/lvol0 mnt/backup/sanbackup"  (assuming of course that
/dev/my_vg/lvol0 is the logical volume you have your GFS partition on, and
   /mnt/backup/ is some scratch area big enough to hold that much data.)
   The idea here is simply to test that reading from the SAN give you
the same data twice. If that works successfully on one node, try it on the other node.
2. Once you verify the hardware is working properly, run gfs_fsck on it.
   The latest version of gfs_fsck can repair most GFS rg corruption.
3. If the file system is fixed okay, you should back it up.
4. You may want to do a similar test, only writing data to the SAN, then reading it back and verifying the results. Obviously this will destroy the data on your SAN unless you are careful, so if this is a production machine, please take measures
   to protect the data before trying anything like this.
5. If you can read and write to the SAN reliably from both nodes without GFS,
   then try using it again with GFS and see if the problem comes back.

Perhaps someone else (the SAN manufacturer?) can recommend hardware
tests you can run to verify the data integrity.

I realize these kinds of tests take a long time to do, but if it's a hardware problem,
you really need to know.  There's a outside chance the problem is somewhere
in the GFS core, but I've personally only seen this type of corruption once or twice so I think it's unlikely. If you can recreate this kind of corruption with some kind of
test, please let us know how.

Regards,

Bob Peterson
Red Hat Cluster Suite


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]