[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] GFS file system corruption?



On Mon, Aug 29, 2005 at 03:06:19PM -0600, Matthew B. Brookover wrote:
> Aj, thank you for the quick response.
> 
> The patches changed the situation.  Postmark now runs without any
> errors, gfs_fsck still has some problems. I do not understand why
> postmark did not report any errors because the patches all appeared to
> be for fsck.

Actually, this is normal output for the fsck.  The way GFS works is that any
metadata that is deleted is actually converted to unused metadata blocks for
later reuse instead of being freed completely.  The way the fsck tracks block
use, it ends up converting all these blocks back to free blocks.  So basically
this is what a normal fsck looks like when run on a gfs filesystem that had
lots of files deleted.
 
> Output from gfs_fsck:
> 
> [root illusion root]# gfs_fsck -y /dev/pool/u_as
> Initializing fsck
> Clearing journals (this may take a while)
> .Cleared journals
> Starting pass1
> Pass1 complete
> Starting pass1b
> Pass1b complete
> Starting pass1c
> Pass1c complete
> Starting pass2
> Pass2 complete
> Starting pass3
> Pass3 complete
> Starting pass4
> Pass4 complete
> Starting pass5
> Converting 134 unused metadata blocks to free data blocks...
> Converting 162 unused metadata blocks to free data blocks...
> Converting 131 unused metadata blocks to free data blocks...
> Converting 152 unused metadata blocks to free data blocks...
> Converting 133 unused metadata blocks to free data blocks...
> Converting 132 unused metadata blocks to free data blocks...
> Converting 152 unused metadata blocks to free data blocks...
> Converting 133 unused metadata blocks to free data blocks...
> Converting 168 unused metadata blocks to free data blocks...
> Converting 161 unused metadata blocks to free data blocks...
> Converting 174 unused metadata blocks to free data blocks...
> Converting 147 unused metadata blocks to free data blocks...
> Converting 148 unused metadata blocks to free data blocks...
> Converting 134 unused metadata blocks to free data blocks...
> Converting 134 unused metadata blocks to free data blocks...
> Converting 137 unused metadata blocks to free data blocks...
> Converting 143 unused metadata blocks to free data blocks...
> Converting 138 unused metadata blocks to free data blocks...
> Converting 132 unused metadata blocks to free data blocks...
> Converting 148 unused metadata blocks to free data blocks...
> Converting 156 unused metadata blocks to free data blocks...
> Converting 179 unused metadata blocks to free data blocks...
> Converting 132 unused metadata blocks to free data blocks...
> Converting 133 unused metadata blocks to free data blocks...
> Converting 164 unused metadata blocks to free data blocks...
> Converting 139 unused metadata blocks to free data blocks...
> Converting 133 unused metadata blocks to free data blocks...
> Converting 132 unused metadata blocks to free data blocks...
> Converting 131 unused metadata blocks to free data blocks...
> Converting 133 unused metadata blocks to free data blocks...
> Converting 131 unused metadata blocks to free data blocks...
> Converting 152 unused metadata blocks to free data blocks...
> Converting 133 unused metadata blocks to free data blocks...
> Converting 141 unused metadata blocks to free data blocks...
> Converting 151 unused metadata blocks to free data blocks...
> Converting 150 unused metadata blocks to free data blocks...
> Converting 174 unused metadata blocks to free data blocks...
> Converting 132 unused metadata blocks to free data blocks...
> Converting 170 unused metadata blocks to free data blocks...
> Converting 140 unused metadata blocks to free data blocks...
> Converting 150 unused metadata blocks to free data blocks...
> Converting 131 unused metadata blocks to free data blocks...
> Converting 133 unused metadata blocks to free data blocks...
> Converting 135 unused metadata blocks to free data blocks...
> Converting 145 unused metadata blocks to free data blocks...
> Converting 143 unused metadata blocks to free data blocks...
> Converting 187 unused metadata blocks to free data blocks...
> Converting 166 unused metadata blocks to free data blocks...
> Converting 152 unused metadata blocks to free data blocks...
> Converting 133 unused metadata blocks to free data blocks...
> Converting 152 unused metadata blocks to free data blocks...
> Converting 168 unused metadata blocks to free data blocks...
> Converting 180 unused metadata blocks to free data blocks...
> Converting 139 unused metadata blocks to free data blocks...
> Converting 134 unused metadata blocks to free data blocks...
> Converting 145 unused metadata blocks to free data blocks...
> Converting 178 unused metadata blocks to free data blocks...
> Converting 131 unused metadata blocks to free data blocks...
> Converting 135 unused metadata blocks to free data blocks...
> Converting 140 unused metadata blocks to free data blocks...
> Converting 131 unused metadata blocks to free data blocks...
> Converting 161 unused metadata blocks to free data blocks...
> Converting 160 unused metadata blocks to free data blocks...
> Converting 151 unused metadata blocks to free data blocks...
> Converting 132 unused metadata blocks to free data blocks...
> Converting 171 unused metadata blocks to free data blocks...
> Converting 156 unused metadata blocks to free data blocks...
> Converting 183 unused metadata blocks to free data blocks...
> Converting 132 unused metadata blocks to free data blocks...
> Converting 142 unused metadata blocks to free data blocks...
> Converting 142 unused metadata blocks to free data blocks...
> Converting 163 unused metadata blocks to free data blocks...
> Converting 166 unused metadata blocks to free data blocks...
> Converting 134 unused metadata blocks to free data blocks...
> Converting 180 unused metadata blocks to free data blocks...
> Converting 156 unused metadata blocks to free data blocks...
> Converting 150 unused metadata blocks to free data blocks...
> Converting 141 unused metadata blocks to free data blocks...
> Converting 143 unused metadata blocks to free data blocks...
> Converting 177 unused metadata blocks to free data blocks...
> Converting 163 unused metadata blocks to free data blocks...
> Converting 136 unused metadata blocks to free data blocks...
> Converting 131 unused metadata blocks to free data blocks...
> Converting 143 unused metadata blocks to free data blocks...
> Converting 174 unused metadata blocks to free data blocks...
> Converting 133 unused metadata blocks to free data blocks...
> Converting 137 unused metadata blocks to free data blocks...
> Converting 133 unused metadata blocks to free data blocks...
> Converting 134 unused metadata blocks to free data blocks...
> Converting 134 unused metadata blocks to free data blocks...
> Converting 132 unused metadata blocks to free data blocks...
> Converting 133 unused metadata blocks to free data blocks...
> Converting 172 unused metadata blocks to free data blocks...
> Converting 169 unused metadata blocks to free data blocks...
> Converting 158 unused metadata blocks to free data blocks...
> Converting 135 unused metadata blocks to free data blocks...
> Converting 131 unused metadata blocks to free data blocks...
> Converting 133 unused metadata blocks to free data blocks...
> Converting 136 unused metadata blocks to free data blocks...
> Converting 131 unused metadata blocks to free data blocks...
> Converting 134 unused metadata blocks to free data blocks...
> Converting 132 unused metadata blocks to free data blocks...
> Converting 140 unused metadata blocks to free data blocks...
> Converting 187 unused metadata blocks to free data blocks...
> Converting 162 unused metadata blocks to free data blocks...
> Converting 174 unused metadata blocks to free data blocks...
> Converting 131 unused metadata blocks to free data blocks...
> Converting 130 unused metadata blocks to free data blocks...
> Converting 159 unused metadata blocks to free data blocks...
> Converting 171 unused metadata blocks to free data blocks...
> Pass5 complete
> [root illusion root]#
> 
> 
> What I did:
> * compiled GFS with the patch
> * shut down GFS, lock_gulmd, ccsd, and pool manager on all 6 nodes
> * loaded the new RPM
> * rebooted all 6 nodes and started up the pool manager, and ran gfs_fsck
> on all of the file sytems.  It was clean.
> * Started gulmd
> * mounted the GFS file systems and ran post mark several times. 
> Postmark did not report any errors.
> * unmounted the GFS file system, and ran gfs_fsck and got the above
> messages.
> 
> Any ideas?
> 
> thanks you for your assistance
> 
> Matt
> 
> On Mon, 2005-08-29 at 10:48, AJ Lewis wrote:
> 
> > On Sun, Aug 28, 2005 at 02:31:13PM -0600, Matthew B. Brookover wrote:
> > > I have 6 computers running Redhat Enterrpise 3 release 5, running kernel
> > > 2.4.21-32.0.1.ELsmp.
> > > 
> > > >From the source code I compiled GFS 6.0.2.20-2.  The SAN is an ISCSI
> > > based storage system from LeftHand Networks.   Using ext3, the postmark
> > > disk test works fine, on a GFS file system, we get a number of errors. 
> > > The output from both postmark runs is below.
> > 
> > Looks like you're running into bugs
> > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=160835
> > and
> > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=160525
> > which have been fixed in the latest code.  Unfortunately there is not a GFS
> > 6.0 srpm released yet that contains this fix, but I've attached a patch that
> > clears them up, as well as a couple other minor issues.
> > 
> > I'm not sure about the postmark errors you're seeing - hopefully someone else
> > on the list can shed some light on them.  But try the fsck with these patches
> > and see if there are any actual problems with the fs shown.
> > 
> > Regards,

> --
> Linux-cluster mailing list
> Linux-cluster redhat com
> http://www.redhat.com/mailman/listinfo/linux-cluster

-- 
AJ Lewis                                   Voice:  612-638-0500
Red Hat                                    E-Mail: alewis redhat com
One Main Street SE, Suite 209
Minneapolis, MN 55414
   
Current GPG fingerprint = D9F8 EDCE 4242 855F A03D  9B63 F50C 54A8 578C 8715
Grab the key at: http://people.redhat.com/alewis/gpg.html or one of the
many keyservers out there...

Attachment: pgpSkpeyrYp4h.pgp
Description: PGP signature


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]