[linux-lvm] EXT3 snapshot not quiesced by LVM2?

Thu Apr 8 18:34:36 UTC 2004

We've finally resolved this issue; it had nothing to do with LVM.

I tried upgrading to the LVM2 2.00.09 and device-mapper 1.00.09 but
still had occasional orphaned inodes when we mounted the snapshot. I
then tried the unstable patches from people.sistina.com/~thornber/dm;
this didn't fix the problem either.

I finally tried using lsof to see what was open on the origin logical
volume. This showed us that mysqld still had a log file open:

mysqld     1198 root    5w   REG      254,1     2978     32771 /opt/data/mysql/mysqldb/slow.log.1 (deleted)

Because slow.log had been logrotated and compressed, the inode was in
fact an orphan. By adding 'mysqladmin flush-logs' to the postrotate section
of our mysql logrotate file, the problem went away.

We've now reverted back to LVM2 2.00.09 and device-mapper 1.00.09.

Thanks to Joe and Steve for your suggestions.

On Mon, 15 Mar 2004 18:23:56 -0700
"Donald Murray P.Eng." <donald.murray at pason.com> wrote:

> Problem:
> LVM2 snapshots of an EXT3 partition do not appear to be taken
> from a quiesced lv. In /var/log/messages, we see messages about
> an orphaned inode.
> 
> Details:
> We're using LVM2 on a RedHat 7.2 box with a 2.4.22 kernel from kernel.org.
> The kernel has been patched with the following patches:
>     linux-2.4.22-VFS-lock.patch
>     linux-2.4.22-devmapper-ioctl.patch
> 
> Once per hour, we use an LVM snapshot to backup a MySQL database. The
> procedure is:
> 1. mysql server: flush tables with read lock; hold this lock until step 5
> 2. sync
> 3. lvcreate --extents N --snapshot --name dbsnapshotlv /dev/datavg/datalv
> 4. mount -o ro /dev/datavg/dbsnapshotlv /mnt/snapshot
> 5. mysql server: unlock tables
> 6. create backup of /mnt/snapshot
> 7. umount /mnt/snapshot && lvremove /dev/datavg/dbsnapshotlv
> 
> This appears to be working, but there is a message in /var/log/messages
> which is a little disturbing. Whenever we create a snapshot and mount
> the resulting lv, we see the following:
> 
> Mar 15 05:00:05 PC kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
> Mar 15 05:00:05 PC kernel: EXT3-fs: write access will be enabled during recovery.
> Mar 15 05:00:05 PC kernel: kjournald starting.  Commit interval 5 seconds
> Mar 15 05:00:05 PC kernel: EXT3-fs: device-mapper(254,4): orphan cleanup on readonly fs
> Mar 15 05:00:05 PC kernel: EXT3-fs: device-mapper(254,4): 1 orphan inode deleted
> Mar 15 05:00:05 PC kernel: EXT3-fs: recovery complete.
> Mar 15 05:00:05 PC kernel: EXT3-fs: mounted filesystem with ordered data mode.
> ...
> Mar 15 06:00:05 PC kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
> Mar 15 06:00:05 PC kernel: EXT3-fs: write access will be enabled during recovery.
> Mar 15 06:00:07 PC kernel: kjournald starting.  Commit interval 5 seconds
> Mar 15 06:00:07 PC kernel: EXT3-fs: device-mapper(254,4): orphan cleanup on readonly fs
> Mar 15 06:00:07 PC kernel: EXT3-fs: device-mapper(254,4): 1 orphan inode deleted
> Mar 15 06:00:07 PC kernel: EXT3-fs: recovery complete.
> Mar 15 06:00:07 PC kernel: EXT3-fs: mounted filesystem with ordered data mode.
> ...
> Mar 16 01:00:06 PC kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
> Mar 16 01:00:06 PC kernel: EXT3-fs: write access will be enabled during recovery.
> Mar 16 01:00:08 PC kernel: kjournald starting.  Commit interval 5 seconds
> Mar 16 01:00:08 PC kernel: EXT3-fs: device-mapper(254,4): orphan cleanup on readonly fs
> Mar 16 01:00:08 PC kernel: EXT3-fs: device-mapper(254,4): 1 orphan inode deleted
> Mar 16 01:00:08 PC kernel: EXT3-fs: recovery complete.
> Mar 16 01:00:08 PC kernel: EXT3-fs: mounted filesystem with ordered data mode.
> 
> 
> When we mount the snapshot read-write, we instead see the following:
> 
> Mar 16 01:09:17 PC kernel: kjournald starting.  Commit interval 5 seconds
> Mar 16 01:09:17 PC kernel: EXT3 FS 2.4-0.9.19, 19 August 2002 on device-mapper(254,4), internal journal
> Mar 16 01:09:17 PC kernel: EXT3-fs: device-mapper(254,4): 1 orphan inode deleted
> Mar 16 01:09:17 PC kernel: EXT3-fs: recovery complete.
> Mar 16 01:09:17 PC kernel: EXT3-fs: mounted filesystem with ordered data mode.
> 
> 
> All we expected to see in /var/log/messages was something more like:
> 
> Mar 17 01:09:03 PC kernel: kjournald starting.  Commit interval 5 seconds
> Mar 17 01:09:03 PC kernel: EXT3 FS 2.4-0.9.19, 19 August 2002 on device-mapper(254,4), internal journal
> Mar 17 01:09:03 PC kernel: EXT3-fs: mounted filesystem with ordered data mode.
> 
> 
> Why would the snapshot volume have an orphaned inode? Doesn't the VFS patch
> allow us to snapshot EXT3? Do we need to quiesce the origin lv some other
> way?
> 
> 
> 
>  ------------------------------------------ 
>  Donald Murray, P.Eng.
>  Senior Software Engineer
>  Pason Systems Corp.
>  Direct: (403) 301-3473
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

 ------------------------------------------ 
 Donald Murray, P.Eng.
 Senior Software Engineer
 Pason Systems Corp.
 Direct: (403) 301-3473