rpms/zerofree/EL-5 import.log, NONE, 1.1 index.html, NONE, 1.1 sparsify.c, NONE, 1.1 zerofree.spec, NONE, 1.1 .cvsignore, 1.1, 1.2 sources, 1.1, 1.2

Mon May 18 08:15:36 UTC 2009

Author: rjones

Update of /cvs/pkgs/rpms/zerofree/EL-5
In directory cvs1.fedora.phx.redhat.com:/tmp/cvs-serv15205/EL-5

Modified Files:
	.cvsignore sources 
Added Files:
	import.log index.html sparsify.c zerofree.spec 
Log Message:
Initial import.

--- NEW FILE import.log ---
zerofree-1_0_1-5_fc11:EL-5:zerofree-1.0.1-5.fc11.src.rpm:1242634384


--- NEW FILE index.html ---
<html>
<head>
<title>Keeping filesystem images sparse</title>
</head>
<body>
<h2>Keeping filesystem images sparse</h2>
<p>
Filesystem images in local files can be used by many PC emulators and
virtual machines
(<a href="http://user-mode-linux.sourceforge.net/">user-mode Linux</a>,
<a href="http://fabrice.bellard.free.fr/qemu/">QEMU</a> and
<a href="http://wiki.xensource.com/xenwiki/">Xen</a>, to name but three).
Typically these filesystems are created as sparse files using
commands like:
<p>
<pre>
   dd if=/dev/zero of=fs.image bs=1024 seek=2000000 count=0
   /sbin/mke2fs fs.image
</pre>
where the enormous <code>seek</code> value causes <code>dd</code> to move
forward by 2GB before writing nothing at all.  This results in the
creation of a sparse file which takes disk space only for blocks which are
actually used:
<p>
<pre>
   $ ls -l fs.image 
   -rw-rw-r--  1 rmy rmy 2048001024 Apr 18 19:10 fs.image
   $ du -s fs.image 
   31692   fs.image
</pre>
As the filesystem is used, more and more of the non-existent blocks are
filled with data and the size of the file on disk grows.  Sometimes it would
be nice to be able to reclaim unused blocks from a filesystem image.  However,
deleting files from the image doesn't return the space to the
underlying filesystem:  even free blocks in the image still consume space.
Reclaiming the space can be achieved in two stages:
<ul>
<li>Fill unused blocks with zeroes
<li>Make the file sparse again
</ul>
<p>
One traditional way to zero unused blocks is to create a file that fills
all the free space:
<p>
<pre>
   dd if=/dev/zero of=junk
   sync
   rm junk
</pre>
<p>
The disadvantage of <code>dd</code> in this context is that it destroys
any sparseness that exists:  free blocks that were originally represented
as holes in the image file are replaced with actual blocks containing
zeroes.
<p>
As an alternative approach, and as practice in mucking about with ext2
filesystems, I've written a utility which scans the free blocks in an
ext2 filesystem and fills any non-zero blocks with zeroes.
(The source, <a href="zerofree-1.0.1.tgz">zerofree-1.0.1.tgz</a>, is
available for download.)  The <code>zerofree</code> utility is faster
than <code>dd</code>, especially when the filesystem is already partly sparse.
The filesystem to be processed should be unmounted or mounted read-only.
<p>
Better than either of these would be to have the guest kernel keep the free
blocks empty.  My original inspiration was the
<a href="http://www.uwsg.iu.edu/hypermail/linux/kernel/0401.3/1058.html">
ext2fs privacy (i.e. secure deletion) patch</a> described in a Linux
kernel mailing list thread.  I've also made use of a later patch for ext3
entitled
<a href="http://marc.theaimsgroup.com/?l=linux-fsdevel&m=113986429313502&w=2">Secure Deletion Functionality in ext3</a>
from the linux-fsdevel mailing list.  (See also the authors' paper on
<a href="http://www.filesystems.org/project-sdfs.html">Secure Deletion File Systems</a>.)
I've modified the patches to make them more suitable for the present purpose.
<ul>
<li><a href="linux-2.6.25-zerofree2.patch">linux-2.6.25-zerofree2.patch</a> (for ext2 filesystems)</li>
<li><a href="linux-2.6.25-zerofree3.patch">linux-2.6.25-zerofree3.patch</a> (for ext3 filesystems)</li>
</ul>
When a filesystem is mounted with the <code>zerofree</code> option (added
by these patches) all the blocks freed when a file is deleted are filled
with zeroes.
Remember, this extra work will hurt disk performance.
Note that the ext3 patch doesn't support data journalling
mode, so deleted metadata isn't zeroed.  It also hasn't been tested
as thoroughly as the patch for ext2.
<p>
However, the above techniques are only half the story:  the empty free
blocks still consume space in the underlying filesystem, so something
must to be done to reclaim that space.  One approach would be to
implement a system call, like the legendary
<a href="http://www.uwsg.iu.edu/hypermail/linux/kernel/0106.3/1180.html">
sys_punch</a>, which could be used to write a utility to make any
suitable file sparse.
<p>
An existing alternative is to use the sparse file handling capabilities
of the GNU <code>cp</code> command to take a copy of the filesystem image with
<code>cp --sparse=always</code> (though this does require the original
and sparse files to exist at the same time, which may be inconvenient).
<p>
As an alternative alternative I've written a utility which can make
any specified files on an ext2 filesystem
sparse, <a href="sparsify.c">sparsify.c</a>.  This doesn't require any
additional disk space to work its magic, but it does require that the
filesystem containing the filesystem image is unmounted, which is just a
different sort of inconvenience.
<p>
(The usual disclaimers apply:  this worked for me
when I tested it but it might destroy your data.  Only use it on
disposable filesystems, or have a full backup available.  <code>e2fsck</code>
is your friend.)
<p>
As an example, suppose we have an unmounted filesystem
image, <code>fs.image</code>, in the directory <code>/data</code>, which is the
root of the <code>/dev/hda2</code> filesystem.  We can reclaim deleted
blocks and make it sparse like so:
<p>
<pre>
   zerofree /data/fs.image
   umount /data
   sparsify /dev/hda2 /fs.image
   mount /data
</pre>
<p>
<hr>
<address>
<a href="mailto:rmy at tigress.co.uk">Ron Yorston</a><br>
29th July 2008
</address>
</body>
</html>


--- NEW FILE sparsify.c ---
/*
 * sparsify - a tool to make files on an ext2 filesystem sparse
 *
 * Copyright (C) 2004 R M Yorston
 *
 * This file may be redistributed under the terms of the GNU General Public
 * License.
 */
#include <ext2fs/ext2fs.h>
#include <stdio.h>
#include <unistd.h>
#include <string.h>

#define USAGE "usage: %s [-n] [-v] filesystem filename ...\n"

struct process_data {
	struct ext2_inode *inode ;
	unsigned char *buf;
	int dryrun ;
	int changed ;
} ;

static int process(ext2_filsys fs, blk_t *blocknr, e2_blkcnt_t blockcnt,
					blk_t ref_block, int ref_offset, void *priv)
{
	struct process_data *p ;
	errcode_t ret ;
	int i, group ;

	p = (struct process_data *)priv ;

	if ( blockcnt >= 0 ) {
		ret = io_channel_read_blk(fs->io, *blocknr, 1, p->buf);
		if ( ret ) {
			return BLOCK_ABORT ;
		}

		for ( i=0; i < fs->blocksize; ++i ) {
			if ( p->buf[i] ) {
				break ;
			}
		}

		if ( i == fs->blocksize && !p->dryrun ) {
			ext2fs_unmark_block_bitmap(fs->block_map, *blocknr) ;
			group = ext2fs_group_of_blk(fs, *blocknr);
			fs->group_desc[group].bg_free_blocks_count++;
			fs->super->s_free_blocks_count++ ;
			/* the inode counts blocks of 512 bytes */
			p->inode->i_blocks  -= fs->blocksize / 512 ;
			*blocknr = 0 ;
			/* direct blocks need to be zeroed in the inode */
			if ( blockcnt < EXT2_NDIR_BLOCKS ) {
				p->inode->i_block[blockcnt] = 0 ;
			}
			p->changed = 1 ;
			return BLOCK_CHANGED ;
		}
	}

	return 0 ;
}

int main(int argc, char **argv)
{
	int verbose = 0 ;
	int dryrun = 0 ;
	errcode_t ret ;
	int flags ;
	int superblock = 0 ;
	int open_flags = EXT2_FLAG_RW ;
	int blocksize = 0 ;
	ext2_filsys current_fs = NULL;
	struct ext2_inode inode ;
	ext2_ino_t root, cwd, inum ;
	int i, c ;
	struct process_data pdata ;

	while ( (c=getopt(argc, argv, "nv")) != -1 ) {
		switch (c) {
		case 'n' :
			dryrun = 1 ;
			break ;
		case 'v' :
			verbose = 1 ;
			break ;
		default :
			fprintf(stderr, USAGE, argv[0]) ;
			return 1 ;
		}
	}

	if ( argc < optind+2 ) {
		fprintf(stderr, USAGE, argv[0]) ;
		return 1 ;
	}

	ret = ext2fs_check_if_mounted(argv[optind], &flags) ;
	if ( ret ) {
		fprintf(stderr, "%s: failed to determine filesystem mount state  %s\n",
					argv[0], argv[optind]) ;
		return 1 ;
	}

	if ( flags & EXT2_MF_MOUNTED ) {
		fprintf(stderr, "%s: filesystem %s is mounted\n",
					argv[0], argv[optind]) ;
		return 1 ;
	}

	ret = ext2fs_open(argv[optind], open_flags, superblock, blocksize,
							unix_io_manager, &current_fs);
	if ( ret ) {
		fprintf(stderr, "%s: failed to open filesystem %s\n",
					argv[0], argv[optind]) ;
		return 1 ;
	}

	pdata.buf = (unsigned char *)malloc(current_fs->blocksize) ;
	if ( pdata.buf == NULL ) {
		fprintf(stderr, "%s: out of memory (surely not?)\n", argv[0]) ;
		return 1 ;
	}

	ret = ext2fs_read_inode_bitmap(current_fs);
	if ( ret ) {
		fprintf(stderr, "%s: error while reading inode bitmap\n", argv[0]);
		return 1 ;
	}

	ret = ext2fs_read_block_bitmap(current_fs);
	if ( ret ) {
		fprintf(stderr, "%s: error while reading block bitmap\n", argv[0]);
		return 1 ;
	}

	root = cwd = EXT2_ROOT_INO ;

	for ( i=optind+1; i<argc; ++i ) {
		if ( verbose ) {
			printf("processing %s\n", argv[i]) ;
		}

		ret = ext2fs_namei(current_fs, root, cwd, argv[i], &inum) ;
		if ( ret ) {
			fprintf(stderr, "%s: failed to find file %s\n", argv[0], argv[i]) ;
			continue ;
		}

		ret = ext2fs_read_inode(current_fs, inum, &inode) ;
		if ( ret ) {
			fprintf(stderr, "%s: failed to open inode %d\n", argv[0], inum) ;
			continue ;
		}

		if ( !ext2fs_inode_has_valid_blocks(&inode) ) {
			fprintf(stderr, "%s: file %s has no valid inodes\n", argv[0],
					argv[i]) ;
			continue ;
		}

		pdata.inode = &inode ;
		pdata.dryrun = dryrun ;
		pdata.changed = 0 ;
		ret = ext2fs_block_iterate2(current_fs, inum, BLOCK_FLAG_DATA_ONLY,
				NULL, process, &pdata) ;
		if ( ret ) {
			fprintf(stderr, "%s: failed to process file %s\n", argv[0],
					argv[i]) ;
			continue ;
		}

		if ( pdata.changed ) {
			ret = ext2fs_write_inode(current_fs, inum, &inode) ;
			if ( ret ) {
				fprintf(stderr, "%s: failed to write inode data %s\n", argv[0],
						argv[i]) ;
				continue ;
			}

			ext2fs_mark_bb_dirty(current_fs) ;
			ext2fs_mark_super_dirty(current_fs) ;
		}
	}

	ret = ext2fs_close(current_fs) ;
	if ( ret ) {
		fprintf(stderr, "%s: error while closing filesystem\n", argv[0]) ;
		return 1 ;
	}

	return 0 ;
}


--- NEW FILE zerofree.spec ---
Summary:        Utility to force unused ext2 inodes and blocks to zero
Name:           zerofree
Version:        1.0.1
Release:        5%{?dist}
License:        GPL+
Group:          System Environment/Libraries

Source0:        http://intgat.tigress.co.uk/rmy/uml/%{name}-%{version}.tgz
Source1:        http://intgat.tigress.co.uk/rmy/uml/sparsify.c
Source2:        http://intgat.tigress.co.uk/rmy/uml/index.html
URL:            http://intgat.tigress.co.uk/rmy/uml/

BuildRequires:  e2fsprogs-devel

BuildRoot:      %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)


%description
zerofree is a utility to set unused filesystem inodes and blocks of an
ext2 filesystem to zero.  This can improve the compressibility and
privacy of an ext2 filesystem.

This tool was inspired by the ext2fs privacy (i.e. secure deletion)
patch described in a Linux kernel mailing list thread.

WARNING: The filesystem to be processed should be unmounted or mounted
read-only.  The tool tries to check this before running, but you
should be careful.


%prep
%setup -q
cp -p %{SOURCE1} .
cp -p %{SOURCE2} .


%build
make CC="gcc $RPM_OPT_FLAGS"
gcc $RPM_OPT_FLAGS sparsify.c -o sparsify -lext2fs


%install
rm -rf $RPM_BUILD_ROOT

install -D -p -m 755 zerofree $RPM_BUILD_ROOT%{_sbindir}/zerofree
install -D -p -m 755 sparsify $RPM_BUILD_ROOT%{_sbindir}/sparsify


%files
%defattr(-,root,root,-)
%doc COPYING index.html
%{_sbindir}/zerofree
%{_sbindir}/sparsify


%clean
rm -rf $RPM_BUILD_ROOT


%changelog
* Fri May 15 2009 Richard W.M. Jones <rjones at redhat.com> - 1.0.1-5
- Include the index file as a source file.
- Improve the description, remove spelling mistakes and other typos.
- Use the upstream SRPM directly, unpacking source from it.
- Fix use of dist macro.
- Pass the RPM OPTFLAGS to C compiler (should also fix debuginfo pkg).
- Use 'cp -p' to preserve timestamps when copying index.html file.
- Fix the defattr line.
- License is GPL+ (any version of the GPL including 1).
- Use a simpler install command to install the binary.
- Fix the upstream URL to point to the real original project.
- Add the sparsify command.

* Thu May 14 2009 Richard W.M. Jones <rjones at redhat.com> - 1.0.1-1
- Initial packaging for Fedora, based on R P Herrold's package.

* Wed May 13 2009 R P Herrold <info at owlriver.com> - 1.0.1-1
- initial packaging


Index: .cvsignore
===================================================================
RCS file: /cvs/pkgs/rpms/zerofree/EL-5/.cvsignore,v
retrieving revision 1.1
retrieving revision 1.2
diff -u -p -r1.1 -r1.2
--- .cvsignore	16 May 2009 00:03:02 -0000	1.1
+++ .cvsignore	18 May 2009 08:15:05 -0000	1.2
@@ -0,0 +1 @@
+zerofree-1.0.1.tgz


Index: sources
===================================================================
RCS file: /cvs/pkgs/rpms/zerofree/EL-5/sources,v
retrieving revision 1.1
retrieving revision 1.2
diff -u -p -r1.1 -r1.2
--- sources	16 May 2009 00:03:02 -0000	1.1
+++ sources	18 May 2009 08:15:05 -0000	1.2
@@ -0,0 +1 @@
+a8c772fdd134448f25ab4e7e12004595  zerofree-1.0.1.tgz


