[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Fedora-livecd-list] [PATCH] cleanupDeleted and container_size - saves 5.5% output size on f7livecd-i686



Phillip Lougher wrote:

Douglas McClendon-2 wrote:
Douglas McClendon wrote:

Ok, so there is a negative consequence of a large container.  As I alluded
to before, mksquashfs apparently does not give any special consideration to sparse files. Naturally it can compress a file containing huge blocks of 0s pretty well, but it still appears to be going to an aweful lot of work to do
that.

Short of adding more intelligent handling of sparse files to squashfs (feasible?), another alternative which satisfies the above resizing scenarios would be this-


Unfortunately, I have been aware for sometime that the Fedora liveCD uses
sparse files, and Squashfs support was lacking.  Happily Squashfs now has
sparse file support (in CVS).  Max block size has also been increased from
64K to 1Mbyte, with the default 128 Kbytes.

very cool.


Some stats based on the Fedora-8-Test-1-Live-i686.iso
original squashfs.img 712495104 (681M)
sparse squashfs.img 709820416 (678M)
sparse squashfs.img with 128Kbyte blocks 700100608 (669M)

So, the compressed 0s used to take up about 2.5 Mbytes.  More significant is
the data and seek time saving in not reading all these useless compressed
0s.

time to do "dd if=os.img of=/dev/null" from CDROM

original squashfs.img, 221 seconds
sparse squashfs.img, 168 seconds
sparse squashfs.img with 128byte blocks, 171 seconds

on average about 20% speedup.

20% speedup of reading data into /dev/null isn't all that useful.

You didn't really mention what you were planning on using it for. Presumably you were also aware of my turboLiveInst patch, which accomplishes the 20% speedup in an actual copy situation.

The sparse support could be used, perhaps with more code being written to support sparse copying to block devices with python, to gain the performance enhancements of turboLiveInst in an alternate manner.

For reasons that I can only speculate to, Jeremy, and everyone else seems to have no interest in my turboLiveInst optimization approach. Perhaps this method will be more palatable for them for some reason.

The other thing that I'm curious about, is the performance impact of moving to 128kbyte blocks. I presume that is the compressed data size. I would like to see if, in some typical usage scenarios, whether or not that has a detrimental impact to desktop and general performance during livecd oepration. (i.e. for every 5kb text file read, it needs to uncompress 128Kb of compressed data, adding ram and latency penalties).

But the 3MB saved from the sparse support, is absolutely FREE. That is positively excellent, and what I had suspected was the case. Also, I'm guessing that the mksquashfs performance on a 4.0G sparse file with 2.0G of data will be vastly improved. (by my definition of vast, i.e. ~10%)

This actually reopens the usefulness of the container-size option which I had included with the first version of the cleanupDeleted patch. I.e. now it is no penalty whatsoever to go with a much larger container size. This will remove the need to perform ugly device-mapper-zero tricks to expand the rootfs size at runtime, if you have enough memory in your overlay device to support that. And assuming of course that either you are using the turboLiveInst patch, or the alternate method described above which requires a little more code to be written.

Jeremy- what do you think? I'd still like to get credit for all the work I did in discovering and fixing the problem. It would be a bit disappointing if you guys went and implemented the above alternate solution and took all the credit. I'd be willing to implement the above alternate solution, if you can give reasonable arguments as to why you think the existing turboLiveInst code is not acceptable.

Regards,

-dmc



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]