[Fedora-livecd-list] [PATCH] cleanupDeleted and container_size - saves 5.5% output size on f7livecd-i686

[Douglas McClendon]

Phillip Lougher wrote:
> 
> Douglas McClendon-2 wrote:
>> Douglas McClendon wrote:
>>
>> Ok, so there is a negative consequence of a large container.  As I alluded
>> to 
>> before, mksquashfs apparently does not give any special consideration to
>> sparse 
>> files.  Naturally it can compress a file containing huge blocks of 0s
>> pretty 
>> well, but it still appears to be going to an aweful lot of work to do
>> that.
>>
>> Short of adding more intelligent handling of sparse files to squashfs 
>> (feasible?), another alternative which satisfies the above resizing
>> scenarios 
>> would be this-
>>
> 
> Unfortunately, I have been aware for sometime that the Fedora liveCD uses
> sparse files, and Squashfs support was lacking.  Happily Squashfs now has
> sparse file support (in CVS).  Max block size has also been increased from
> 64K to 1Mbyte, with the default 128 Kbytes.

very cool.

> 
> Some stats based on the Fedora-8-Test-1-Live-i686.iso 
> 
> original squashfs.img 712495104 (681M)
> sparse squashfs.img 709820416 (678M)
> sparse squashfs.img with 128Kbyte blocks 700100608 (669M)
> 
> So, the compressed 0s used to take up about 2.5 Mbytes.  More significant is
> the data and seek time saving in not reading all these useless compressed
> 0s.
> 
> time to do "dd if=os.img of=/dev/null" from CDROM
> 
> original squashfs.img, 221 seconds
> sparse squashfs.img, 168 seconds
> sparse squashfs.img with 128byte blocks, 171 seconds
> 
> on average about 20% speedup.

20% speedup of reading data into /dev/null isn't all that useful.

You didn't really mention what you were planning on using it for. 
Presumably you were also aware of my turboLiveInst patch, which 
accomplishes the 20% speedup in an actual copy situation.

The sparse support could be used, perhaps with more code being written 
to support sparse copying to block devices with python, to gain the 
performance enhancements of turboLiveInst in an alternate manner.

For reasons that I can only speculate to, Jeremy, and everyone else 
seems to have no interest in my turboLiveInst optimization approach. 
Perhaps this method will be more palatable for them for some reason.

The other thing that I'm curious about, is the performance impact of 
moving to 128kbyte blocks.  I presume that is the compressed data size. 
  I would like to see if, in some typical usage scenarios, whether or 
not that has a detrimental impact to desktop and general performance 
during livecd oepration.  (i.e. for every 5kb text file read, it needs 
to uncompress 128Kb of compressed data, adding ram and latency penalties).

But the 3MB saved from the sparse support, is absolutely FREE.  That is 
positively excellent, and what I had suspected was the case.  Also, I'm 
guessing that the mksquashfs performance on a 4.0G sparse file with 2.0G 
of data will be vastly improved.  (by my definition of vast, i.e. ~10%)

This actually reopens the usefulness of the container-size option which 
I had included with the first version of the cleanupDeleted patch.  I.e. 
now it is no penalty whatsoever to go with a much larger container size. 
  This will remove the need to perform ugly device-mapper-zero tricks to 
expand the rootfs size at runtime, if you have enough memory in your 
overlay device to support that.  And assuming of course that either you 
are using the turboLiveInst patch, or the alternate method described 
above which requires a little more code to be written.

Jeremy- what do you think?  I'd still like to get credit for all the 
work I did in discovering and fixing the problem.  It would be a bit 
disappointing if you guys went and implemented the above alternate 
solution and took all the credit.  I'd be willing to implement the above 
alternate solution, if you can give reasonable arguments as to why you 
think the existing turboLiveInst code is not acceptable.

Regards,

-dmc