[Thincrust-devel] issues with zip packaging for appliances

[Bryan Kearney]

Perry Myers wrote:
> Daniel P. Berrange wrote:
>> On Thu, Nov 06, 2008 at 01:08:29AM -0500, Perry Myers wrote:
>>> Perry Myers wrote:
>>>> I've been playing with the -p zip option to appliance-creator and 
>>>> ran into an issue that I'm not sure how we can solve...
>>>>
>>>> If the output format of a disk image is raw there are two sizes, the 
>>>> logical size and the actual size.  So if I create a 20GB disk 
>>>> partition in the kickstart, but only use 1.5 GB the file appears to 
>>>> be 20GB but actually only takes 1.5GB on disk.
>>>>
>>>> When appliance-creator attempts to add this disk image to the zip 
>>>> archive it fails because the logical size of the file is > 4GB which 
>>>> is the limitation of an individual file in a zip archive.  You can 
>>>> pass the allowZip64=True parameter to Zip in python to allow the 
>>>> zip64 extensions to remove this limitation, but then the zip file 
>>>> can no longer be extracted using the standard unzip command line 
>>>> tools (also not sure if native Windows zip can handle this)
>>>>
>>>> It will probably be fairly common that appliance images will have 
>>>> logical sizes of > 4GB, so zip files are really only useful for 
>>>> packaging appliances if the image is qcow compressed instead of 
>>>> raw.  If the images are qcow, then the logical=actual and will 
>>>> generally be < 4GB.
>>>>
>>>> We can consider using another archiving format like tar, but then we 
>>>> can't  extract appliances on Windows with the default tools there.  
>>>> We can force qcow compression for zip packaging but that limits our 
>>>> hypervisor usage.
>>> Other random interesting datapoints...
>>>
>>> for the oVirt Appliance, if we have the output format as raw the 
>>> image size is 1.5GB since it is sparse
>>>
>>> with the output format as qcow2 the image only drops to 1.2GB
>>>
>>> if I take that qcow2 image and put it in a zip file the resulting zip 
>>> file is around 400MB
>>>
>>> if I zip the raw 1.5GB image it's also around 400MB (maybe a little 
>>> bit more but not much)  This only worked with the Zip64 extensions 
>>> turned on though
>>>
>>> So for distribution of appliances qcow2 does not really get you any 
>>> savings.  You'd still have to use a compressed package format.  We've 
>>> been doing that up until now by putting the image into an RPM which 
>>> compresses it.
>>>
>>> So the fundamental question here is... what packaging format can we 
>>> use that will have good compression, is natively supported on a wide 
>>> variety of platforms (Unix, Windows, Linux) and will hold raw images 
>>> that are fairly large.
>>
>> Well for Linux world ZIP, tar.gz, tar.bz2 are commonly installed. You
>> won't find things like 7-zip in many commercial distros - eg RHEL.
>>
>> For Windows world, ZIP is obviously standard natively done by Explorer,
>> but WinZIP supports tar.gz too.
>> Not sure what coverage of zip64 is like in Windows ?
> 
> Perhaps what we need to do is just use tar.gz file and require that 
> Windows users grab 7-zip or WinZIP in order to extract.

I think this is fine. If the user is comfortable with installing virtual 
appliances into a hypervisor I believe they will be able to handle this 
requirement.

-- bk