[Thincrust-devel] issues with zip packaging for appliances

[Perry Myers]

Bryan Kearney wrote:
> Perry Myers wrote:
>> Daniel P. Berrange wrote:
>>> On Thu, Nov 06, 2008 at 01:08:29AM -0500, Perry Myers wrote:
>>>> Perry Myers wrote:
>>>>> I've been playing with the -p zip option to appliance-creator and 
>>>>> ran into an issue that I'm not sure how we can solve...
>>>>>
>>>>> If the output format of a disk image is raw there are two sizes, 
>>>>> the logical size and the actual size.  So if I create a 20GB disk 
>>>>> partition in the kickstart, but only use 1.5 GB the file appears to 
>>>>> be 20GB but actually only takes 1.5GB on disk.
>>>>>
>>>>> When appliance-creator attempts to add this disk image to the zip 
>>>>> archive it fails because the logical size of the file is > 4GB 
>>>>> which is the limitation of an individual file in a zip archive.  
>>>>> You can pass the allowZip64=True parameter to Zip in python to 
>>>>> allow the zip64 extensions to remove this limitation, but then the 
>>>>> zip file can no longer be extracted using the standard unzip 
>>>>> command line tools (also not sure if native Windows zip can handle 
>>>>> this)
>>>>>
>>>>> It will probably be fairly common that appliance images will have 
>>>>> logical sizes of > 4GB, so zip files are really only useful for 
>>>>> packaging appliances if the image is qcow compressed instead of 
>>>>> raw.  If the images are qcow, then the logical=actual and will 
>>>>> generally be < 4GB.
>>>>>
>>>>> We can consider using another archiving format like tar, but then 
>>>>> we can't  extract appliances on Windows with the default tools 
>>>>> there.  We can force qcow compression for zip packaging but that 
>>>>> limits our hypervisor usage.
>>>> Other random interesting datapoints...
>>>>
>>>> for the oVirt Appliance, if we have the output format as raw the 
>>>> image size is 1.5GB since it is sparse
>>>>
>>>> with the output format as qcow2 the image only drops to 1.2GB
>>>>
>>>> if I take that qcow2 image and put it in a zip file the resulting 
>>>> zip file is around 400MB
>>>>
>>>> if I zip the raw 1.5GB image it's also around 400MB (maybe a little 
>>>> bit more but not much)  This only worked with the Zip64 extensions 
>>>> turned on though
>>>>
>>>> So for distribution of appliances qcow2 does not really get you any 
>>>> savings.  You'd still have to use a compressed package format.  
>>>> We've been doing that up until now by putting the image into an RPM 
>>>> which compresses it.
>>>>
>>>> So the fundamental question here is... what packaging format can we 
>>>> use that will have good compression, is natively supported on a wide 
>>>> variety of platforms (Unix, Windows, Linux) and will hold raw images 
>>>> that are fairly large.
>>>
>>> Well for Linux world ZIP, tar.gz, tar.bz2 are commonly installed. You
>>> won't find things like 7-zip in many commercial distros - eg RHEL.
>>>
>>> For Windows world, ZIP is obviously standard natively done by Explorer,
>>> but WinZIP supports tar.gz too.
>>> Not sure what coverage of zip64 is like in Windows ?
>>
>> Perhaps what we need to do is just use tar.gz file and require that 
>> Windows users grab 7-zip or WinZIP in order to extract.
> 
> I think this is fine. If the user is comfortable with installing virtual 
> appliances into a hypervisor I believe they will be able to handle this 
> requirement.

David, can you quickly put in support for tar.gz and tar.bz2 as output 
formats in addition to zip?  Right now appliance tools only supports zip.

Thanks!

Perry