Improving availability and guaranteeing integrity in ISO - internal sha1sums

[David Timms]

Jesse Keating wrote:
> On Friday 08 June 2007 14:24:47 Anthony Bryan wrote:
>> I was hoping Fedora could investigate using Metalinks for their ISO
>> downloads. Metalink is an XML format for listing all the ways you can
>> get a file or collection of files (mirrors + their location, rsync,
>> p2p) along with checksums to automatically repair a file in case of
>> error, signatures, language, OS/arch, and other metadata. It's mainly
>> used for large files like ISOs, where errors can be very frustrating.
>>
>> It's supported by about 20 programs on unix, mac, and win, including
>> aria2 (already in the Fedora repos). It's used by openSUSE,
>> OpenOffice.org, cURL, and many other distributions.
>>
>> Here's a screenshot of a Metalink download in the DownThemAll Firefox
>> extension (nightly build). What you don't see are all the mirrors and
>> checksums.
>> http://code.downthemall.net/maierman/metaselect4.png
>>
>> http://en.wikipedia.org/wiki/Metalink
> 
> This is something interesting, and I wonder if we could make use of 
> MirrorManager ( https://hosted.fedoraproject.org/projects/mirrormanager ) to 
> have dynamic .metalink files created with updated mirror readiness info.  
> Certainly something that looks worth looking into.

I recently worked on a tiny tool to use already downloaded files to 
reconstruct a full iso-image {eg Fedora test to Fedora release iso}. I 
could have made this a lot more sure that it was doing the right thing 
by comparing sha1sum of the individual files with reference sha1sum 
files. {for interest see:
http://www.redhat.com/archives/fedora-test-list/2007-June/msg00018.html }

The simplest way to do this would be for the iso spin system to perform 
an sha1sum * > SHA1SUM within each directory of an iso spin, and have 
each result inserted into the corresponding directory.

I could see other uses for embedding checksums within the iso archive :
- live test the files written within an iso image:
   - insert media {gets automounted}
   - cd /media/whatever
   - sha1sum -c SHA1SUM
   - actually proves the disc/drive/PC can properly read the individual 
files.
This could be extended to a simple tool in the root folder that can 
verify that all files in all folders on the cd/dvd are readable and 
correct. {python would be my weapon of choice, but }.

- Enable fast download method like above/jigdo/bittorrent to work on per 
file contained within the {iso} archive, rather than the iso file {whole 
or in fixed sized pieces that span/split files}
   - This makes it easier for a download tool to verify that a file 
found as a potential content of an iso is actually the correct file to 
insert.

Is inserting per folder checksum results into the iso image a do-able 
thing ? Would a patch to achieve such be considered ?

DaveT.