JTE - Jigdo Template Export(Jigdo模板导出)

JTE - Jigdo Template Export

介绍

Jigdo 是一个有助于分发向CD DVD镜像这样的打文件的工具.详细信息请看 Richard Atterer's 的站点.

使用Jigdo使用户更加有效的下载发布在web上的Debian CD或DVD( 节约了时间和带宽 )

Jigdo 普通并且强大,她可以用于任何由小文件组成的大文件,但是这样代价也是十分大的.从ISO文件创建.jigdo文件效率非常低,要处理在ISO上的文件,jigdo要计算和比较其中的每个文件,在4.5G这样大的DVD上要花上几个小时的时间.

有几个改进这个过程的方法:

1. 修改 jigdo让他知道 ISO镜像的内部结构并且可以高效的扫描文件( 不好,对于 jigdo不是十分通用 )
2. 编写帮助工具,dump额外的信息用于使jigdo处理ISO文件
3. 修补( patch )mkisofs 使之在制作ISO文件使生成相应的 .jigdo索引文件和 .template模板文件.

我现在完成了第三个,被称作 JTE( jigdo 模板输出).代码运行的很好,花非常少的时间分别运行了JTE 和 jigdo,输出的.jigdo文件和 .template文件也能正常的工作.

mkimage, 一个简单而且快速的从 .jigdo文件和.template文件重新构造镜像文件的工具. 他不具有任何下载,并修复缺失文件的功能,但是他可以列出你需要的缺失的文件.对于已经有本地iso镜像的用户,他的运行速度更快.

iso-image.pl is a CGI script to wrap around mkimage if you'd like to be able to offer images for HTTP download without using up multiple gigabytes of disk space. And for added network efficiency the perl CGI also supports HTTP v1.1 byte ranges so clients can resume aborted downloads.

jigit is a first attempt at a user-friendly wrapper for mkimage on a user's machine.

The addition of these extra tools means that I'm now distributing JTE as a source tarball and Debian packages rather than just a mkisofs patch; the patch is inside the source tarball too.

NEW:The packages and source are now named jigit to match the wrapper script.

下载

NOTE: the mkisofs patch in the jigit source tarball was generated against an already-built Debian cdrtools tree. To be able to apply the patch I have, you will need to run the Debian build on cdrtools (e.g. dpkg-buildpackage -rfakeroot) first.

Jigit has now been uploaded and accepted into the Debian archive, so you should get binary packages from there (current is 1.14-2). Source and backported versions are here:

• jigit_1.14.orig.tar.gz
• jigit_1.14-woody1.diff.gz
• jigit_1.14-woody1.dsc
• jigit_1.14-woody1_i386.changes
• jigit_1.14-woody1_i386.deb

Older versions of the JTE code are still available:

Version	Filename	Signature	Applies against	Notes
1.12 (2004/09/05)	jte-1.12.tar.bz2	jte-1.12.tar.bz2.asc	cdrtools 2.0+a34-1	Improved the documentation, listing new options to mkimage. Fixed/updated the debian-cd patch.
1.11 (2004/09/05)	jte-1.11.tar.bz2	jte-1.11.tar.bz2.asc	cdrtools 2.0+a34-1	Added rsync sums into JTE output. Added jigit script. Added extra options to mkimage to support jigit.
1.10 (2004/09/01)	jte-1.10.tar.bz2	jte-1.10.tar.bz2.asc	cdrtools 2.0+a34-1	Removed the last md5 check; no longer needed. Added new -jigdo-force-md5 option; any files matching specified paths MUST match an MD5 supplied, or fail. Fixed a couple of typos. Added initial work for bz2 support. debian-cd diff now includes updated grab_md5 and jigdo_cleanup scripts that must be used. mkisofs now leaves a blank [Servers] section in the .jigdo file - jigdo_cleanup does the work.
1.9 (2004/08/30)	jte-1.9.tar.bz2	jte-1.9.tar.bz2.asc	cdrtools 2.0+a34-1	Added MD5 reverse lookups to pick up boot files; helps reduce CD/DVD #1 size substantially, BUT -md5-list option now mandatory. Fixed sizing bug in HFS hybrid template code.
1.8 (2004/08/27)	jte-1.8.tar.bz2	jte-1.8.tar.bz2.asc	cdrtools 2.0+a34-1	Tweaked the -exclude option to reduce template file sizes
1.7 (2004/08/19)	jte-1.7.tar.bz2	jte-1.7.tar.bz2.asc	cdrtools 2.0+a34-1	Fixed a couple of silly bugs introduced; don't assume template creation. Added README files and man page sections for the new boot support.
1.6 (2004/07/14)	jte-1.6.tar.bz2	jte-1.6.tar.bz2.asc	cdrtools 2.0+a30-1	Added code to check MD5 sums in mkisofs; should no longer need to run mirror_check any more. See the -md5-list option.
1.5 (2004/07/05)	jte-1.5.tar.bz2	jte-1.5.tar.bz2.asc	cdrtools 2.0+a30-1	Minor bug-fixes and updates to mkimage Added iso-image.pl wrapper script
1.4 (2004/06/26)	jte-1.4.tar.bz2	jte-1.4.tar.bz2.asc	cdrtools 2.0+a30-1	Added mkimage Moved utils and patch into a tarball
1.3 (2004/06/18)	mkisofs-JTE-v1.3.gz	mkisofs-JTE-v1.3.gz.asc	cdrtools 2.0+a30-1	Fixed crash without jigdo options Cleaned up mipsel boot code Make it easier to specify the boot command line for hppa Minor cosmetic .jigdo cleanup
1.2 (2004/06/14)	mkisofs-JTE-v1.2.gz	mkisofs-JTE-v1.2.gz.asc	cdrtools 2.0+a27-1	Added boot support for alpha, hppa, mips, mipsel Cosmetic changes to the .jigdo file
1.1 (2004/06/10)	mkisofs-JTE-v1.1.gz	mkisofs-JTE-v1.1.gz.asc	cdrtools 2.0+a27-1	Removed .jte file code Large cleanup to reduce patch size
1.0 (2004/06/07)	mkisofs-JTE.patch.gz	mkisofs-JTE.patch.gz.asc	cdrtools 2.0+a27-1	Initial release Produced .jigdo, .template and .jte files

如何使用JTE

To use the jigdo creation code, specify the location of the output .jigdo and .template files alongside the ISO image. You can also specify the minimum size beneath which files will just be dropped into the binary template file data rather than listed as separate files to be found on the mirror, and exclude patterns to ignore certain files in the same way. And paths in the original filesystem can be mapped onto more global namespaces using the [Servers] section in the .jigdo file. For example:

mkisofs -J -r -o /home/steve/test1.iso /
        -jigdo-jigdo /home/steve/test1.jigdo /
        -jigdo-template /home/steve/test1.template /
        -jigdo-min-file-size 16384 /
        -jigdo-ignore "README*" /
        -jigdo-force-md5 "/pool/" /
        -jigdo-map Debian=/mirror/debian /
        -md5-list /home/steve/md5.list /
        /mirror/jigdo-test

If the -jigdo-* options are not used, the normal mkisofs execution path is not affected at all. The above invocation will create 3 output files (.iso, .jigdo and .template). Multiple -jigdo-ignore and -jigdo-map options are accepted, for multiple ignore and map patterns.

Use the -md5-list option to specify the location of a list of files and their md5sums in normal md5sum format. mkisofs will then compare the checksum of each file it is asked to write against the checksum of that file in the list. It will abort on any mismatches. The MD5 list file must list all the files that are expected to be found and listed in the output .jigdo file. The -jigdo-force-md5 option specifies a path where all files are expected to have an MD5 entry (e.g. /pool/). Then if any files do not have a match, they must have been corrupted and mkisofs will abort.

More options have now been added in version 1.2 onwards so that you can specify the location of boot files within the ISO image. Previously the four architectures alpha, hppa, mips and mipsel needed separate tools to make an ISO image bootable. This also made life very hard when trying to produce jigdo files. Instead, I've folded boot support for those architectures into this patch so that mkisofs will do all the work (see below). I'm working with Joerg Schilling now on merging at least the boot support into upstream cdrtools in a future release.

Alpha
-alpha-boot <FILE>	Specify the location of the boot image (relative to the root of the ISO image)
Hppa
-hppa-cmdline <CMDLINE>	Specify the hppa boot command line. Separate elements with commas or spaces.
-hppa-kernel-32 <FILE>	Specify the location of the 32-bit boot image (relative to the root of the ISO image)
-hppa-kernel-64 <FILE>	Specify the location of the 64-bit boot image (relative to the root of the ISO image)
-hppa-bootloader <FILE>	Specify the location of the bootloader code (iplboot, relative to the root of the ISO image)
-hppa-ramdisk <FILE>	Specify the location of the ramdisk (relative to the root of the ISO image)
Mips
-mips-boot <FILE>	Specify the location of the boot image (relative to the root of the ISO image)
Mipsel Mipsel is awkward because we have to parse the ELF header of the boot loader and write some locations from it into the boot sector. Ick!
-mipsel-boot <FILE>	Specify the location of the boot image (relative to the root of the ISO image)

JTE 怎样工作

I've hooked all the places in mkisofs where it will normally write image data. All the normal data write calls (directory entries etc.) I simply copy through and build into the template file. Any file data entries are instead passed through with information about the original file. If that file is large enough (see -jigdo-min-file-size above), I grab the filename and the MD5 of the file's data. If that MD5, size and length match an entry in the md5-list, I can just write a file match record into the template file (and then the jigdo file) instead of the file data itself.

JTE有多快?

On my laptop (600MHz P3, slow laptop disk) I can make a template file in parallel with the ISO image from a typical 500MB data set in about 2 minutes. By simply not creating the ISO (-o /dev/null), this time halves again. The data set I'm using here is a copy of the woody i386 r2 update CD, as it's a handy image I had lying around.

On my faster home server machine (1.7GHz Athlon, 512MB RAM, fast SCSI disks), I can produce a 7GB DVD iso image with the jigdo and template files in about 8 minutes. A debian-cd run from start to finish to create DVD images takes about 25 minutes per architecture.

Mkisofs is normally I/O-bound on this system, but when running the jigdo creation code it's now CPU bound - it's now running 2 MD5 checksums on each data block that it sees. To boost performance when creating images on a large SMP machine, running several copies of debian-cd in parallel should parallelise nicely - ideally run the CD and DVD versions of each architecture together to get maximum benefit from the dentry and page cache.

如何使用mkimage

mkimage is a faster, local-only version of "jigdo-file make-image", again written in portable C. It takes a few options:

-f <MD5 file>	Specify a file containing MD5sums for files we should attempt to use when rebuilding the image
-j <jigdo file>	Specify the input jigdo file
-t <template file>	Specify the input template file
-m <item=path>	Map <item> to <path> to find the files in the mirror
-M <Missing file>	Don't attempt to build the image; just verify that all the components needed are available. If some are missing, list them in the specified file.
-v	Make the output logging more verbose.
-l <log file>	Specify a logfile. If not specified, will log to stderr just like mkisofs
-q	Don't bother checking md5sums of the input files, or of the output image. WARNING: this may lead to corrupt images, but is much faster.
-s <start offset>	Specify where to start in the image (in bytes). If not specified, will start at the beginning (offset 0). Added for iso-image.pl use
-e <end offset>	Specify where to end in the image (in bytes). If not specified, will run all the way to the end of the image. Added for iso-image.pl use
-z	Don't attempt to reassemble the image; simply parse the image descriptor in the template file and print the image size. Added for iso-image.pl use

Specifying a start or end offset implies -q - it's difficult to check MD5 sums if the full image is not generated!

如何使用iso-image.pl

iso-image.pl is a small perl wrapper script written to drive mkimage and turn it into a CGI. It will parse the incoming request (including byte-ranges) and call mkimage to actually generate the image pieces wanted.

Configuration is simple: place iso-image.pl in a cgi-bin directory and set various paths in the script:

• The path to mkimage
• A log file location
• The path to the template and jigdo files
• Other options to mkimage (e.g. -q and match locations)

如何使用jigit

jigit will automatically download the files needed to create a Debian/Ubuntu CD, using as many files available locally as possible before downloading any that are missing.

Configure jigit by editing /etc/jigit.conf or $HOME/.jigit.conf. The two settings that matter are:

HOST	The base URL of the jigit update site.
TMPDIR	Where jigit should store its temporary files. These may take up a LOT of space!

Then run jigit <CD name> to grab the CD you need. If you have any local files that may reduce the amount you need to download (e.g. from an earlier CD or a local mirror), tell jigit where to find them. It will automatically look in your apt cache too.

It should look something like this:

sledge:~$ jigit wibble
Downloading config:   http://tack/jigit/wibble.conf
Downloading jigdo:    http://tack/jigit/wibble.jigdo
Downloading template: http://tack/jigit/wibble.template
If you have a mirror, or any previous CD or CD image(s) available,
where are they mounted?
Say "none" if you have none; separate multiple entries with spaces
> [none] 

Checking MD5 sums of files in /var/cache/apt/archives:
 /var/cache/apt/archives/netpbm_2%3a10.0-8_amd64.deb                        
Checking MD5 sums of files in /mirror/jigit-test/jigit/files:
 /mirror/jigit-test/jigit/files/md5-list                                    

Unable to recreate image from template file /mirror/jigit-test/jigit/jigdo/wibble.template
/mirror/jigit-test/jigit/jigdo/missing-list contains the list of missing files
Need to download 739 files to complete the image
    0 files missing; all needed files available                              

Image should be 467847168 bytes
Image MD5 should be 90bc9f792371c5c0dae185450d8e9f23
Creating ISO image /mirror/jigit-test/jigit/jigdo/wibble.iso
 100.00%  template data                                               
Output image MD5 is 90bc9f792371c5c0dae185450d8e9f23
Output image length is 467847168
Image created successfully in /mirror/jigit-test/jigit/jigdo/wibble.iso
sledge:~$ 

Debian-cd integration for JTE

A recent diff against CVS HEAD of debian-cd is included in the tarball - see JTE-patch1.gz. Let me know if you have any problems...

还剩下什么要做?

1. Testing! :-) This is where you lot come in! Please play with this some more and let me know if you have any problems, especially with data corruption.
2. More documentation.
3. Push patches upstream. This may take a while...
4. Bzip2 support - I've added this experimentally, but it's disabled until jigdo itself gets it.

I hope people find this useful - at the moment I shudder at the thought of releasing sarge (10+ CDs, netinst, business card, 2 DVDs per arch) without making this kind of change. It'll take a week to generate the release images otherwise...