|
jpayne@68
|
1
|
|
jpayne@68
|
2 History of LZMA Utils and XZ Utils
|
|
jpayne@68
|
3 ==================================
|
|
jpayne@68
|
4
|
|
jpayne@68
|
5 Tukaani distribution
|
|
jpayne@68
|
6
|
|
jpayne@68
|
7 In 2005, there was a small group working on the Tukaani distribution,
|
|
jpayne@68
|
8 which was a Slackware fork. One of the project's goals was to fit the
|
|
jpayne@68
|
9 distro on a single 700 MiB ISO-9660 image. Using LZMA instead of gzip
|
|
jpayne@68
|
10 helped a lot. Roughly speaking, one could fit data that took 1000 MiB
|
|
jpayne@68
|
11 in gzipped form into 700 MiB with LZMA. Naturally, the compression
|
|
jpayne@68
|
12 ratio varied across packages, but this was what we got on average.
|
|
jpayne@68
|
13
|
|
jpayne@68
|
14 Slackware packages have traditionally had .tgz as the filename suffix,
|
|
jpayne@68
|
15 which is an abbreviation of .tar.gz. A logical naming for LZMA
|
|
jpayne@68
|
16 compressed packages was .tlz, being an abbreviation of .tar.lzma.
|
|
jpayne@68
|
17
|
|
jpayne@68
|
18 At the end of the year 2007, there was no distribution under the
|
|
jpayne@68
|
19 Tukaani project anymore, but development of LZMA Utils was kept going.
|
|
jpayne@68
|
20 Still, there were .tlz packages around, because at least Vector Linux
|
|
jpayne@68
|
21 (a Slackware based distribution) used LZMA for its packages.
|
|
jpayne@68
|
22
|
|
jpayne@68
|
23 First versions of the modified pkgtools used the LZMA_Alone tool from
|
|
jpayne@68
|
24 Igor Pavlov's LZMA SDK as is. It was fine, because users wouldn't need
|
|
jpayne@68
|
25 to interact with LZMA_Alone directly. But people soon wanted to use
|
|
jpayne@68
|
26 LZMA for other files too, and the interface of LZMA_Alone wasn't
|
|
jpayne@68
|
27 comfortable for those used to gzip and bzip2.
|
|
jpayne@68
|
28
|
|
jpayne@68
|
29
|
|
jpayne@68
|
30 First steps of LZMA Utils
|
|
jpayne@68
|
31
|
|
jpayne@68
|
32 The first version of LZMA Utils (4.22.0) included a shell script called
|
|
jpayne@68
|
33 lzmash. It was a wrapper that had a gzip-like command-line interface. It
|
|
jpayne@68
|
34 used the LZMA_Alone tool from LZMA SDK to do all the real work. zgrep,
|
|
jpayne@68
|
35 zdiff, and related scripts from gzip were adapted to work with LZMA and
|
|
jpayne@68
|
36 were part of the first LZMA Utils release too.
|
|
jpayne@68
|
37
|
|
jpayne@68
|
38 LZMA Utils 4.22.0 included also lzmadec, which was a small (less than
|
|
jpayne@68
|
39 10 KiB) decoder-only command-line tool. It was written on top of the
|
|
jpayne@68
|
40 decoder-only C code found from the LZMA SDK. lzmadec was convenient in
|
|
jpayne@68
|
41 situations where LZMA_Alone (a few hundred KiB) would be too big.
|
|
jpayne@68
|
42
|
|
jpayne@68
|
43 lzmash and lzmadec were written by Lasse Collin.
|
|
jpayne@68
|
44
|
|
jpayne@68
|
45
|
|
jpayne@68
|
46 Second generation
|
|
jpayne@68
|
47
|
|
jpayne@68
|
48 The lzmash script was an ugly and not very secure hack. The last
|
|
jpayne@68
|
49 version of LZMA Utils to use lzmash was 4.27.1.
|
|
jpayne@68
|
50
|
|
jpayne@68
|
51 LZMA Utils 4.32.0beta1 introduced a new lzma command-line tool written
|
|
jpayne@68
|
52 by Ville Koskinen. It was written in C++, and used the encoder and
|
|
jpayne@68
|
53 decoder from C++ LZMA SDK with some little modifications. This tool
|
|
jpayne@68
|
54 replaced both the lzmash script and the LZMA_Alone command-line tool
|
|
jpayne@68
|
55 in LZMA Utils.
|
|
jpayne@68
|
56
|
|
jpayne@68
|
57 Introducing this new tool caused some temporary incompatibilities,
|
|
jpayne@68
|
58 because the LZMA_Alone executable was simply named lzma like the new
|
|
jpayne@68
|
59 command-line tool, but they had a completely different command-line
|
|
jpayne@68
|
60 interface. The file format was still the same.
|
|
jpayne@68
|
61
|
|
jpayne@68
|
62 Lasse wrote liblzmadec, which was a small decoder-only library based
|
|
jpayne@68
|
63 on the C code found from LZMA SDK. liblzmadec had an API similar to
|
|
jpayne@68
|
64 zlib, although there were some significant differences, which made it
|
|
jpayne@68
|
65 non-trivial to use it in some applications designed for zlib and
|
|
jpayne@68
|
66 libbzip2.
|
|
jpayne@68
|
67
|
|
jpayne@68
|
68 The lzmadec command-line tool was converted to use liblzmadec.
|
|
jpayne@68
|
69
|
|
jpayne@68
|
70 Alexandre Sauvé helped converting the build system to use GNU
|
|
jpayne@68
|
71 Autotools. This made it easier to test for certain less portable
|
|
jpayne@68
|
72 features needed by the new command-line tool.
|
|
jpayne@68
|
73
|
|
jpayne@68
|
74 Since the new command-line tool never got completely finished (for
|
|
jpayne@68
|
75 example, it didn't support the LZMA_OPT environment variable), the
|
|
jpayne@68
|
76 intent was to not call 4.32.x stable. Similarly, liblzmadec wasn't
|
|
jpayne@68
|
77 polished, but appeared to work well enough, so some people started
|
|
jpayne@68
|
78 using it too.
|
|
jpayne@68
|
79
|
|
jpayne@68
|
80 Because the development of the third generation of LZMA Utils was
|
|
jpayne@68
|
81 delayed considerably (3-4 years), the 4.32.x branch had to be kept
|
|
jpayne@68
|
82 maintained. It got some bug fixes now and then, and finally it was
|
|
jpayne@68
|
83 decided to call it stable, although most of the missing features were
|
|
jpayne@68
|
84 never added.
|
|
jpayne@68
|
85
|
|
jpayne@68
|
86
|
|
jpayne@68
|
87 File format problems
|
|
jpayne@68
|
88
|
|
jpayne@68
|
89 The file format used by LZMA_Alone was primitive. It was designed with
|
|
jpayne@68
|
90 embedded systems in mind, and thus provided only a minimal set of
|
|
jpayne@68
|
91 features. The two biggest problems for non-embedded use were the lack
|
|
jpayne@68
|
92 of magic bytes and an integrity check.
|
|
jpayne@68
|
93
|
|
jpayne@68
|
94 Igor and Lasse started developing a new file format with some help
|
|
jpayne@68
|
95 from Ville Koskinen. Also Mark Adler, Mikko Pouru, H. Peter Anvin,
|
|
jpayne@68
|
96 and Lars Wirzenius helped with some minor things at some point of the
|
|
jpayne@68
|
97 development. Designing the new format took quite a long time (actually,
|
|
jpayne@68
|
98 too long a time would be a more appropriate expression). It was mostly
|
|
jpayne@68
|
99 because Lasse was quite slow at getting things done due to personal
|
|
jpayne@68
|
100 reasons.
|
|
jpayne@68
|
101
|
|
jpayne@68
|
102 Originally the new format was supposed to use the same .lzma suffix
|
|
jpayne@68
|
103 that was already used by the old file format. Switching to the new
|
|
jpayne@68
|
104 format wouldn't have caused much trouble when the old format wasn't
|
|
jpayne@68
|
105 used by many people. But since the development of the new format took
|
|
jpayne@68
|
106 such a long time, the old format got quite popular, and it was decided
|
|
jpayne@68
|
107 that the new file format must use a different suffix.
|
|
jpayne@68
|
108
|
|
jpayne@68
|
109 It was decided to use .xz as the suffix of the new file format. The
|
|
jpayne@68
|
110 first stable .xz file format specification was finally released in
|
|
jpayne@68
|
111 December 2008. In addition to fixing the most obvious problems of
|
|
jpayne@68
|
112 the old .lzma format, the .xz format added some new features like
|
|
jpayne@68
|
113 support for multiple filters (compression algorithms), filter chaining
|
|
jpayne@68
|
114 (like piping on the command line), and limited random-access reading.
|
|
jpayne@68
|
115
|
|
jpayne@68
|
116 Currently the primary compression algorithm used in .xz is LZMA2.
|
|
jpayne@68
|
117 It is an extension on top of the original LZMA to fix some practical
|
|
jpayne@68
|
118 problems: LZMA2 adds support for flushing the encoder, uncompressed
|
|
jpayne@68
|
119 chunks, eases stateful decoder implementations, and improves support
|
|
jpayne@68
|
120 for multithreading. Since LZMA2 is better than the original LZMA, the
|
|
jpayne@68
|
121 original LZMA is not supported in .xz.
|
|
jpayne@68
|
122
|
|
jpayne@68
|
123
|
|
jpayne@68
|
124 Transition to XZ Utils
|
|
jpayne@68
|
125
|
|
jpayne@68
|
126 The early versions of XZ Utils were called LZMA Utils. The first
|
|
jpayne@68
|
127 releases were 4.42.0alphas. They dropped the rest of the C++ LZMA SDK.
|
|
jpayne@68
|
128 The code was still directly based on LZMA SDK but ported to C and
|
|
jpayne@68
|
129 converted from a callback API to a stateful API. Later, Igor Pavlov
|
|
jpayne@68
|
130 made a C version of the LZMA encoder too; these ports from C++ to C
|
|
jpayne@68
|
131 were independent in LZMA SDK and LZMA Utils.
|
|
jpayne@68
|
132
|
|
jpayne@68
|
133 The core of the new LZMA Utils was liblzma, a compression library with
|
|
jpayne@68
|
134 a zlib-like API. liblzma supported both the old and new file format.
|
|
jpayne@68
|
135 The gzip-like lzma command-line tool was rewritten to use liblzma.
|
|
jpayne@68
|
136
|
|
jpayne@68
|
137 The new LZMA Utils code base was renamed to XZ Utils when the name
|
|
jpayne@68
|
138 of the new file format had been decided. The liblzma compression
|
|
jpayne@68
|
139 library retained its name though, because changing it would have
|
|
jpayne@68
|
140 caused unnecessary breakage in applications already using the early
|
|
jpayne@68
|
141 liblzma snapshots.
|
|
jpayne@68
|
142
|
|
jpayne@68
|
143 The xz command-line tool can emulate the gzip-like lzma tool by
|
|
jpayne@68
|
144 creating appropriate symlinks (e.g. lzma -> xz). Thus, practically
|
|
jpayne@68
|
145 all scripts using the lzma tool from LZMA Utils will work as is with
|
|
jpayne@68
|
146 XZ Utils (and will keep using the old .lzma format). Still, the .lzma
|
|
jpayne@68
|
147 format is more or less deprecated. XZ Utils will keep supporting it,
|
|
jpayne@68
|
148 but new applications should use the .xz format, and migrating old
|
|
jpayne@68
|
149 applications to .xz is often a good idea too.
|
|
jpayne@68
|
150
|
