In computing, DEFLATE is a lossless data compression algorithm and associated file format that uses a combination of the LZ77 algorithm and Huffman coding. It was originally defined by Phil Katz for version 2 of his PKZIP archiving tool. The file format was later specified in RFC1951.

The original algorithm as designed by Katz was patented as U.S. Patent 5,051,745 and assigned to PKWARE, Inc.^[2][3] As stated in the RFC document, an algorithm producing DEFLATE files is widely thought to be implementable in a manner not covered by patents.^[1] This has led to its widespread use, for example in 

Stream format[edit]

A Deflate stream consists of a series of blocks. Each block is preceded by a 3-bit header:

• First bit: Last-block-in-stream marker:
  • 1: this is the last block in the stream.
  • 0: there are more blocks to process after this one.
• Second and third bits: Encoding method used for this block type:
  • 00: a stored/raw/literal section, between 0 and 65,535 bytes in length.
  • 01: a static Huffman compressed block, using a pre-agreed Huffman tree.
  • 10: a compressed block complete with the Huffman table supplied.
  • 11: reserved, don't use.

The stored block option adds minimal overhead, and is used for data that is incompressible.

Most compressible data will end up being encoded using method 10, the dynamic Huffman encoding, which produces an optimised Huffman tree customised for each block of data individually. Instructions to generate the necessary Huffman tree immediately follow the block header. The static Huffman option is used for short messages, where the fixed saving gained by omitting the tree outweighs the percentage compression loss due to using a non-optimal (thus, not technically Huffman) code.

Compression is achieved through two steps:

• The matching and replacement of duplicate strings with pointers.
• Replacing symbols with new, weighted symbols based on frequency of use.

Duplicate string elimination[edit]

Within compressed blocks, if a duplicate series of bytes is spotted (a repeated string), then a back-reference is inserted, linking to the previous location of that identical string instead. An encoded match to an earlier string consists of an 8-bit length (3–258 bytes) and a 15-bit distance (1–32,768 bytes) to the beginning of the duplicate. Relative back-references can be made across any number of blocks, as long as the distance appears within the last 32 KB of uncompressed data decoded (termed the sliding window).

If the distance is less than the length, the duplicate overlaps itself, indicating repetition. For example, a run of 10 identical bytes can be encoded as one byte, followed by a duplicate of length 9 beginning 1 byte ago.

Bit reduction[edit]

The second compression stage consists of replacing commonly used symbols with shorter representations and less commonly used symbols with longer representations. The method used is Huffman coding which creates an unprefixed tree of non-overlapping intervals, where the length of each sequence is inversely proportional to the probability of that symbol needing to be encoded. The more likely a symbol has to be encoded, the shorter its bit-sequence will be.

A tree is created, containing space for 288 symbols:

• 0–255: represent the literal bytes/symbols 0–255.
• 256: end of block – stop processing if last block, otherwise start processing next block.
• 257–285: combined with extra-bits, a match length of 3–258 bytes.
• 286, 287: not used, reserved and illegal but still part of the tree.

A match length code will always be followed by a distance code. Based on the distance code read, further "extra" bits may be read in order to produce the final distance. The distance tree contains space for 32 symbols:

• 0–3: distances 1–4
• 4–5: distances 5–8, 1 extra bit
• 6–7: distances 9–16, 2 extra bits
• 8–9: distances 17–32, 3 extra bits
• ...
• 26–27: distances 8,193–16,384, 12 extra bits
• 28–29: distances 16,385–32,768, 13 extra bits
• 30–31: not used, reserved and illegal but still part of the tree.

Note that for the match distance symbols 2–29, the number of extra bits can be calculated as {\displaystyle {\frac {n}{2}}-1}.

The code is itself a canonical Huffman code sent by giving the bit length of the code for each symbol. The bit lengths are themselves run-length encoded to produce as compact a representation as possible. As an alternative to including the tree representation, the "static tree" option provides a standard fixed Huffman tree. The compressed size using the static tree can be computed using the same statistics (the number of times each symbol appears) as are used to generate the dynamic tree, so it is easy for a compressor to choose whichever is smaller.

Encoder/compressor[edit]

During the compression stage, it is the encoder that chooses the amount of time spent looking for matching strings. The zlib/gzip reference implementation allows the user to select from a sliding scale of likely resulting compression-level vs. speed of encoding. Options range from 0 (do not attempt compression, just store uncompressed) to 9representing the maximum capability of the reference implementation in zlib/gzip.

Other Deflate encoders have been produced, all of which will also produce a compatible bitstream capable of being decompressed by any existing Deflate decoder. Differing implementations will likely produce variations on the final encoded bit-stream produced. The focus with non-zlib versions of an encoder has normally been to produce a more efficiently compressed and smaller encoded stream.

Deflate64/Enhanced Deflate[edit]

Deflate64, specified by PKWARE, is a proprietary variant of the Deflate procedure. The fundamental mechanisms remain the same. What has changed is the increase in dictionary size from 32 KB to 64 KB, an extension of the distance codes to 16 bits so that they may address a range of 64 KB, and the length code, which is extended to 16 bits so that it may define lengths of three to 65538 bytes.^[4] This leads to Deflate64 having a slightly higher compression ratio and a slightly lower compression time than Deflate.^[5] Several free and/or open source projects support Deflate64, such as 7-Zip,^[6] while others, such as zlib, do not, as a result of the proprietary nature of the procedure^[7] and the very modest performance increase over Deflate.^[8]

Using Deflate in new software[edit]

Implementations of Deflate are freely available in many languages. C programs typically use the zlib library (licensed under the zlib License, which allows use with both free and proprietary software). Programs written using the Borland dialects of Pascal can use paszlib; a C++ library is included as part of 7-Zip/AdvanceCOMP. Java includes support as part of the standard library (in java.util.zip). Microsoft .NET Framework 2.0 base class library supports it in the System.IO.Compression namespace. Programs in Ada can use Zip-Ada (pure) or the ZLib-Ada thick binding to zlib.

Encoder implementations[edit]

• PKZIP: the first implementation, originally done by Phil Katz as part of PKZip.
• zlib/gzip: standard reference implementation used in a huge amount of software, owing to public availability of the source code and a license allowing inclusion into other software.
• Crypto++: contains a public domain implementation in C++ aimed mainly at reducing potential security vulnerabilities. The author, Wei Dai states "This code is less clever, but hopefully more understandable and maintainable [than zlib]".
• 7-Zip/AdvanceCOMP: written by Igor Pavlov in C++, this version is freely licensed and tends to achieve higher compression than zlib at the expense of CPU usage. Has an option to use the DEFLATE64 storage format.
• PuTTY 'sshzlib.c': a standalone implementation, capable of full decode, but static tree only creation, by Simon Tatham. MIT licensed.
• Plan 9 from Bell Labs operating system's libflate implements deflate compression.
• Hyperbac: uses its own proprietary lossless compression library (written in C++ and Assembly) with an option to implement the DEFLATE64 storage format.
• Zopfli: C implementation by Google that achieves highest compression at the expense of CPU usage. ZopfliPNG is a variation of Zopfli for use with PNGs. Apache licensed.

AdvanceCOMP uses the higher compression ratio version of Deflate as implemented by 7-Zip (or optionally Zopfli in recent versions) to enable recompression of gzip, PNG, MNGand ZIP files with the possibility of achieving smaller file sizes than zlib is able to at maximum settings.

Hardware encoders[edit]

• AHA361-PCIX/AHA362-PCIX from Comtech AHA. Comtech produced a PCI-X card (PCI-ID: 193f:0001) capable of compressing streams using Deflate at a rate of up to 3.0 Gbit/s (375 MB/s) for incoming uncompressed data. Accompanying the Linux kernel driver for the AHA361-PCIX is an "ahagzip" utility and customised "mod_deflate_aha" capable of using the hardware compression from Apache. The hardware is based on a Xilinx Virtex FPGA and four custom AHA3601 ASICs. The AHA361/AHA362 boards are limited to only handling static Huffman blocks and require software to be modified to add support — the cards were not able to support the full Deflate specification, meaning they could only reliably decode their own output (a stream that did not contain any dynamic Huffman type 2 blocks).
• StorCompress 300/MX3 from Indra Networks. This is a range of PCI (PCI-ID: 17b4:0011) or PCI-X cards featuring between one and six compression engines with claimed processing speeds of up to 3.6 Gbit/s (450 MB/s). A version of the cards are available with the separate brand WebEnhance specifically designed for web-serving use rather than SAN or backup use; a PCIe revision, the MX4E is also produced.
• AHA363-PCIe/AHA364-PCIe/AHA367-PCIe. In 2008, Comtech started producing two PCIe cards (PCI-ID: 193f:0363/193f:0364) with a new hardware AHA3610 encoder chip. The new chip was designed to be capable of a sustained 2.5 Gbit/s. Using two of these chips, the AHA363-PCIe board can process Deflate at a rate of up to 5.0 Gbit/s (625 MB/s) using the two channels (two compression and two decompression). The AHA364-PCIe variant is an encode-only version of the card designed for out-going load balancers and instead has multiple register sets to allow 32 independent virtual compression channels feeding two physical compression engines. Linux, Microsoft Windows, and OpenSolaris kernel device drivers are available for both of the new cards, along with a modified zlib system library so that dynamically linked applications can automatically use the hardware support without internal modification. The AHA367-PCIe board (PCI-ID: 193f:0367) is similar to the AHA363-PCIe but uses four AHA3610 chips for a sustained compression rate of 10 Gbit/s (1250 MB/s). Unlike the AHA362-PCIX, the decompression engines on the AHA363-PCIe and AHA367-PCIe boards are fully deflate compliant.
• Nitrox and Octeon processors from Cavium, Inc. contain high-speed hardware deflate and inflate engines compatible with both ZLIB and GZIP with some devices able to handle multiple simultaneous data streams.
• Intel Communications Chipset 89xx Series (Cave Creek) for the Intel Xeon E5-2600 and E5-2400 Processor Series (Sandy Bridge-EP/EN) supports hardware compression and decompression using QuickAssist Technology. Depending on the chipset, compression and decompression rates of 5Gbit/s, 10Gbit/s, or 20Gbit/s are available.^[9]

Decoder/decompressor[edit]

Inflate is the decoding process that takes a Deflate bit stream for decompression and correctly produces the original full-size data or file.

Inflate-only implementations[edit]

The normal intent with an alternative Inflate implementation is highly optimised decoding speed, or extremely predictable RAM usage for micro-controller embedded systems.

• Assembly
  • 6502 inflate, written by Piotr Fusik in 6502 assembly language.
  • SAMflate, written by Andrew Collier in Z80 assembly language with optional memory paging support for the SAM Coupé, and made available under the BSD/GPL/LGPL/DFSG licenses.
  • gunzip, written by Laurens Holst in Z80 assembly language for the MSX, licensed under BSD.
  • inflate.asm, a fast and efficient implementation in M68000 machine language, written by Keir Fraser and released into the Public Domain.

• C/C++
  • kunzip by Michael Kohn and unrelated to "KZIP". Comes with C source-code under the GNU LGPL license. Used in the GIMP installer.
  • puff.c (zlib), a small, unencumbered, single-file reference implementation included in the /contrib/puff directory of the zlib distribution.
  • tinf written by Jørgen Ibsen in ANSI C and comes with zlib license. Adds about 2k code.
  • tinfl.c (miniz), Public domain Inflate implementation contained entirely in a single C function.
• PCDEZIP, Bob Flanders and Michael Holmes, published in PC Magazine 1994-01-11.
• inflate.cl by John Foderaro. Self-standing Common Lisp decoder distributed with a GNU LGPL license.
• inflate.s7i/gzip.s7i, a pure-Seed7 implementation of Deflate and gzip decompression, by Thomas Mertes. Made available under the GNU LGPL license.
• pyflatea pure-Python stand-alone Deflate (gzip) and bzip2 decoder by Paul Sladen. Written for research/prototyping and made available under the BSD/GPL/LGPL/DFSGlicenses.
• deflateluaa pure-Lua implementation of Deflate and gzip/zlib decompression, by David Manura.
• inflatea pure-Javascript implementation of Inflate by Chris Dickinson
• pakoJavaScript speed-optimized port of zlib. Contains separate build with inflate only.

Hardware decoders[edit]

• from BitSim. Hardware implementation of Inflate. Part of BitSim's BADGE (Bitsim Accelerated Display Graphics Engine) controller offering for embedded systems.