ddn.compressor.unix

ddn.compressor.unix

Rock-solid implementation of the historic UNIX compress format (".Z") built around a self-contained D port of the classic ncompress 4.2 LZW codec.

The module implements the streaming Compressor / Decompressor interfaces from ddn.api.compressor and registers itself under:

---------------------

  • Enum format: CompressionFormat.unix.
  • String id : "unix".
  • Provider : "ddn".

---------------------

Design notes: ---

  • The actual LZW codec is an internal port of ncompress42.d

(https://github.com/ZILtoid1991/lzwford) and is therefore known to be interoperable with classic UNIX compress.

  • To keep the public API simple and robust, the ddn-facing

compressor and decompressor accumulate all input in memory and invoke the ncompress engine when finish() is called. This satisfies the streaming interface contract while avoiding subtle corner cases in incremental LZW decoding.

  • No external D packages are used; all functionality is contained

in this module plus the standard library.

CREDITS: László Szerémi for the ncompress port to D. This module is based on his work:

https://github.com/ZILtoid1991/lzwford
var primetab UNIX_Z_FLAG_BLOCK_MODE UNIX_Z_MAGIC_0 UNIX_Z_MAGIC_1 UNIX_Z_MAX_BITS UNIX_Z_MIN_BITS
alias Byte cmp_code_int code_int count_int NCmpStreamReader NCmpStreamWriter
enum NCompressError
struct NCompressCtxt PrivState
class UnixZCompressor UnixZDecompressor
fn bitsFromOptions clear_htab clear_tab_prefixof createPrivState makeUnixCompressor makeUnixDecompressor nCompress nDecompress nFreeCompress nInitCompress nInitDecompress throwForNCompressError

Module Initializers 1

shared static this()

Types 11

aliasByte = ubyte

Byte type used internally by the ncompress port.

aliasNCmpStreamReader = int delegate(Byte * bytes, size_t numBytes, void * rwCtxt)

Reader delegate used by the ncompress codec.

The reader must fill at most numBytes bytes into the buffer pointed to by bytes and return:

  • > 0 : number of bytes actually read,
  • 0 : end of stream,
  • -1 : error (mapped to an NCompressError).
aliasNCmpStreamWriter = int delegate(const Byte * bytes, size_t numBytes, void * rwCtxt)

Writer delegate used by the ncompress codec.

The writer must push exactly numBytes bytes from bytes to the output sink and return the number of bytes accepted or -1 on error.

structNCompressCtxt

Context structure used by the ncompress codec.

Instances of this struct are passed to nInitCompress / nInitDecompress and then to nCompress / nDecompress.

Fields
NCmpStreamReader readerReader delegate supplying uncompressed or compressed data.
NCmpStreamWriter writerWriter delegate receiving compressed or decompressed data.
void * rwCtxtOptional opaque pointer forwarded to the reader/writer.
void * privPointer to private state owned by the codec implementation.
enumNCompressError

Error codes returned by the ncompress codec.

NCMP_OK = 0Operation completed successfully.
NCMP_READ_ERRORReader delegate reported an error.
NCMP_WRITE_ERRORWriter delegate reported an error.
NCMP_DATA_ERRORInvalid compressed data format.
NCMP_BITS_ERRORCompressed with too large a `bits` parameter.
NCMP_OTHER_ERROROther internal error.
aliascode_int = c_long

Integer types mirroring the original C implementation.

aliascount_int = c_long
aliascmp_code_int = c_long
structPrivState

Private state used internally by the ncompress codec.

This struct is allocated dynamically and referenced via NCompressCtxt.priv.

Fields
int block_mode
int maxbits
count_int[(1 << 17)] htab
ushort[(1 << 17)] codetab
Byte[(8192 + 64)] inbuf
Byte[(8192 + 2048)] outbuf
c_long bytes_in
c_long bytes_out
classUnixZCompressor : Compressor

UNIX .Z compressor that implements the ddn Compressor interface using the embedded ncompress codec.

The compressor buffers all input until finish() is called, at which point the underlying codec is invoked once to generate a complete .Z stream.

Fields
private CompressionOptions _opts
private OutputSink _sink
private bool _finished
private ubyte[] _input
private ulong _bytesIn
private ulong _bytesOut
private size_t _readPos
Methods
CompressionOptions options() @property constReturn the options used to create this compressor.
void setOutputSink(OutputSink sink)Set the output sink delegate that receives compressed bytes.
void setProgressCallback(ProgressCallback callback)Set an optional progress callback.
void write(const(ubyte)[] data)Buffer more uncompressed data for later compression.
void flush(FlushMode mode = FlushMode.SYNC)Flush pending data. For `.Z` this is equivalent to `sync` and does not force a full frame; real compression happens in `finish()`.
void finish()Finalise the stream by invoking the ncompress codec and emitting the resulting `.Z` data to the output sink.
void reset()Reset the compressor to its initial state.
ulong bytesInTotal() @property constTotal uncompressed bytes seen so far.
ulong bytesOutTotal() @property constTotal compressed bytes emitted so far.
bool setDictionary(const(ubyte)[] dict)UNIX `.Z` format does not support user-specified dictionaries.
bool isFinished() @property constReturns true if finish() has been called and the stream is closed for further writes.
Constructors
this(CompressionOptions opts)Create a new UNIX `.Z` compressor.
Destructors
~thisDestructor ensures cleanup of the input buffer even if finish() was not called.
classUnixZDecompressor : Decompressor

UNIX .Z decompressor implementing the ddn Decompressor interface using the embedded ncompress codec.

Compressed data is buffered until finish() is called, where the codec is invoked once to decode the full stream.

Fields
private DecompressionOptions _opts
private OutputSink _sink
private bool _finished
private ubyte[] _input
private ulong _bytesIn
private ulong _bytesOut
private size_t _readPos
Methods
DecompressionOptions options() @property constReturn the options used to create this decompressor.
void setOutputSink(OutputSink sink)Set the output sink delegate that receives decompressed bytes.
void setProgressCallback(ProgressCallback callback)Set an optional progress callback.
void write(const(ubyte)[] data)Buffer additional compressed data.
void finish()Signal end-of-input and run the decoder.
void reset()Reset the decompressor to its initial state.
ulong bytesInTotal() @property constTotal compressed bytes consumed so far.
ulong bytesOutTotal() @property constTotal decompressed bytes produced so far.
bool setDictionary(const(ubyte)[] dict)UNIX `.Z` does not use external dictionaries; this is a no-op that always returns `true`.
bool isFinished() @property constReturns true if finish() has been called and the stream is closed for further writes.
Constructors
this(DecompressionOptions opts)Create a new UNIX `.Z` decompressor.
Destructors
~thisDestructor ensures cleanup of the input buffer even if finish() was not called.

Functions 12

private fnint bitsFromOptions(CompressionOptions opts)Map `CompressionOptions.numericLevel` to the UNIX `.Z` `bits` parameter used by the ncompress codec.
fnvoid clear_htab(PrivState * ps)Clear the hashtable that backs the compressor dictionary.
fnvoid clear_tab_prefixof(PrivState * ps)Clear the prefix table used by the decompressor.
fnvoid createPrivState(NCompressCtxt * ctxt, int bits)Allocate and initialize the private state for a context if needed.
fnvoid nInitCompress(NCompressCtxt * ctxt, int bits)Initialise a context for compression.
fnvoid nInitDecompress(NCompressCtxt * ctxt)Initialise a context for decompression.
fnvoid nFreeCompress(NCompressCtxt * ctxt)Free all resources associated with a context.
fnNCompressError nCompress(NCompressCtxt * ctxt)Compress the stream defined by `ctxt.reader` into `ctxt.writer`.
fnNCompressError nDecompress(NCompressCtxt * ctxt)Decompress the stream defined by `ctxt.reader` into `ctxt.writer`.
private fnvoid throwForNCompressError(NCompressError err, string op)Map an `NCompressError` to a `CompressionError` exception.
fnCompressor makeUnixCompressor(CompressionOptions opts)Factory function constructing a `UnixZCompressor`.
fnDecompressor makeUnixDecompressor(DecompressionOptions opts)Factory function constructing a `UnixZDecompressor`.

Variables 6

private enumvarUNIX_Z_MIN_BITS = 9

Minimum LZW code size in bits for UNIX .Z (inclusive).

private enumvarUNIX_Z_MAX_BITS = 16

Maximum LZW code size in bits for UNIX .Z (inclusive).

private enumvarUNIX_Z_MAGIC_0 = 0x1F

First magic byte for UNIX .Z streams.

private enumvarUNIX_Z_MAGIC_1 = 0x9D

Second magic byte for UNIX .Z streams.

private enumvarUNIX_Z_FLAG_BLOCK_MODE = 0x80

Bit flag in the header indicating block compression mode.

varint[256] primetab

Prime table used by the ncompress hash function.