ddn.compressor.snappy

ddn.compressor.snappy

Snappy compression provider for ddn.api.compressor.

This module provides a complete Snappy compression and decompression implementation conforming to the Snappy raw format specification. It includes:

  • Streaming Compressor and Decompressor classes implementing the

ddn.api.compressor interface.

  • Hash-based match finding for LZ77-style compression.
  • Full support for all Snappy element types (literals, 1/2/4-byte copies).
  • Interoperability with the standard Snappy library format.

Snappy is designed for speed rather than maximum compression ratio. It achieves reasonable compression at very high speeds, making it suitable for real-time applications, database storage, and RPC systems.

Format specification: https://github.com/google/snappy/blob/main/format_description.txt

Copyright

BSD-3-Clause
var SNAPPY_BLOCK_SIZE SNAPPY_HASH_BITS SNAPPY_HASH_SIZE SNAPPY_MAX_LEN_1 SNAPPY_MAX_LEN_2 SNAPPY_MAX_OFFSET_1 SNAPPY_MAX_OFFSET_2 SNAPPY_MIN_MATCH
class SnappyCompressor SnappyDecompressor
fn compressBlock compressBlockFast decodeVarint decompressBlock emitCopy emitLiteral encodeCopy encodeLiteral encodeVarint findMatch makeSnappyCompressor makeSnappyDecompressor snappyHash snappyHashFast

Module Initializers 1

shared static this()

Types 2

classSnappyCompressor : Compressor

Snappy compressor implementing the Compressor interface.

Buffers all input until finish() is called, then compresses and emits the complete Snappy stream.

Fields
private CompressionOptions _opts
private OutputSink _sink
private ubyte[] _buffer
private ulong _bytesIn
private ulong _bytesOut
private bool _finished
Methods
CompressionOptions options() @property constReturns the compression options.
void setOutputSink(OutputSink sink)Set the output sink for compressed data.
void setProgressCallback(ProgressCallback callback)Set an optional progress callback.
ulong bytesInTotal() @property constReturns total bytes written to the compressor.
ulong bytesOutTotal() @property constReturns total bytes emitted by the compressor.
void write(const(ubyte)[] data)Write data to be compressed.
void flush(FlushMode mode = FlushMode.SYNC)Flush pending data.
void finish()Finish compression and emit all output.
void reset()Reset the compressor for reuse.
bool setDictionary(const(ubyte)[] dict)Set a compression dictionary (not supported by Snappy).
bool isFinished() @property constReturns true if finish() has been called and the stream is closed for further writes.
Constructors
this(CompressionOptions opts)Construct a new Snappy compressor.
classSnappyDecompressor : Decompressor

Snappy decompressor implementing the Decompressor interface.

Buffers all input until finish() is called, then decompresses and emits the complete uncompressed data.

Fields
private DecompressionOptions _opts
private OutputSink _sink
private ubyte[] _buffer
private ulong _bytesIn
private ulong _bytesOut
private bool _finished
Methods
DecompressionOptions options() @property constReturns the decompression options.
void setOutputSink(OutputSink sink)Set the output sink for decompressed data.
void setProgressCallback(ProgressCallback callback)Set an optional progress callback.
ulong bytesInTotal() @property constReturns total bytes written to the decompressor.
ulong bytesOutTotal() @property constReturns total bytes emitted by the decompressor.
void write(const(ubyte)[] data)Write compressed data to be decompressed.
void finish()Finish decompression and emit all output.
void reset()Reset the decompressor for reuse.
bool setDictionary(const(ubyte)[] dict)Set a decompression dictionary (not supported by Snappy).
bool isFinished() @property constReturns true if finish() has been called and the stream is closed for further writes.
Constructors
this(DecompressionOptions opts)Construct a new Snappy decompressor.

Functions 14

private fnsize_t encodeVarint(uint value, ubyte[] output)Encode a value as a little-endian varint.
private fnsize_t decodeVarint(const(ubyte)[] input, out uint value)Decode a little-endian varint from input.
private fnsize_t snappyHash(const(ubyte)[] data)Compute hash for a 4-byte sequence for match finding.
private fnuint snappyHashFast(const(ubyte) * p) pure nothrow @nogcFast inline hash using pointer cast for 32-bit LE read.
private fnauto findMatch(const(ubyte)[] data, size_t pos, ref size_t[SNAPPY_HASH_SIZE] hashTable)Find a match at the current position using the hash table.
private fnsize_t encodeLiteral(const(ubyte)[] literal, ubyte[] output)Encode a literal element.
private fnsize_t encodeCopy(size_t offset, size_t length, ubyte[] output)Encode a copy (backreference) element.
private fnvoid emitLiteral(ref ubyte * op, const(ubyte) * src, size_t len) pure nothrow @nogcEmit a literal element using raw pointers.
private fnvoid emitCopy(ref ubyte * op, size_t offset, size_t length) pure nothrow @nogcEmit a copy (backreference) element using raw pointers.
private fnsize_t compressBlockFast(const(ubyte)[] src, ubyte[] outBuf) @trusted @nogc nothrowCompress a block of data using Snappy algorithm.
private fnsize_t compressBlock(const(ubyte)[] input, ubyte[] output)Compress a block of data using Snappy format.
private fnsize_t decompressBlock(const(ubyte)[] input, size_t expectedLen, ubyte[] output)Decompress a Snappy-compressed block.
fnCompressor makeSnappyCompressor(CompressionOptions opts)Create a Snappy compressor.
fnDecompressor makeSnappyDecompressor(DecompressionOptions opts)Create a Snappy decompressor.

Variables 8

private enumvarSNAPPY_MIN_MATCH = 4

Minimum match length for Snappy (4 bytes).

private enumvarSNAPPY_MAX_OFFSET_1 = 2047

Maximum offset for 1-byte copy (11 bits = 2047).

private enumvarSNAPPY_MAX_OFFSET_2 = 65535

Maximum offset for 2-byte copy (16 bits = 65535).

private enumvarSNAPPY_MAX_LEN_1 = 11

Maximum length for 1-byte copy encoding.

private enumvarSNAPPY_MAX_LEN_2 = 64

Maximum length for 2-byte and 4-byte copy encoding.

private enumvarSNAPPY_HASH_BITS = 14

Hash table size (64K entries for good match finding).

private enumvarSNAPPY_HASH_SIZE = 1 << SNAPPY_HASH_BITS
private enumvarSNAPPY_BLOCK_SIZE = 32 * 1024

Block size for compression (32KB, matching reference implementation).