ddn.compressor.lzo

LZO compression provider for ddn.api.compressor.

This module provides a basic LZO compression and decompression implementation using a simplified LZO1X-style format. It includes:

Streaming Compressor and Decompressor classes implementing the

ddn.api.compressor interface.

Hash-based match finding with configurable compression levels (0-9).
Simple block format with headers and checksums.
Efficient compression suitable for in-memory data.

LZO (Lempel-Ziv-Oberhumer) is a lossless data compression algorithm that is focused on decompression speed. It is widely used in applications where fast decompression is critical.

Important: This implementation uses a simplified format that is

NOT compatible with the lzop command-line tool or standard LZO file format. It is designed for use within the ddn.compressor API for internal data compression where interoperability with external tools is not required.

Compression levels control the trade-off between speed and compression ratio:

Level 0: Store-only mode (no compression, fastest).
Levels 1-3: Fast compression with smaller hash tables.
Levels 4-6: Default compression with medium hash tables.
Levels 7-9: Better compression with larger hash tables.

The default compression level is 3, providing a good balance between compression ratio and speed.

Copyright

BSD-3-Clause

var LZO_BLOCK_SIZE LZO_FLAG_ADLER32_C LZO_FLAG_ADLER32_D LZO_HASH_MULT LZO_HASH_SENTINEL LZO_LIB_VERSION LZO_MAGIC LZO_MAX_MATCH LZO_MAX_OFFSET LZO_MIN_MATCH LZO_VERSION

struct DecompressResult LzoHashTable

class LzoCompressor LzoDecompressor

fn adler32 compressBlock compressBlockFast countMatch countMatchPtr decompressBlock decompressBlockFast encodeMatch findBestMatch flushLiterals hashBitsForLevel hashTableSize lzoHashIndex makeLzoCompressor makeLzoDecompressor readLE32p

Module Initializers 1

shared static this()

Types 4

private structLzoHashTable

Hash table used by the LZO match finder.

Fields

uint[] table

uint _hashBits

Methods

void allocate(uint hashBits)Allocate or reallocate the table for `hashBits`.

void reset()Reset all entries to the empty-sentinel value.

uint length() @property constNumber of addressable entries in the table.

uint get(uint index) constFetch the stored position for `index`.

void set(uint index, uint pos)Store `pos` at `index`.

uint indexFor(const(ubyte) * p) constCompute the hash index for pointer `p`.

private structDecompressResult

Result structure for fast decompression operations.

Contains information about bytes consumed from input and bytes produced to output, along with success/error status.

Fields

size_t bytesConsumedNumber of bytes consumed from the compressed input.

size_t bytesProducedNumber of bytes produced to the decompressed output.

bool successWhether decompression completed successfully.

ErrorCode errorCodeError code if success is false.

const(char)[] errorMessageError message if success is false.

classLzoCompressor : Compressor

LZO compressor that implements ddn.api.compressor.Compressor.

This class provides streaming compression using the LZO1X algorithm. Internally it uses a pre-allocated malloc'd hash table and a buffer-offset tracking scheme to avoid unnecessary copies on block boundaries.

Fields

private CompressionOptions _opts

private OutputSink _sink

private bool _finished

private ubyte[] _buffer

private size_t _bufferOffset

private ulong _bytesIn

private ulong _bytesOut

private uint * _hashTablePtrPre-allocated hash table (raw pointer, `malloc`'d).

private size_t _hashTableSizeNumber of entries in the pre-allocated hash table.

private uint _hashBitsNumber of hash bits for the pre-allocated hash table.

private ubyte * _outBufPtrPre-allocated output buffer for compression (raw pointer, `malloc`'d).

private size_t _outBufCapCapacity of the pre-allocated output buffer.

private ubyte * _blockBufPtrPre-allocated block buffer for emitBlock (raw pointer, `malloc`'d). This avoids GC allocations from `~=` appends.

private size_t _blockBufCapCapacity of the pre-allocated block buffer.

Methods

CompressionOptions options() @property constReturn the options used to create this compressor.

void setOutputSink(OutputSink sink)Set the output sink delegate that will receive compressed chunks.

void setProgressCallback(ProgressCallback callback)Set an optional progress callback.

ulong bytesInTotal() @property constTotal uncompressed bytes consumed since last reset.

ulong bytesOutTotal() @property constTotal compressed bytes produced since last reset.

void write(const(ubyte)[] data)Feed more uncompressed data.

void flush(FlushMode mode = FlushMode.SYNC)Flush pending output.

void finish()Finalize the stream. No further writes are allowed.

void reset()Reset the compression stream to initial state.

bool setDictionary(const(ubyte)[] dict)Set dictionary (not supported for LZO).

bool isFinished() @property constReturns true if finish() has been called and the stream is closed for further writes.

private int _getLevel()Get compression level from options.

private void emitBlock(const(ubyte)[] uncompressed, const(ubyte)[] compressed)Emit a compressed block with header.

private void _allocateHashTable()Allocate the hash table via `malloc` based on the current level.

private void _freeHashTable()Free the pre-allocated hash table if it has been allocated.

private void _allocateOutBuf()Allocate the output buffer via `malloc`.

private void _freeOutBuf()Free the pre-allocated output buffer if it has been allocated.

private void _allocateBlockBuf()Allocate the block buffer via `malloc`.

private void _freeBlockBuf()Free the pre-allocated block buffer if it has been allocated.

private void _compactBuffer()Compact the internal buffer by discarding already-consumed bytes.

Constructors

this(CompressionOptions opts)Create a compressor with provided options.

Destructors

~thisDestructor – frees the `malloc`'d hash table, output buffer, and block buffer.

classLzoDecompressor : Decompressor

LZO decompressor that implements ddn.api.compressor.Decompressor.

Fields

private DecompressionOptions _opts

private OutputSink _sink

private bool _finished

private ubyte[] _inputBuffer

private ulong _bytesIn

private ulong _bytesOut

private size_t _inputOffsetOffset into `inputBuffer` for the next unread byte.

private ubyte * _decompBufPre-allocated output buffer for decompression (`malloc`'d).

private size_t _decompBufSizeCapacity of `decompBuf` in bytes.

Methods

private void _allocateDecompBuffer()Allocate the initial decompression output buffer.

private void _freeDecompBuffer()Free the decompression output buffer.

private void _ensureDecompBufSize(size_t needed)Ensure the decompression buffer is at least `needed` bytes.

private void _compactInputBuffer()Compact the input buffer by moving unconsumed data to the front.

DecompressionOptions options() @property constReturn the options used to create this decompressor.

void setOutputSink(OutputSink sink)Set the output sink delegate for decompressed data.

void setProgressCallback(ProgressCallback callback)Set an optional progress callback.

ulong bytesInTotal() @property constTotal compressed bytes consumed since last reset.

ulong bytesOutTotal() @property constTotal decompressed bytes produced since last reset.

void write(const(ubyte)[] data)Feed more compressed data.

void finish()Finish the stream.

void reset()Reset the decompression stream.

bool setDictionary(const(ubyte)[] dict)Set dictionary (not supported for LZO).

bool isFinished() @property constReturns true if finish() has been called and the stream is closed for further writes.

private bool tryDecompressBlock()Try to decompress one complete block from the input buffer.

Constructors

this(DecompressionOptions opts)Create a decompressor with provided options.

Destructors

~thisDestructor – frees the `malloc`'d decompression buffer.

Functions 16

private fnuint hashBitsForLevel(int level) pure nothrow @nogcCompute the number of hash bits to use for a given compression level.

private fnuint hashTableSize(uint hashBits) pure nothrow @nogcCompute the hash-table size from the number of hash bits.

private fnuint readLE32p(const(ubyte) * p)Read a 32-bit little-endian word from an unaligned pointer.

private fnuint lzoHashIndex(const(ubyte) * p, uint hashBits)Compute the LZO hash index for the 3-byte sequence at `p`.

private fnsize_t countMatch(const(ubyte)[] src, size_t pos1, size_t pos2, size_t maxLen)Count the number of matching bytes starting at two positions.

private fnauto findBestMatch(const(ubyte)[] src, size_t pos, ref LzoHashTable hashTab, size_t maxOff)Find the best match at the current position.

private fn

size_t countMatchPtr(const(ubyte) *  p1,  const(ubyte) *  p2,
       size_t maxLen,
       const(ubyte) *  end1,  const(ubyte) *  end2)

pure nothrow @nogcCount matching bytes between two raw pointers, with a maximum length.

private fn

size_t compressBlockFast(const(ubyte) *  src,  size_t srcLen,
       ubyte *  outBuf,  size_t outCap,
       int  level,
       uint *  ht =  null,
       size_t htSize =  0,
       uint  htBits =  0)

@trusted @nogc nothrowCompress a single block using the fast pointer-based path.

private fnubyte[] compressBlock(const(ubyte)[] src, int level)Compress a single block using LZO1X algorithm.

private fnvoid flushLiterals(ref ubyte[] dst, const(ubyte)[] lit)Flush literals to output buffer.

private fnvoid encodeMatch(ref ubyte[] dst, size_t off, size_t len)Encode a match to the output buffer.

private fn

DecompressResult decompressBlockFast(
       const(ubyte) *  src,
       size_t srcLen,
       ubyte *  outBuf,
       size_t outBufLen)

@system nothrowFast LZO1X block decompression using pre-allocated buffer and memcpy.

private fnubyte[] decompressBlock(const(ubyte)[] src)Decompress a single LZO1X block.

private fnuint adler32(const(ubyte)[] data, uint init = 1)Compute Adler32 checksum.

fnCompressor makeLzoCompressor(CompressionOptions opts)Factory function that constructs an `LzoCompressor`.

fnDecompressor makeLzoDecompressor(DecompressionOptions opts)Factory function that constructs an `LzoDecompressor`.

Variables 11

private varubyte[3] LZO_MAGIC

LZO magic number for LZO1X format

private enumvarLZO_VERSION = 0x10

LZO file format version

private enumvarLZO_LIB_VERSION = 0x09

LZO library version we're compatible with

private varsize_t LZO_MIN_MATCH

Minimum match length in LZO format

private varsize_t LZO_MAX_MATCH

Maximum match length in LZO format

private varsize_t LZO_MAX_OFFSET

Maximum offset for back-references (64KB)

private varsize_t LZO_BLOCK_SIZE

Block size for LZO files (default: 256KB)

private enumvarLZO_FLAG_ADLER32_D = 0x0001

LZO file header flags

private enumvarLZO_FLAG_ADLER32_C = 0x0002

private enumvarLZO_HASH_MULT = 0x9E37_79B1u

Multiplicative constant used by LZO for hashing 3-byte sequences.

private enumvarLZO_HASH_SENTINEL = 0xFFFF_FFFFu

Sentinel value indicating an empty hash table slot.