ddn.compressor.lzo

LZO compression provider for ddn.api.compressor.

This module provides a basic LZO compression and decompression implementation using a simplified LZO1X-style format. It includes:

Streaming Compressor and Decompressor classes implementing the ddn.api.compressor interface.
Hash-based match finding with configurable compression levels (0-9).
Simple block format with headers and checksums.
Efficient compression suitable for in-memory data.

LZO (Lempel-Ziv-Oberhumer) is a lossless data compression algorithm that is focused on decompression speed. It is widely used in applications where fast decompression is critical.

Important: This implementation uses a simplified format that is NOT compatible with the lzop command-line tool or standard LZO file format. It is designed for use within the ddn.compressor API for internal data compression where interoperability with external tools is not required.

Compression levels control the trade-off between speed and compression ratio:

Level 0: Store-only mode (no compression, fastest).
Levels 1-3: Fast compression with smaller hash tables.
Levels 4-6: Default compression with medium hash tables.
Levels 7-9: Better compression with larger hash tables.

The default compression level is 3, providing a good balance between compression ratio and speed.

Copyright

BSD-3-Clause

var LZO_BLOCK_SIZE LZO_HASH_MULT LZO_HASH_SENTINEL LZO_MAX_MATCH LZO_MAX_OFFSET LZO_MIN_MATCH

struct DecompressResult

class LzoCompressor LzoDecompressor

fn adler32 compressBlock compressBlockFast decompressBlock decompressBlockFast hashBitsForLevel hashTableSize makeLzoCompressor makeLzoDecompressor readLE32 readLE32p

Module Initializers 1

shared static this()

Types 3

private structDecompressResult

Result structure for fast decompression operations.

Contains information about bytes consumed from input and bytes produced to output, along with success/error status.

Fields

size_t bytesConsumedNumber of bytes consumed from the compressed input.

size_t bytesProducedNumber of bytes produced to the decompressed output.

bool successWhether decompression completed successfully.

CompressionErrorCode errorCodeError code if success is false.

const(char)[] errorMessageError message if success is false.

classLzoCompressor : Compressor

LZO compressor that implements ddn.api.compressor.Compressor.

This class provides streaming compression using the LZO1X algorithm. Internally it uses a pre-allocated malloc'd hash table and a buffer-offset tracking scheme to avoid unnecessary copies on block boundaries.

Fields

private CompressionOptions _opts

private OutputSink _sink

private bool _finished

private ubyte[] _buffer

private size_t _bufferOffset

private ulong _bytesIn

private ulong _bytesOut

private uint * _hashTablePtrPre-allocated hash table (raw pointer, `malloc`'d).

private size_t _hashTableSizeNumber of entries in the pre-allocated hash table.

private uint _hashBitsNumber of hash bits for the pre-allocated hash table.

private ubyte * _outBufPtrPre-allocated output buffer for compression (raw pointer, `malloc`'d).

private size_t _outBufCapCapacity of the pre-allocated output buffer.

private ubyte * _blockBufPtrPre-allocated block buffer for emitBlock (raw pointer, `malloc`'d). This avoids GC allocations from `~=` appends.

private size_t _blockBufCapCapacity of the pre-allocated block buffer.

Methods

CompressionOptions options() @property constReturn the options used to create this compressor.

void setOutputSink(OutputSink sink)Set the output sink delegate that will receive compressed chunks.

void setProgressCallback(ProgressCallback callback)Set an optional progress callback.

ulong bytesInTotal() @property constTotal uncompressed bytes consumed since last reset.

ulong bytesOutTotal() @property constTotal compressed bytes produced since last reset.

void write(const(ubyte)[] data)Feed more uncompressed data.

void flush(FlushMode mode = FlushMode.SYNC)Flush pending output.

void finish()Finalize the stream. No further writes are allowed.

void reset()Reset the compression stream to initial state.

bool setDictionary(const(ubyte)[] dict)Set dictionary (not supported for LZO).

bool isFinished() @property constReturns true if finish() has been called and the stream is closed for further writes.

private int _getLevel()Get compression level from options.

private void emitBlock(const(ubyte)[] uncompressed, const(ubyte)[] compressed)Emit a compressed block with header.

private void _allocateHashTable()Allocate the hash table via `malloc` based on the current level.

private void _freeHashTable()Free the pre-allocated hash table if it has been allocated.

private void _allocateOutBuf()Allocate the output buffer via `malloc`.

private void _freeOutBuf()Free the pre-allocated output buffer if it has been allocated.

private void _allocateBlockBuf()Allocate the block buffer via `malloc`.

private void _freeBlockBuf()Free the pre-allocated block buffer if it has been allocated.

private void _compactBuffer()Compact the internal buffer by discarding already-consumed bytes.

Constructors

this(CompressionOptions opts)Create a compressor with provided options.

Destructors

~thisDestructor – frees the `malloc`'d hash table, output buffer, and block buffer.

classLzoDecompressor : Decompressor

LZO decompressor that implements ddn.api.compressor.Decompressor.

Fields

private DecompressionOptions _opts

private OutputSink _sink

private bool _finished

private ubyte[] _inputBuffer

private ulong _bytesIn

private ulong _bytesOut

private size_t _inputOffsetOffset into `inputBuffer` for the next unread byte.

private ubyte * _decompBufPre-allocated output buffer for decompression (`malloc`'d).

private size_t _decompBufSizeCapacity of `decompBuf` in bytes.

Methods

private void _allocateDecompBuffer()Allocate the initial decompression output buffer.

private void _freeDecompBuffer()Free the decompression output buffer.

private void _ensureDecompBufSize(size_t needed)Ensure the decompression buffer is at least `needed` bytes.

private void _compactInputBuffer()Compact the input buffer by moving unconsumed data to the front.

DecompressionOptions options() @property constReturn the options used to create this decompressor.

void setOutputSink(OutputSink sink)Set the output sink delegate for decompressed data.

void setProgressCallback(ProgressCallback callback)Set an optional progress callback.

ulong bytesInTotal() @property constTotal compressed bytes consumed since last reset.

ulong bytesOutTotal() @property constTotal decompressed bytes produced since last reset.

void write(const(ubyte)[] data)Feed more compressed data.

void finish()Finish the stream.

void reset()Reset the decompression stream.

bool setDictionary(const(ubyte)[] dict)Set dictionary (not supported for LZO).

bool isFinished() @property constReturns true if finish() has been called and the stream is closed for further writes.

private bool tryDecompressBlock()Try to decompress one complete block from the input buffer.

Constructors

this(DecompressionOptions opts)Create a decompressor with provided options.

Destructors

~thisDestructor – frees the `malloc`'d decompression buffer.

Functions 11

private fnuint hashBitsForLevel(int level) pure nothrow @nogcCompute the number of hash bits to use for a given compression level.

private fnuint hashTableSize(uint hashBits) pure nothrow @nogcCompute the hash-table size from the number of hash bits.

private fnuint readLE32p(const(ubyte) * p)Read a 32-bit little-endian word from an unaligned pointer.

private fn

size_t compressBlockFast(const(ubyte) *  src,  size_t srcLen,
       ubyte *  outBuf,  size_t outCap,
       int  level,
       uint *  ht =  null,
       size_t htSize =  0,
       uint  htBits =  0)

@trusted @nogc nothrowCompress a single block using the fast pointer-based path.

private fnubyte[] compressBlock(const(ubyte)[] src, int level)Compress a single block using LZO1X algorithm.

private fn

DecompressResult decompressBlockFast(
       const(ubyte) *  src,
       size_t srcLen,
       ubyte *  outBuf,
       size_t outBufLen)

@system nothrowFast LZO1X block decompression using pre-allocated buffer and memcpy.

private fnubyte[] decompressBlock(const(ubyte)[] src)Decompress a single LZO1X block.

private fnuint adler32(const(ubyte)[] data, uint init = 1)Compute Adler32 checksum.

fnCompressor makeLzoCompressor(CompressionOptions opts)Factory function that constructs an `LzoCompressor`.

fnDecompressor makeLzoDecompressor(DecompressionOptions opts)Factory function that constructs an `LzoDecompressor`.

private fnuint readLE32(const(ubyte)[] buf, size_t off) pure @safe

Variables 6

private varsize_t LZO_MIN_MATCH = 3

Minimum match length in LZO format

private varsize_t LZO_MAX_MATCH = 264

Maximum match length in LZO format

private varsize_t LZO_MAX_OFFSET = 65536

Maximum offset for back-references (64KB)

private varsize_t LZO_BLOCK_SIZE = 256 * 1024

Block size for LZO files (default: 256KB)

private enumvarLZO_HASH_MULT = 0x9E37_79B1u

Multiplicative constant used by LZO for hashing 3-byte sequences.

private enumvarLZO_HASH_SENTINEL = 0xFFFF_FFFFu

Sentinel value indicating an empty hash table slot.