ddn.api.compressor

A robust and flexible compression API for the D programming language.

Design goals (inspired by popular APIs such as zlib, zstd, Brotli, LZ4, Java's java.util.zip, .NET, and Go's compress):

Format-agnostic core interfaces (Gzip/Deflate/Zlib/Zstd/Brotli/LZ4/BZip2/XZ/Raw/Auto).
Streaming interface with a push model and an output sink (delegate) to avoid forcing a specific IO abstraction.
Buffer-to-buffer convenience helpers for simple use cases.
Configurable options (level/strategy/window/container/checksum/dictionary/threads, etc.).
Extensible registry so providers (implementations) can register themselves without coupling.
Clear error model with portable error codes.

This module defines the API only. Concrete implementations (e.g., GZIP) should live in their own packages/modules and register themselves via the provider registration functions herein.

var DEFAULT_PROGRESS_INTERVAL gProviders

alias CompressorFactoryFn DecompressorFactoryFn OutputSink ProgressCallback

enum ChecksumType CompressionFormat CompressionLevel CompressionStrategy ErrorCode FlushMode

struct CompressionOptions CompressOutputRange CompressRange DecompressionOptions DecompressOutputRange DecompressRange FormatCapabilities ProgressInfo Provider

class CompressionError OutputBufferHolder

iface Compressor Decompressor

fn compressBuffer compressOutputRange compressRange decompressBuffer decompressOutputRange decompressRange detectFormat estimateCompressedSize formatIdNormalize makeCompressor makeCompressorById makeDecompressor makeDecompressorById providersFor registerProvider selectProvider tryParseFormatId

Types 23

enumCompressionFormat : ushort

Supported container/format kinds (extensible).

AUTO_DETECT = 0Detect from data or options when possible

USER = 1Custom format provided by user

RAW = 2Raw DEFLATE stream without container (RFC 1951)

DEFLATE_ZLIB = 3DEFLATE with zlib wrapper format (RFC 1950)

GZIP = 4DEFLATE with gzip wrapper format (RFC 1952)

DEFLATE = 5DEFLATE content (often with raw or zlib container)

BROTLI = 6Brotli compression format (RFC 7932)

ZSTD = 7Zstandard compression format (RFC 8878)

LZ4 = 8LZ4 compression format (https://lz4.github.io/lz4/)

BZIP2 = 9Bzip2 compression format (https://sourceware.org/bzip2/)

XZ = 10XZ compression format with LZMA2 (https://tukaani.org/xz/)

UNIX = 11Historic UNIX `.Z` LZW format (compress)

SNAPPY = 12Snappy compression format (https://github.com/google/snappy)

LZO = 13LZO compression format (https://www.oberhumer.com/opensource/lzo/)

enumCompressionLevel : short

Compression level presets (exact scale is provider/format dependent).

NONE = 0

FASTEST = 1

FAST = 2

DEFAULT = 3

GOOD = 4

BETTER = 5

BEST = 6Provider specific numeric levels are permitted via options.numericLevel.

enumCompressionStrategy : ubyte

Compression strategy hint (zlib-style, optional for providers that support it).

DEFAULT = 0

FILTERED = 1

HUFFMAN_ONLY = 2

RLE = 3

FIXED = 4

enumFlushMode : ubyte

Flush mode for streaming compressors (zlib-style semantics).

NONE = 0

SYNC = 1

FULL = 2

FINISH = 3

enumChecksumType : ubyte

Checksum type to emit/validate at the container level (when applicable).

AUTOMATIC = 0

NONE = 1

ADLER32 = 2

CRC32 = 3

CRC64 = 4

XXH32 = 5

XXH64 = 6

enumErrorCode : int

Error codes used by CompressionError.

OK = 0

INVALID_INPUT

UNSUPPORTED_FORMAT

DATA_ERROR

BUF_ERROR

MEMORY_ERROR

INTERNAL_ERROR

CLOSED

NEED_DICTIONARY

TRUNCATED

CANCELLED

classCompressionError : Exception

Exception class for compression/decompression related errors.

Fields

ErrorCode code

Constructors

this(string msg,  ErrorCode code =  ErrorCode.INTERNAL_ERROR,  string file =  __FILE__,  size_t line =  __LINE__,  Throwable next =  null)

structFormatCapabilities

Capabilities declared by a provider for a given format.

These flags describe the behavioral characteristics of a compression provider, allowing users to make informed decisions about memory usage and streaming behavior.

Fields

bool streaming

bool dictionaries

bool multithreaded

bool independentBlocks

bool buffersAllInput

structCompressionOptions

Options shared by compressors.

Fields

CompressionFormat format

CompressionLevel level

CompressionStrategy strategy

ChecksumType checksum

short numericLevel

int windowBits

int blockSize

int threads

const(ubyte)[] dictionary

size_t progressIntervalBytesMinimum bytes between progress callback invocations (0 = provider default, typically 64 KiB).

structDecompressionOptions

Options shared by decompressors.

Fields

CompressionFormat format

bool autoDetectFormat

const(ubyte)[] dictionary

size_t progressIntervalBytesMinimum bytes between progress callback invocations (0 = provider default, typically 64 KiB).

structProgressInfo

Progress information passed to callbacks during compression/decompression.

This struct provides real-time statistics about the operation's progress, allowing applications to display progress bars, log status, or implement cancellation support.

Fields

ulong bytesIn

ulong bytesOut

float ratio

bool canCancel

aliasProgressCallback = bool delegate(const ProgressInfo info)

Progress callback delegate.

Called periodically during write() and finish() operations to report progress. The callback frequency is controlled by progressIntervalBytes in the options.

Parameters

info Current progress information

Returns

true to continue processing, false to request cancellation.

If the provider supports cancellation and the callback returns false, the operation will throw CompressionError with ErrorCode.CANCELLED.

aliasOutputSink = void delegate(const(ubyte)[] chunk)

Output sink delegate used by streaming interfaces.

interfaceCompressor

Compressor streaming interface (push model).

Methods

CompressionOptions options() @property const;Return the options the compressor was created with (may be adjusted by provider).

void setOutputSink(OutputSink sink)Set the output sink to receive produced compressed bytes. Must be set before write/flush/finish.

void setProgressCallback(ProgressCallback callback)Set an optional progress callback.

void write(const(ubyte)[] data)Feed more uncompressed data into the stream.

void flush(FlushMode mode = FlushMode.SYNC)Flush pending output according to mode. For finish semantics, prefer finish().

void finish()Finalize the stream. After finish() the compressor is closed for further writes.

void reset()Reset the stream to initial state (options retained). Output sink remains unchanged.

ulong bytesInTotal() @property const;Total bytes seen/emitted so far (since last reset()).

ulong bytesOutTotal() @property const;

bool setDictionary(const(ubyte)[] dict)Optional: set or update dictionary. Return true if accepted, false if not supported.

bool isFinished() @property const;Returns true if finish() has been called and the stream is closed for further writes.

interfaceDecompressor

Decompressor streaming interface (push model).

Methods

DecompressionOptions options() @property const;Return the options the decompressor was created with.

void setOutputSink(OutputSink sink)Set the output sink to receive produced decompressed bytes. Must be set before write/finish.

void setProgressCallback(ProgressCallback callback)Set an optional progress callback.

void write(const(ubyte)[] data)Feed more compressed data into the stream.

void finish()Signal end-of-input; allows the decompressor to validate checksums and emit any trailing bytes.

void reset()Reset the stream to initial state (options retained). Output sink remains unchanged.

ulong bytesInTotal() @property const;Total bytes seen/emitted so far (since last reset()).

ulong bytesOutTotal() @property const;

bool setDictionary(const(ubyte)[] dict)Optional: provide a dictionary (if required). Return true if accepted or not needed.

bool isFinished() @property const;Returns true if finish() has been called and the stream is closed for further writes.

aliasCompressorFactoryFn = Compressor function(CompressionOptions)

Functions used by providers to create new stream instances.

aliasDecompressorFactoryFn = Decompressor function(DecompressionOptions)

structProvider

Provider descriptor used for registration.

Each provider is uniquely identified by the combination of vendor and format. The full provider name is constructed as vendor-format (e.g., "ddn-gzip", "ddn-bzip2").

Fields

CompressionFormat format

string vendorVendor identifier for this provider implementation.

int priority

FormatCapabilities caps

CompressorFactoryFn makeCompressor

DecompressorFactoryFn makeDecompressor

Methods

string toString() constReturns the full provider name in the format "vendor-format".

private classOutputBufferHolder

Heap-allocated buffer holder to avoid delegate capture issues with structs.

When a delegate captures a struct member, it captures a pointer to the struct. If the struct is moved or copied, the pointer becomes invalid. This class provides a stable reference for the output buffer that remains valid across struct moves.

Fields

ubyte[] buffer

Methods

void append(const(ubyte)[] chunk)

void clear()

size_t length() @property const

ubyte[] data() @property

structCompressRange(SourceRange) if (isInputRange!SourceRange && is(ElementType!SourceRange : const(ubyte)[]))

An input range that lazily compresses data from a source range.

This wraps the existing Compressor interface to provide pull-based semantics. Data is compressed on-demand as the range is iterated.

The source range must yield elements convertible to const(ubyte)[].

Parameters

SourceRange The type of the source input range providing uncompressed data chunks.

Fields

SourceRange _source

Compressor _compressor

const(ubyte)[] _currentOutput

OutputBufferHolder _bufferHolder

bool _finished

bool _sourceExhausted

Methods

bool empty() @property constInput range primitive: check if range is exhausted.

const(ubyte)[] front() @property constInput range primitive: get current element.

void popFront()Input range primitive: advance to next element.

void _primeOutput()Produce output by feeding source data to compressor.

Constructors

this(SourceRange source, CompressionOptions opts, string providerName = null)Construct a compression range.

this()Disabled default constructor - must be constructed with source and options.

structDecompressRange(SourceRange) if (isInputRange!SourceRange && is(ElementType!SourceRange : const(ubyte)[]))

An input range that lazily decompresses data from a source range.

This wraps the existing Decompressor interface to provide pull-based semantics. Data is decompressed on-demand as the range is iterated.

The source range must yield elements convertible to const(ubyte)[].

Parameters

SourceRange The type of the source input range providing compressed data chunks.

Fields

SourceRange _source

Decompressor _decompressor

const(ubyte)[] _currentOutput

OutputBufferHolder _bufferHolder

bool _finished

bool _sourceExhausted

Methods

bool empty() @property constInput range primitive: check if range is exhausted.

const(ubyte)[] front() @property constInput range primitive: get current element.

void popFront()Input range primitive: advance to next element.

void _primeOutput()Produce output by feeding source data to compressor.

Constructors

this(SourceRange source, DecompressionOptions opts, string providerName = null)Construct a decompression range.

this()Disabled default constructor - must be constructed with source and options.

structCompressOutputRange

An output range that compresses data written to it.

Compressed output is forwarded to a provided sink delegate. This is useful when you want to use range algorithms that write to an output range.

Call finish() when done writing to finalize the compressed stream.

Fields

Compressor _compressor

bool _finished

Methods

void put(const(ubyte)[] data)Output range primitive: accept data.

void finish()Finalize compression. Must be called when done writing.

void flush(FlushMode mode = FlushMode.SYNC)Flush pending data without finishing.

bool isFinished() @property constReturns true if finish() has been called.

ulong bytesInTotal() @property constReturns total uncompressed bytes written so far.

ulong bytesOutTotal() @property constReturns total compressed bytes produced so far.

Constructors

this(OutputSink sink, CompressionOptions opts, string providerName = null)Construct a compression output range.

this()Disabled default constructor - must be constructed with sink and options.

structDecompressOutputRange

An output range that decompresses data written to it.

Decompressed output is forwarded to a provided sink delegate. This is useful when you want to use range algorithms that write to an output range.

Call finish() when done writing to finalize the decompressed stream.

Fields

Decompressor _decompressor

bool _finished

Methods

void put(const(ubyte)[] data)Output range primitive: accept data.

void finish()Finalize decompression. Must be called when done writing.

bool isFinished() @property constReturns true if finish() has been called.

ulong bytesInTotal() @property constReturns total compressed bytes written so far.

ulong bytesOutTotal() @property constReturns total decompressed bytes produced so far.

Constructors

this(OutputSink sink, DecompressionOptions opts, string providerName = null)Construct a decompression output range.

this()Disabled default constructor - must be constructed with sink and options.

Functions 25

fnvoid registerProvider(Provider provider)Register a provider for a given format as specified by `provider.format`.

fnProvider[] providersFor(CompressionFormat fmt)Query available providers for a format.

private fnbool tryParseFormatId(string formatId, out CompressionFormat fmt)Internal helper: map a textual format id to a `CompressionFormat`.

fnProvider[] providersFor(string formatId)Query available providers for a string-identified format. The textual `formatId` is mapped to a `CompressionFormat` and the corresponding enum-based registry entry is returned.

fnCompressionFormat detectFormat(const(ubyte)[] headerBytes)Detect compression format from header bytes.

private fnProvider selectProvider(CompressionFormat fmt, const(char)[] providerName)Pick the best matching provider by format and optional full provider name.

private fnProvider selectProvider(string formatId, const(char)[] providerName)Pick the best matching provider by string formatId and optional full provider name.

private fnstring formatIdNormalize(string s) pure @safeNormalize a formatId for use as a registry key (lowercase).

fnCompressor makeCompressor(CompressionOptions opts, string providerName = null)Create a streaming compressor for the requested options.

fnDecompressor makeDecompressor(DecompressionOptions opts, string providerName = null)Create a streaming decompressor for the requested options.

fnCompressor makeCompressorById(string formatId, CompressionOptions opts, string providerName = null)Create a streaming compressor for a string-identified custom format.

fnDecompressor makeDecompressorById(string formatId, DecompressionOptions opts, string providerName = null)Create a streaming decompressor for a string-identified custom format.

ubyte[] compressBuffer(const(ubyte)[]  input,  CompressionOptions opts =  CompressionOptions.init,  string providerName =  null)

Convenience: compress a whole buffer using GC allocations. Not suitable for untrusted gigantic inputs.

ubyte[] decompressBuffer(const(ubyte)[]  input,  DecompressionOptions opts =  DecompressionOptions.init,  string providerName =  null)

Convenience: decompress a whole buffer using GC allocations.

fnubyte[] compressBuffer(CompressionFormat fmt, const(ubyte)[] input, string providerName = null)Convenience: compress a whole buffer with just a format specification.

fnubyte[] decompressBuffer(CompressionFormat fmt, const(ubyte)[] input, string providerName = null)Convenience: decompress a whole buffer with just a format specification.

fnsize_t estimateCompressedSize(CompressionFormat fmt, size_t inputSize) pure nothrow @nogcEstimate the maximum compressed output size for a given input size and format.

auto compressRange(Range)(Range source,  CompressionOptions opts,  string providerName =  null) if (isInputRange!Range &&  is(ElementType!Range :  const(ubyte)[]))

Convenience function to create a compression range.

auto compressRange(Range)(Range source,  CompressionFormat fmt) if (isInputRange!Range &&  is(ElementType!Range :  const(ubyte)[]))

Convenience overload to create a compression range with just a format.

auto decompressRange(Range)(Range source,  DecompressionOptions opts,  string providerName =  null) if (isInputRange!Range &&  is(ElementType!Range :  const(ubyte)[]))

Convenience function to create a decompression range.

auto decompressRange(Range)(Range source,  CompressionFormat fmt) if (isInputRange!Range &&  is(ElementType!Range :  const(ubyte)[]))

Convenience overload to create a decompression range with just a format.

fnauto compressOutputRange(OutputSink sink, CompressionOptions opts, string providerName = null)Convenience function to create a compression output range.

fnauto compressOutputRange(OutputSink sink, CompressionFormat fmt)Convenience overload to create a compression output range with just a format.

fnauto decompressOutputRange(OutputSink sink, DecompressionOptions opts, string providerName = null)Convenience function to create a decompression output range.

fnauto decompressOutputRange(OutputSink sink, CompressionFormat fmt)Convenience overload to create a decompression output range with just a format.

Variables 2

enumvarDEFAULT_PROGRESS_INTERVAL = 64 * 1024

Default progress callback interval in bytes (64 KiB).

private varProvider[][CompressionFormat] gProviders