ddn.api.compressor

A robust and flexible compression API for the D programming language.

Design goals (inspired by popular APIs such as zlib, zstd, Brotli, LZ4, Java's java.util.zip, .NET, and Go's compress):

Format-agnostic core interfaces (Gzip/Deflate/Zlib/Zstd/Brotli/LZ4/BZip2/XZ/Raw/Auto).
Streaming interface with a push model and an output sink (delegate) to avoid forcing a specific IO abstraction.
Buffer-to-buffer convenience helpers for simple use cases.
Configurable options (level/strategy/window/container/checksum/dictionary/threads, etc.).
Extensible registry so providers (implementations) can register themselves without coupling.
Clear error model with portable error codes.

This module defines the API only. Concrete implementations (e.g., GZIP) should live in their own packages/modules and register themselves via the provider registration functions herein.

var DEFAULT_PROGRESS_INTERVAL gProviders gProvidersLock TEST_IDENTITY_PROVIDER_NAME TEST_IDENTITY_PROVIDER_VENDOR

alias CompressorFactoryFn DecompressorFactoryFn OutputSink ProgressCallback

enum ChecksumType CompressionErrorCode CompressionFormat CompressionLevel CompressionStrategy FlushMode

struct CompressionOptions CompressionProvider CompressOutputRange CompressRange DecompressionOptions DecompressOutputRange DecompressRange FormatCapabilities ProgressInfo

class CompressionError IdentityCompressor IdentityDecompressor OutputBufferHolder

iface Compressor Decompressor

fn compressBuffer compressOutputRange compressRange decompressBuffer decompressOutputRange decompressRange detectFormat estimateCompressedSize formatIdNormalize makeCompressor makeCompressorById makeDecompressor makeDecompressorById providersFor registerIdentityCodec registerProvider selectProvider tryParseFormatId

Module Initializers 1

shared static this()

Types 25

enumCompressionFormat : ushort

Supported container/format kinds (extensible).

Each member denotes exactly one wire format; combinations are expressed by picking the matching container member.

AUTO_DETECT = 0Detect from data or options when possible

USER = 1Custom format provided by user

RAW = 2Raw DEFLATE stream without container (RFC 1951)

DEFLATE_ZLIB = 3DEFLATE data inside a zlib container (RFC 1950)

GZIP = 4DEFLATE data inside a gzip container (RFC 1952)

DEFLATE = 5Bare DEFLATE payload with unspecified container; prefer RAW or DEFLATE_ZLIB

BROTLI = 6Brotli compression format (RFC 7932)

ZSTD = 7Zstandard compression format (RFC 8878)

LZ4 = 8LZ4 compression format (https://lz4.github.io/lz4/)

BZIP2 = 9Bzip2 compression format (https://sourceware.org/bzip2/)

XZ = 10XZ compression format with LZMA2 (https://tukaani.org/xz/)

UNIX = 11Historic UNIX `.Z` LZW format (compress)

SNAPPY = 12Snappy compression format (https://github.com/google/snappy)

LZO = 13LZO compression format (https://www.oberhumer.com/opensource/lzo/)

enumCompressionLevel : short

Compression level presets (exact scale is provider/format dependent).

Provider-specific numeric levels beyond these presets are permitted via CompressionOptions.numericLevel.

NONE = 0Store only (no compression)

FASTEST = 1Fastest speed, lowest ratio

FAST = 2Fast

DEFAULT = 3Provider default

GOOD = 4Better ratio

BETTER = 5Even better ratio

BEST = 6Best ratio the provider offers

enumCompressionStrategy : ubyte

Compression strategy hint (zlib-style, optional for providers that support it).

DEFAULT = 0CompressionProvider default strategy

FILTERED = 1Filtered data (e.g. deltas)

HUFFMAN_ONLY = 2Huffman coding only

RLE = 3Run-length encoding hint

FIXED = 4Fixed Huffman codes

enumFlushMode : ubyte

Flush mode for streaming compressors (zlib-style semantics).

NONE = 0No special action

SYNC = 1Make output available to consumer without ending the frame/stream

FULL = 2Flush block/frame if supported

FINISH = 3Finish the stream (finalize checksums/frames)

enumChecksumType : ubyte

Checksum type to emit/validate at the container level (when applicable).

AUTOMATIC = 0CompressionProvider chooses sensible default for selected format

NONE = 1No container checksum

ADLER32 = 2Adler-32

CRC32 = 3CRC-32

CRC64 = 4CRC-64

XXH32 = 5xxHash 32-bit

XXH64 = 6xxHash 64-bit

enumCompressionErrorCode : int

Error codes used by CompressionError.

OK = 0No error

INVALID_INPUTInvalid arguments or malformed request

UNSUPPORTED_FORMATFormat or feature unsupported by the selected provider

DATA_ERRORCorrupt input or checksum mismatch

BUF_ERRORInsufficient buffer or unexpected EOF in pull models

MEMORY_ERRORAllocation or output-limit failure

INTERNAL_ERRORCompressionProvider-internal failure

CLOSEDOperation on a finished/closed stream

NEED_DICTIONARYDecompressor requires a dictionary

TRUNCATEDIncomplete input stream

CANCELLEDOperation cancelled by progress callback

classCompressionError : Exception

Exception class for compression/decompression related errors.

Fields

CompressionErrorCode codeMachine-readable error code carried by this exception.

Constructors

this(string msg,  CompressionErrorCode code =  CompressionErrorCode.INTERNAL_ERROR,
          string file =  __FILE__,  size_t line =  __LINE__,
          Throwable next =  null)

Construct the error.

structFormatCapabilities

Capabilities declared by a provider for a given format.

These flags describe the behavioral characteristics of a compression provider, allowing users to make informed decisions about memory usage and streaming behavior.

Fields

bool streaming

bool dictionaries

bool multithreaded

bool independentBlocks

bool buffersAllInput

structCompressionOptions

Options shared by compressors.

Fields

CompressionFormat format

CompressionLevel level

CompressionStrategy strategy

ChecksumType checksum

short numericLevelCompressionProvider-specific numeric level (e.g., zstd 1..22). If < 0, ignored.

int windowBitsWindow and block tuning (provider dependent; ignored if unsupported).

int blockSize

int threadsStreaming & threading hints.

const(ubyte)[] dictionaryOptional dictionary; if empty, not used.

size_t progressIntervalBytesMinimum bytes between progress callback invocations (0 = provider default, typically 64 KiB).

structDecompressionOptions

Options shared by decompressors.

Fields

CompressionFormat format

bool autoDetectFormatIf true, decompressor will attempt to auto-detect the format from input.

const(ubyte)[] dictionaryOptional dictionary to use (if required/requested by stream).

size_t progressIntervalBytesMinimum bytes between progress callback invocations (0 = provider default, typically 64 KiB).

size_t maxOutputSizeMaximum total decompressed bytes; `decompressBuffer` aborts with MEMORY_ERROR beyond this limit, guarding against decompression bombs (0 = unlimited).

structProgressInfo

Progress information passed to callbacks during compression/decompression.

This struct provides real-time statistics about the operation's progress, allowing applications to display progress bars, log status, or implement cancellation support.

Fields

ulong bytesInTotal bytes consumed so far (uncompressed for compressor, compressed for decompressor)

ulong bytesOutTotal bytes produced so far (compressed for compressor, decompressed for decompressor)

float ratioCurrent ratio, always `bytesOut / bytesIn` (compression ratio for compressors, expansion factor for decompressors)

bool canCancel

aliasProgressCallback = bool delegate(const ProgressInfo info)

Progress callback delegate.

Called periodically during write() and finish() operations to report progress. The callback frequency is controlled by progressIntervalBytes in the options.

Parameters

info Current progress information

Returns

true to continue processing, false to request cancellation.

If the provider supports cancellation and the callback returns false, the operation will throw CompressionError with CompressionErrorCode.CANCELLED.

aliasOutputSink = void delegate(const(ubyte)[] chunk)

Output sink delegate used by streaming interfaces.

interfaceCompressor

Compressor streaming interface (push model).

Methods

CompressionOptions options() @property const;Return the options the compressor was created with (may be adjusted by provider).

void setOutputSink(OutputSink sink)Set the output sink to receive produced compressed bytes. Must be set before write/flush/finish.

void setProgressCallback(ProgressCallback callback)Set an optional progress callback.

void write(const(ubyte)[] data)Feed more uncompressed data into the stream.

void flush(FlushMode mode = FlushMode.SYNC)Flush pending output according to mode. For finish semantics, prefer finish().

void finish()Finalize the stream. After finish() the compressor is closed for further writes.

void reset()Reset the stream to initial state (options retained). Output sink remains unchanged.

ulong bytesInTotal() @property const;Total bytes seen/emitted so far (since last reset()).

ulong bytesOutTotal() @property const;

bool setDictionary(const(ubyte)[] dict)Optional: set or update dictionary. Return true if accepted, false if not supported.

bool isFinished() @property const;Returns true if finish() has been called and the stream is closed for further writes.

interfaceDecompressor

Decompressor streaming interface (push model).

Methods

DecompressionOptions options() @property const;Return the options the decompressor was created with.

void setOutputSink(OutputSink sink)Set the output sink to receive produced decompressed bytes. Must be set before write/finish.

void setProgressCallback(ProgressCallback callback)Set an optional progress callback.

void write(const(ubyte)[] data)Feed more compressed data into the stream.

void finish()Signal end-of-input; allows the decompressor to validate checksums and emit any trailing bytes.

void reset()Reset the stream to initial state (options retained). Output sink remains unchanged.

ulong bytesInTotal() @property const;Total bytes seen/emitted so far (since last reset()).

ulong bytesOutTotal() @property const;

bool setDictionary(const(ubyte)[] dict)Optional: provide a dictionary (if required). Return true if accepted or not needed.

bool isFinished() @property const;Returns true if finish() has been called and the stream is closed for further writes.

aliasCompressorFactoryFn = Compressor function(CompressionOptions)

Functions used by providers to create new stream instances.

aliasDecompressorFactoryFn = Decompressor function(DecompressionOptions)

Factory function type for creating decompressors.

structCompressionProvider

CompressionProvider descriptor used for registration.

Each provider is uniquely identified by the combination of vendor and format. The full provider name is constructed as vendor-format (e.g., "ddn-gzip", "ddn-bzip2").

Fields

CompressionFormat format

string vendorVendor identifier for this provider implementation.

int priority

FormatCapabilities caps

CompressorFactoryFn makeCompressor

DecompressorFactoryFn makeDecompressor

Methods

string toString() constReturns the full provider name in the format "vendor-format".

private classOutputBufferHolder

Heap-allocated buffer holder to avoid delegate capture issues with structs.

When a delegate captures a struct member, it captures a pointer to the struct. If the struct is moved or copied, the pointer becomes invalid. This class provides a stable reference for the output buffer that remains valid across struct moves.

Fields

ubyte[] bufferBytes accumulated so far.

Methods

void append(const(ubyte)[] chunk)Append one output chunk.

void clear()Drop all accumulated bytes.

size_t length() @property constNumber of accumulated bytes.

ubyte[] data() @propertyThe accumulated bytes.

structCompressRange(SourceRange) if (isInputRange!SourceRange && is(ElementType!SourceRange : const(ubyte)[]))

An input range that lazily compresses data from a source range.

This wraps the existing Compressor interface to provide pull-based semantics. Data is compressed on-demand as the range is iterated.

The source range must yield elements convertible to const(ubyte)[].

Parameters

SourceRange The type of the source input range providing uncompressed data chunks.

Fields

private SourceRange _source

private Compressor _compressor

private const(ubyte)[] _currentOutput

private OutputBufferHolder _bufferHolder

private bool _finished

private bool _sourceExhausted

Methods

bool empty() @property constInput range primitive: check if range is exhausted.

const(ubyte)[] front() @property constInput range primitive: get current element.

void popFront()Input range primitive: advance to next element.

private void _primeOutput()Produce output by feeding source data to compressor.

Constructors

this(SourceRange source, CompressionOptions opts, string providerName = null)Construct a compression range.

this()Disabled default constructor - must be constructed with source and options.

structDecompressRange(SourceRange) if (isInputRange!SourceRange && is(ElementType!SourceRange : const(ubyte)[]))

An input range that lazily decompresses data from a source range.

This wraps the existing Decompressor interface to provide pull-based semantics. Data is decompressed on-demand as the range is iterated.

The source range must yield elements convertible to const(ubyte)[].

Parameters

SourceRange The type of the source input range providing compressed data chunks.

Fields

private SourceRange _source

private Decompressor _decompressor

private const(ubyte)[] _currentOutput

private OutputBufferHolder _bufferHolder

private bool _finished

private bool _sourceExhausted

Methods

bool empty() @property constInput range primitive: check if range is exhausted.

const(ubyte)[] front() @property constInput range primitive: get current element.

void popFront()Input range primitive: advance to next element.

private void _primeOutput()Produce output by feeding source data to compressor.

Constructors

this(SourceRange source, DecompressionOptions opts, string providerName = null)Construct a decompression range.

this()Disabled default constructor - must be constructed with source and options.

structCompressOutputRange

An output range that compresses data written to it.

Compressed output is forwarded to a provided sink delegate. This is useful when you want to use range algorithms that write to an output range.

Call finish() when done writing to finalize the compressed stream.

Fields

private Compressor _compressor

private bool _finished

Methods

void put(const(ubyte)[] data)Output range primitive: accept data.

void finish()Finalize compression. Must be called when done writing.

void flush(FlushMode mode = FlushMode.SYNC)Flush pending data without finishing.

bool isFinished() @property constReturns true if finish() has been called.

ulong bytesInTotal() @property constReturns total uncompressed bytes written so far.

ulong bytesOutTotal() @property constReturns total compressed bytes produced so far.

Constructors

this(OutputSink sink, CompressionOptions opts, string providerName = null)Construct a compression output range.

this()Disabled default constructor - must be constructed with sink and options.

structDecompressOutputRange

An output range that decompresses data written to it.

Decompressed output is forwarded to a provided sink delegate. This is useful when you want to use range algorithms that write to an output range.

Call finish() when done writing to finalize the decompressed stream.

Fields

private Decompressor _decompressor

private bool _finished

Methods

void put(const(ubyte)[] data)Output range primitive: accept data.

void finish()Finalize decompression. Must be called when done writing.

bool isFinished() @property constReturns true if finish() has been called.

ulong bytesInTotal() @property constReturns total compressed bytes written so far.

ulong bytesOutTotal() @property constReturns total decompressed bytes produced so far.

Constructors

this(OutputSink sink, DecompressionOptions opts, string providerName = null)Construct a decompression output range.

this()Disabled default constructor - must be constructed with sink and options.

private classIdentityCompressor : Compressor

A pass-through compressor used by API-level unittests.

Fields

private CompressionOptions _opts

private void delegate(const(ubyte)[]) _sink

private bool _finished

Methods

CompressionOptions options() @property const

void setOutputSink(OutputSink s)

void setProgressCallback(ProgressCallback callback)

void write(const(ubyte)[] data)

void flush(FlushMode mode = FlushMode.SYNC)

void finish()

void reset()

ulong bytesInTotal() @property const

ulong bytesOutTotal() @property const

bool setDictionary(const(ubyte)[] dict)

bool isFinished() @property const

Constructors

this(CompressionOptions o)

private classIdentityDecompressor : Decompressor

A pass-through decompressor used by API-level unittests.

Fields

private DecompressionOptions _opts

private void delegate(const(ubyte)[]) _sink

private bool _finished

Methods

DecompressionOptions options() @property const

void setOutputSink(OutputSink s)

void setProgressCallback(ProgressCallback callback)

void write(const(ubyte)[] data)

void finish()

void reset()

ulong bytesInTotal() @property const

ulong bytesOutTotal() @property const

bool setDictionary(const(ubyte)[] dict)

bool isFinished() @property const

Constructors

this(DecompressionOptions o)

Functions 26

fnvoid registerProvider(CompressionProvider provider)Register a provider for a given format as specified by `provider.format`.

fnCompressionProvider[] providersFor(CompressionFormat fmt)Query available providers for a format. Returns a copy; mutating it does not affect the registry. Thread-safe.

private fnbool tryParseFormatId(string formatId, out CompressionFormat fmt)Internal helper: map a textual format id to a `CompressionFormat`.

fnCompressionProvider[] providersFor(string formatId)Query available providers for a string-identified format. The textual `formatId` is mapped to a `CompressionFormat` and the corresponding enum-based registry entry is returned.

fnCompressionFormat detectFormat(const(ubyte)[] headerBytes)Detect compression format from header bytes.

fnCompressionProvider selectProvider(CompressionFormat fmt, const(char)[] providerName)Pick the best matching provider by format and optional full provider name.

fnCompressionProvider selectProvider(string formatId, const(char)[] providerName)Pick the best matching provider by string formatId and optional full provider name.

private fnstring formatIdNormalize(string s) pure @safeNormalize a formatId for use as a registry key (lowercase).

fnCompressor makeCompressor(CompressionOptions opts, string providerName = null)Create a streaming compressor for the requested options.

fnDecompressor makeDecompressor(DecompressionOptions opts, string providerName = null)Create a streaming decompressor for the requested options.

fnCompressor makeCompressorById(string formatId, CompressionOptions opts, string providerName = null)Create a streaming compressor for a string-identified custom format.

fnDecompressor makeDecompressorById(string formatId, DecompressionOptions opts, string providerName = null)Create a streaming decompressor for a string-identified custom format.

ubyte[] compressBuffer(const(ubyte)[]  input,
       CompressionOptions opts =  CompressionOptions.init,
       string providerName =  null)

Compress a whole buffer in one call.

ubyte[] decompressBuffer(const(ubyte)[]  input,
       DecompressionOptions opts =  DecompressionOptions.init,
       string providerName =  null)

Decompress a whole buffer in one call.

fnubyte[] compressBuffer(CompressionFormat fmt, const(ubyte)[] input, string providerName = null)Convenience: compress a whole buffer with just a format specification.

fnubyte[] decompressBuffer(CompressionFormat fmt, const(ubyte)[] input, string providerName = null)Convenience: decompress a whole buffer with just a format specification.

fnsize_t estimateCompressedSize(CompressionFormat fmt, size_t inputSize) pure nothrow @nogcEstimate the maximum compressed output size for a given input size and format.

auto compressRange(Range)(Range source,  CompressionOptions opts,  string providerName =  null) if (isInputRange!Range &&  is(ElementType!Range :  const(ubyte)[]))

Convenience function to create a compression range.

auto compressRange(Range)(Range source,  CompressionFormat fmt) if (isInputRange!Range &&  is(ElementType!Range :  const(ubyte)[]))

Convenience overload to create a compression range with just a format.

auto decompressRange(Range)(Range source,  DecompressionOptions opts,  string providerName =  null) if (isInputRange!Range &&  is(ElementType!Range :  const(ubyte)[]))

Convenience function to create a decompression range.

auto decompressRange(Range)(Range source,  CompressionFormat fmt) if (isInputRange!Range &&  is(ElementType!Range :  const(ubyte)[]))

Convenience overload to create a decompression range with just a format.

fnauto compressOutputRange(OutputSink sink, CompressionOptions opts, string providerName = null)Convenience function to create a compression output range.

fnauto compressOutputRange(OutputSink sink, CompressionFormat fmt)Convenience overload to create a compression output range with just a format.

fnauto decompressOutputRange(OutputSink sink, DecompressionOptions opts, string providerName = null)Convenience function to create a decompression output range.

fnauto decompressOutputRange(OutputSink sink, CompressionFormat fmt)Convenience overload to create a decompression output range with just a format.

private fnvoid registerIdentityCodec()

Variables 5

enumvarDEFAULT_PROGRESS_INTERVAL = 64 * 1024

Default progress callback interval in bytes (64 KiB).

private varCompressionProvider[][CompressionFormat] gProviders

private varObject gProvidersLock

Guards every access to gProviders so registration and lookups are race-free.

private enumvarTEST_IDENTITY_PROVIDER_VENDOR = "apiidentitycodec"

private enumvarTEST_IDENTITY_PROVIDER_NAME = "apiidentitycodec-lz4"