ddn.df.io.ddndf

The columnar file format of ddn.df (ADR-2026-10-24-15): a self-describing binary serialization of a DataFrame's sealed chunks — the same buffers and type mapping as the Arrow C Data Interface, in a simple framed format (doc/DDN-DF.md). The interchange boundary (the C Data Interface in ddn.df.arrow) is what Polars, pyarrow and DuckDB speak; this file format is ddn-df's own cache and persistence layer, designed for round-trip fidelity with Arrow-compatible semantics. A true Arrow IPC (FlatBuffers) writer and reader for interop with external Arrow tooling is the dependency-gated follow-up.

Format (all integers little-endian): magic "DDNDF1\0\0" (8 bytes) n_columns uint32 per column: name_len uint32 name bytes kind ubyte unit ubyte (TimeUnit) meta element_kind(1)[dict_kind(1)] precision(1) (6 bytes, always present) row_count uint64 null_count uint64 n_children uint32 validity validity_bytes_len(uint32) values values_bytes_len(uint32) extra extra_bytes_len(uint32) children recursively

Copyright

License

BSD-3-Clause

var DFC_MAGIC DFC_MAGIC_LEN

fn appendU32 appendU64 buildColumnar parseColumnar readChunkAt readColumnar readU32 readU64 readU8 writeChunk writeColumnar

Functions 11

private fnvoid appendU32(ref ubyte[] arr, uint v) @trusted pure nothrow

private fnvoid appendU64(ref ubyte[] arr, ulong v) @trusted pure nothrow

fnubyte[] buildColumnar(DataFrame df) @trustedSerializes a DataFrame to the columnar file format.

fnvoid writeColumnar(string path, DataFrame df) @trustedWrites a DataFrame to a columnar file.

private fnvoid writeChunk(ref ubyte[] out_, const ArrayData chunk) @trusted

private fnuint readU32(const(ubyte)[] d, ref size_t p) @trusted pure nothrow

private fnulong readU64(const(ubyte)[] d, ref size_t p) @trusted pure nothrow

private fnubyte readU8(const(ubyte)[] d, ref size_t p) @trusted pure nothrow

fnDataFrame parseColumnar(const(ubyte)[] data) @trustedParses columnar file bytes into a DataFrame.

fnDataFrame readColumnar(string path) @trustedReads a columnar file into a DataFrame.

private fn

ArrayData readChunkAt(const(ubyte)[]  data,  ref  size_t p,
       string name)

@trusted

Variables 2

enumvarDFC_MAGIC = "DDNDF1"

enumvarDFC_MAGIC_LEN = 8