ddn.data.arrow.ipc

The Arrow IPC file reader of ddn.df (ADR-2026-10-24-15, amended 2026-10-29): the .arrows/Feather V2 format read through a self-owned minimal FlatBuffers subset (the Message/Schema/Field/ RecordBatch/Footer/Block tables the format defines - no FlatBuffers library, no generated code). Admitted types mirror the C Data tier (flat kinds, large_list/list, dictionary-encoded categorical, decimal128); STRUCT and unsupported shapes reject naming themselves. Uncompressed bodies and LZ4_FRAME compression decode (the DDN pure-D codec); other codecs reject naming themselves. Every length, offset and count is validated before any allocation: malformed input throws, never a partial frame. The private DDNDF1 format (ddn.df.arrow_ipc) is a distinct boundary, not relabelled here.

Copyright

License

BSD-3-Clause

var CODEC_LZ4_FRAME CODEC_ZSTD IPC_CONTINUATION IPC_MAGIC IPC_MAX_METADATA ipcColumnsDecoded MSG_DICTIONARY MSG_RECORD_BATCH TYPE_BOOL TYPE_DATE TYPE_DECIMAL TYPE_DURATION TYPE_FLOAT TYPE_INT TYPE_LARGE_LIST TYPE_LARGE_UTF8 TYPE_LIST TYPE_STRUCT TYPE_TIME TYPE_TIMESTAMP TYPE_UTF8

enum IpcCompression

struct BatchWalk BodyWriter FbBuilder FbTable FbVector FieldSpec Message TableSlots

class IpcBatchReader IpcBatchWriter

fn buildField buildIpcBytes buildMessage buildRecordBatch buildSchema buildTypeField checkMagic collectBuffers collectNodes dictionaryLargeUtf8 encapsulate finishTable flatChild intOf ipcError isSignedKind lz4Decode messageAt physicalSize readFields readLeI32 readLeU32 recodeCategorical

Types 11

private structFbTable

Fields

const(ubyte)[] buf

size_t pos

Methods

private size_t fieldAt(uint slot) const @safe pure

byte i8(uint slot, byte def = 0) const @safe pure

ubyte u8(uint slot, ubyte def = 0) const @safe pure

short i16(uint slot, short def = 0) const @safe pure

int i32(uint slot, int def = 0) const @safe pure

long i64(uint slot, long def = 0) const @safe pure

private size_t indirect(size_t at) const @safe pure

FbTable table(uint slot) const @safe pure

string str(uint slot) const @trusted pure

FbVector vector(uint slot) const @safe pure

FbTable root(const(ubyte)[] buf) static @safe pure

private ushort readU16(size_t at) const @safe pure

private uint readU32At(const(ubyte)[] buf, size_t at) static @safe pure

long i64At(size_t at) const @safe pure

int i32At(size_t at) const @safe pure

private uint readU32(size_t at) const @safe pure

private int readI32(size_t at) const @safe pure

private structFbVector

Fields

size_t length

size_t pos

Methods

FbTable tableAt(FbTable fb, size_t i) const @safe pure

void blockAt(FbTable fb,  size_t i,  out  long  offset,  out  int  metaLen,
          out  long  bodyLength)

const @safe pure

private structFieldSpec

Fields

string name

DataType dtype

bool largeList

bool largeUtf8

long dictId

private structMessage

Fields

ubyte headerType

FbTable header

const(ubyte)[] body

private structBatchWalk

Fields

private FbTable batch

private const(ubyte)[] body

private long[][] nodes

private long[][] buffers

private size_t nextNode

private size_t nextBuffer

private bool lz4

private bool loaded

Methods

private void load() @safe

private long[] nodeCounts() @safe

private const(ubyte)[] bufferBytes() @safe

private Bitmap validity(long[] n) @trusted

void skipArray(FieldSpec spec) @trusted

ArrayData readArray(FieldSpec spec) @trusted

Constructors

this(Message msg)

classIpcBatchReader

The Arrow IPC batch reader: the codec tier below the frame and query adapters. It owns its input - either a whole in-memory file image or a file path whose handle opens lazily at the first batch and closes on close() (idempotent; the footer loads through an open-read-close pass so an unpulled source holds no handle). The footer, schema and dictionaries load eagerly; each nextBatch decodes one record batch into a ColumnBatch (one chunk per column). Projection is the codec's column-subset capability: projected fields skip decoding entirely, under the DECIMAL totality rule (a DECIMAL column stays in the scan because its decode may reject).

Fields

private const(ubyte)[] whole

private string path

private File file

private FieldSpec[] fields

private long[][] dictBlocks

private long[][] batchBlocks

private ArrayData[long] dictValues

private size_t nextBlock

private bool closed

private string[] projectedNames

private bool footerLoaded

Methods

private void ensureFooter() @trusted

private

long[][] readBlocks(FbTable footerTable,
          const(ubyte)[]  buf,  uint  slot)

static @safe pure

private void loadDictionaries(ref File f) @trusted

private Message readMessage(ref File f, long[] b) @trusted

Schema schema() @trustedReturns: the file's structural schema (every field; the projection does not change it).

Nullable!ColumnBatch nextBatch() @trustedDecodes the next record batch into a `ColumnBatch` (one chunk per column; projected fields skip decoding). Returns null at the end or after `close`.

private Message loadBlock(long[] b) @trusted

void close() @trustedCloses the owned handle (idempotent); further pulls return null.

IpcBatchReader reacquire() @safeA fresh reader over the same input with the same projection (one independent pull lifecycle per acquisition).

IpcBatchReader project(scope string[] columns) @safe

private bool isProjected(scope string[] columns, string n) static @safe pure nothrow

Constructors

this(const(ubyte)[] bytes)Reads from a whole in-memory file image.

this(string path)Reads from a file path; the handle opens lazily at the first batch (Windows holds a delete/renamed file hostage to any open handle, and a source is often never pulled - every collection pulls an ac...

private structFbBuilder

Fields

private ubyte[] buf

Methods

private void prepend(in ubyte[] what) @trusted pure nothrow

private void padTo(ubyte align_) @trusted pure nothrow

size_t u8_(ubyte v) @trusted pure nothrow

size_t u16_(ushort v) @trusted pure nothrow

size_t i16_(short v) @trusted pure nothrow

size_t u32_(uint v) @trusted pure nothrow

size_t i32_(int v) @trusted pure nothrow

size_t i64_(long v) @trusted pure nothrow

size_t str_(string s) @trusted pure nothrow

size_t offsetVector(size_t[] objectEnds) @trusted pure nothrow

size_t struct16Vector(long[][] pairs) @trusted pure nothrow

size_t blockVector(long[][] blocks) @trusted pure nothrow

size_t bufLength() @safe pure nothrowThe current builder length (bytes written so far).

void patch4AtEnd(size_t e, uint v) @trusted pure nothrow

ubyte[] finish(size_t rootEnd) @trusted pure nothrow

private structTableSlots

Fields

private uint[] slots

private ubyte[] kinds

private long[] values

private size_t[] writeEnds

Methods

void addU8(uint slot, ubyte v) @safe pure nothrow

void addI16(uint slot, short v) @safe pure nothrow

void addI32(uint slot, int v) @safe pure nothrow

void addI64(uint slot, long v) @safe pure nothrow

void addOffset(uint slot, size_t targetEnd) @safe pure nothrow

uint maxSlot() @safe pure nothrow

enumIpcCompression

The body compression of written files (none by default; LZ4 as the explicit option - our own bytes stay codec-free unless asked).

NONE

LZ4_FRAME

private structBodyWriter

Fields

ubyte[] body

long[][] bufferEntries

IpcCompression compression

Methods

void addBuffer(const(ubyte)[] raw) @trusted

void addBitmap(const ArrayData chunk) @trusted

private void pad8() @safe pure nothrow

classIpcBatchWriter

Fields

private File file

private string[] names

private DataType[] dtypes

private long[][] batchBlocks

private IpcCompression compression

private bool finished_

private long written

Methods

void addBatch(ColumnBatch batch) @trusted

private void writeBytes(const(ubyte)[] bytes) @trusted

void finish() @trusted

bool finished() const @safe pure nothrowReturns: whether finish already ran.

void abandon() @trusted

Constructors

this(File file,  string[]  names,  DataType[]  dtypes,
          IpcCompression compression)

Takes ownership of the opened output file and writes the magic and schema message.

Functions 23

private fnDataFrameException ipcError(string what) @safe pure

private fnuint readLeU32(const(ubyte)[] b, size_t at) @safe pure

private fnint readLeI32(const(ubyte)[] b, size_t at) @safe pure

private fnFieldSpec[] readFields(FbTable schema) @safe pure

private fnFieldSpec flatChild(FbTable child, string parent) @safe pure

private fnDataType intOf(int bits, bool signed) @safe pure

private fn

Message messageAt(const(ubyte)[]  bytes,
       long  offsetIn,  int  metaLenIn,  long  bodyLengthIn)

@safe pure

private fnsize_t physicalSize(Kind k) @safe pure nothrow

private fnconst(ubyte)[] lz4Decode(const(ubyte)[] raw) @trusted

private fn

ArrayData recodeCategorical(ArrayData codes,
       ArrayData labels)

@trusted

private fnbool dictionaryLargeUtf8(FieldSpec[] fields, long id) @safe pure nothrow

private fnvoid checkMagic(const(ubyte)[] eight) @trusted pure

private fnsize_t finishTable(ref FbBuilder b, ref TableSlots t) @trusted pure nothrow

private fnvoid collectNodes(const ArrayData chunk, ref long[][] nodes) @safe pure

private fnvoid collectBuffers(ref BodyWriter body, const ArrayData chunk) @trusted

private fn

void buildTypeField(ref  FbBuilder b,  DataType dtype,
       out  size_t typeEnd,  out  ubyte  typeTag)

@trusted pure nothrow

private fnbool isSignedKind(Kind k) @safe pure nothrow

private fn

size_t buildField(ref  FbBuilder b,  string name,  DataType dtype,
       long  fieldOrdinal)

@trusted pure

private fn

size_t buildSchema(ref  FbBuilder b,  string[]  names,
       DataType[]  dtypes)

@trusted pure

private fn

size_t buildMessage(ref  FbBuilder b,  ubyte  headerType,
       size_t headerEnd,  long  bodyLength)

@trusted pure nothrow

private fn

size_t buildRecordBatch(ref  FbBuilder b,  long  length,
       long[][]  nodes,  long[][]  buffers,  bool  lz4)

@trusted pure nothrow

private fnubyte[] encapsulate(ubyte[] fb) @trusted pure nothrow

ubyte[] buildIpcBytes(ColumnBatch batch,
       IpcCompression compression =  IpcCompression.NONE)

@trustedSerializes a frame to the Arrow IPC file format (`.arrows`, Feather V2): one schema message, one dictionary batch per categorical column, one record-batch message per aligned chunk row (a frame who...

Variables 21

private enumvarIPC_MAGIC = "ARROW1"

private enumvarIPC_CONTINUATION = 0xFFFF_FFFF

private enumvarIPC_MAX_METADATA = 64 * 1024 * 1024

private enumvarMSG_DICTIONARY = 2

private enumvarMSG_RECORD_BATCH = 3

private enumvarTYPE_INT = 2

private enumvarTYPE_FLOAT = 3

private enumvarTYPE_UTF8 = 5

private enumvarTYPE_BOOL = 6

private enumvarTYPE_DECIMAL = 7

private enumvarTYPE_TIME = 9

private enumvarTYPE_TIMESTAMP = 10

private enumvarTYPE_DATE = 8

private enumvarTYPE_DURATION = 18

private enumvarTYPE_LIST = 12

private enumvarTYPE_STRUCT = 13

private enumvarTYPE_LARGE_UTF8 = 20

private enumvarTYPE_LARGE_LIST = 21

private enumvarCODEC_LZ4_FRAME = 0

private enumvarCODEC_ZSTD = 1

varsize_t ipcColumnsDecoded