ddn.df.io.csv

The CSV layer of ddn.df: reading frames from CSV text and files over ddn.data.csv (T29; D18; ADR-2026-10-18-16). Writing follows in T30.

Every ddn.data.csv result is a CsvResult, checked where it is produced; a failure becomes a DataFrameException carrying the line. CsvResult never reaches the ddn.df API. The header is read here (dialect.header = false plus makeHeaderIndex) so schema fields match by name; floats special-case NaN/inf/-inf (which fromCsv!double does not handle); string cells copy without unescaped() unless needsUnescape is set; one chunk per chunkRows (default 65 536) so real input exercises the multi-chunk paths.

Opt-in schema inference (ADR-2026-10-24-19) is a two-pass candidate join over the same field parsing: inferCsvSchema walks the sampled rows joining per-cell candidates in a semilattice with STRING on top, and the inferred read replays the explicit reader with the composed schema under strict field counts. The explicit-schema signatures, defaults and behaviour are unchanged.

Copyright

License

BSD-3-Clause

enum CsvCandidate

struct CellSource ColumnWriter CsvInferPolicy CsvReadPolicy CsvTextBlock CsvWritePolicy RenderedColumn RowBuffer RowRenderSpec SegmentScan TmpSink

fn appendCell appendRowField candidateOf cellText dataTypeOfCandidate emptyFrame ensureCapacity failCell finishBlocks floatCell inferCsvSchema inferJoinBlocks inferWalk isBoolWord isCsvReadableKind isFloatSpelling isNullCell isNumericCandidate joinCandidate parseLongChecked parseRows putInt putUInt readCsv readCsvText readCsvTextBlocks readCsvTextCore renderColumn renderInts renderUInts rowRenderSpecOf signedCellText splitCsvBlocks splitCsvBlocksParallel splitCsvBlocksSerial unsignedCellText writeCsv writeCsvFile writeRowsFast writeRowsViaWriter

Types 12

structCsvReadPolicy

How readCsv interprets text and structures the frame.

Fields

bool hasHeaderWhether the first row holds column names (matched by name).

string[] nullValuesField texts that mean null (in addition to the empty field).

ddn.data.csv.CsvDialect dialectThe CSV dialect (delimiter, quoting); `header` is forced off.

size_t chunkRowsRows per chunk; one chunk per block.

structColumnWriter

One column's block writer: the builder selected by the column kind.

Fields

DataType _dtype

BooleanBuilder _bools

StringBuilder _strings

PrimitiveBuilder!long _longs

PrimitiveBuilder!double _doubles

Methods

ColumnWriter of(DataType dtype) static

void nullCell()

ArrayData finish()

private structCsvTextBlock

One block of CSV text between unquoted newlines: firstLine is the 1-based line number of the block's first row, newlines how many newline characters the block contains.

Fields

const(char)[] text

size_t firstLine

size_t newlines

private structSegmentScan

One segment's scan under the even-start assumption: the quote parity of the whole segment and the last newline of each preceding parity class (a newline is truly unquoted when the parity at the segment start makes its local class even).

Fields

bool parity

size_t lastEven

size_t lastOdd

structCsvInferPolicy

How opt-in CSV schema inference scans and types columns: a two-pass candidate join over the same field parsing as the explicit reader (ADR-2026-10-24-19). Inference is additive - the explicit-schema signatures, defaults and behaviour are unchanged.

Fields

CsvReadPolicy readThe reading policy reused verbatim (`hasHeader`, `nullValues`, `dialect`, `chunkRows`).

size_t sampleRowsData rows scanned for the schema. The default scans every row, so no late data can contradict the schema; `0` types every column STRING (Polars' `infer_schema_length = 0`).

bool tryParseTemporalWhether DATETIME/TIME candidates are tested (Polars' `try_parse_dates`; off by default, temporals then infer STRING).

Schema overridesColumns whose kinds are forced, matched by name when `read.hasHeader` and by position otherwise; every other column is inferred. Flat reader kinds only.

private enumCsvCandidate : ubyte

One cell's provisional kind during inference; a column's kind is the join of its cells' candidates (ADR-2026-10-24-19).

NULLA null cell (empty or a `nullValues` token); joins away.

BOOLEAN`true`/`false`, word form only - `1`/`0` are integers.

INT64`fromCsv!long` accepts.

UINT64`fromCsv!long` rejects on magnitude and `fromCsv!ulong` accepts.

FLOAT64`fromCsv!double` accepts, or a special float spelling.

DATETIMEThe strict ISO-8601 date-time grammar (opt-in).

TIMEThe strict `HH:MM:SS[.f{1,9}]` grammar (opt-in).

STRINGAnything else; also the top of the join lattice.

structCsvWritePolicy

How writeCsv renders the frame: one column-major render pass per column (ADR-2026-10-18-21).

Fields

bool writeHeaderWhether to write the column names as the first row.

ddn.data.csv.CsvDialect dialectThe output dialect; the newline defaults to FORCE_LF below.

string nullValueThe text written for null slots.

private structRenderedColumn

One column rendered to cell text: the concatenated cell bytes plus the rows + 1 offsets delimiting every cell - the write path's column-major scratch, so the row loop never allocates per cell.

Fields

char[] bytesThe concatenated cell bytes (unquoted; the writer quotes).

size_t[] offsetsThe cell boundaries: cell `r` is `bytes[offsets[r] .. offsets[r + 1]]`.

Methods

char[] cell(size_t r) @safe pure nothrowParams: r = the row

private structRowRenderSpec

The writeCsv fast-path row-rendering tables, precomputed once per call from the dialect: which bytes force quoting and how a quote inside a quoted cell escapes. The rules mirror ddn.data.csv.CsvWriter exactly - byte parity with it is the contract, and the differential unittests hold it.

Fields

bool[256] forcesQuote`forcesQuote[b]` when byte `b` alone forces quoting (the delimiter, the quote char, CR, LF).

char delimiterThe field separator written between cells.

char quoteThe quote char wrapped around quoted cells.

bool doubleQuoteWhether a quote inside a quoted cell doubles (RFC 4180).

bool backslashEscapeWhether a quote inside a quoted cell backslash-escapes instead (only consulted when `doubleQuote` is false).

bool trimWhitespaceWhether leading/trailing blanks force quoting.

private structRowBuffer

The per-segment row buffer of the fast path: a plain growing byte array with a write position - cheaper than an Appender for the row loop's many tiny puts, and its unique ownership lets the finished block become the segment's string without a copy.

Fields

private char[] _buf

private size_t _pos

Methods

void reserve(size_t n) @safe pure nothrowPre-grows the buffer to at least `n` bytes.

void putByte(char c) @safe pure nothrowAppends one byte.

void putSlice(scope const(char)[] s) @safe pure nothrowAppends a slice.

private void grow(size_t need) @safe pure nothrowGrows the buffer to at least `need` bytes (amortised doubling).

string take() @trusted pure nothrowReturns: the written bytes as the segment's output block. The buffer is uniquely owned and abandoned by the caller, so `assumeUnique` moves it without a copy.

private structTmpSink

A minimal output sink for std.format.formatValue over a stack buffer: put overloads for one char and a slice, a write position, no allocation. With a default-initialised FormatSpec!char this is exactly to!string's formatting pipeline (which std.conv itself drives the same way), minus the appender allocation and the runtime format-string parsing of sformat.

Fields

char[] buf

size_t pos

Methods

void put(char c) @safe pure nothrowAppends one character.

void put(scope const(char)[] s) @safe pure nothrowAppends a slice.

private structCellSource

One column's sequential cell source for the row renderer: walks the column's chunks in ascending row order, formatting each cell straight into the row buffer - integers through the in-buffer decimal writer, floats through formatValue (the to!string pipeline without its allocation), temporals through the strict into-buffer formatters, strings copied from the chunk without an intermediate render. The typed view of the current chunk is cached and rebuilt only on chunk crossings, so the per-cell loop does no view construction; the kind dispatch is one jump table per cell.

Fields

private const(ArrayData)[] _chunks

private size_t _chunk

private size_t _chunkStart

private size_t _chunkLen

private Kind _kind

private BooleanArray _bol

private StringArray _str

private PrimitiveArray!byte _i8

private PrimitiveArray!short _i16

private PrimitiveArray!int _i32

private PrimitiveArray!long _i64

private PrimitiveArray!ubyte _u8

private PrimitiveArray!ushort _u16

private PrimitiveArray!uint _u32

private PrimitiveArray!ulong _u64

private PrimitiveArray!float _f32

private PrimitiveArray!double _f64

Methods

CellSource of(Series col, size_t fromRow) static @safe pure nothrowParams: col = the column to serve fromRow = the first global row to serve

private void enterChunk(size_t i) @safe pure nothrow

void appendCell(ref  RowBuffer buf,  size_t r,
          ref  const  RowRenderSpec spec,  string nullValue)

@safeAppends the cell of global row `r` to `buf`, quoted per `spec`.

Functions 44

DataFrame readCsvText(const(char)[]  text,  Schema schema,
       CsvReadPolicy policy =  CsvReadPolicy.init)

Reads a frame from CSV text with an explicit schema.

private fn

DataFrame readCsvTextCore(const(char)[]  text,  Schema schema,
       CsvReadPolicy policy,  size_t strictWidth)

The reader core shared by the explicit and inferred entries.

private fn

ArrayData[][] parseRows(Rows)(Rows rows,  size_t line,
       const  size_t[]  columnOf,  Schema schema,  CsvReadPolicy policy,
       size_t strictWidth)

Parses `rows` (a `byRows` range positioned at a data row) into per- column chunk lists, one chunk per `policy.chunkRows` rows. `line` is the 1-based line number of `rows.front`; both the serial and...

private fn

CsvTextBlock[] splitCsvBlocks(const(char)[]  text,  size_t maxBlocks,
       char  quote =  '"')

Splits `text` into at most `maxBlocks` blocks, cutting only at newlines outside quoted fields (quote parity is tracked, so an embedded newline never splits a field). The cut points are a pure funct...

private fn

CsvTextBlock[] splitCsvBlocksSerial(const(char)[]  text,
       size_t maxBlocks,  char  quote)

The single-pass walk (small texts, single thread, sparse newlines).

private fn

CsvTextBlock[] splitCsvBlocksParallel(const(char)[]  text,
       size_t maxBlocks,  char  quote)

private fn

DataFrame readCsvTextBlocks(const  CsvTextBlock[]  blocks,
       Schema schema,  CsvReadPolicy policy,  ddn.data.csv.CsvDialect dialect,
       size_t strictWidth)

private fnDataFrame finishBlocks(const ArrayData[][][] results, Schema schema)

DataFrame readCsv(string path,  Schema schema,
       CsvReadPolicy policy =  CsvReadPolicy.init)

Reads a frame from a CSV file with an explicit schema.

private fnbool isNumericCandidate(CsvCandidate c) @safe pure nothrowReturns: whether `c` is in the numeric chain INT64-UINT64-FLOAT64.

private fnCsvCandidate joinCandidate(CsvCandidate a, CsvCandidate b) @safe pure nothrowJoins two candidates into one column kind: the lattice of ADR-2026-10-24-19 with STRING on top. NULL joins away; the numeric chain collapses upward (integers beyond `long` have no exact common inte...

private fnDataType dataTypeOfCandidate(CsvCandidate c) @safe pure nothrowReturns: the candidate's `DataType` (NULL maps to STRING).

private fnbool isCsvReadableKind(Kind k) @safe pure nothrowReturns: whether the kind is a flat kind the CSV reader parses.

private fnbool isBoolWord(const(char)[] t) @safe pure nothrowReturns: whether `t` is `true`/`false` in any ASCII case (word form only - `fromCsv!bool` also accepts `1`/`0`, which inference types as integers so a later `2` cannot flip a column's kind).

private fnbool isFloatSpelling(const(char)[] t) @safe pure nothrowReturns: whether `t` is one of the explicit reader's special float spellings (`fromCsv!double` parses digits/dot/exponent only).

private fn

CsvCandidate candidateOf(ddn.data.csv.FieldView cell,
       const  CsvReadPolicy read,  bool  tryParseTemporal)

@safeOne cell's candidate: null, word-form boolean, `long`, `ulong`, `double` (or a special spelling), an opt-in strict temporal, else STRING. The grammars are pairwise disjoint given word-form booleans...

private fn

CsvCandidate[] inferWalk(Rows)(Rows rows,  size_t line,  size_t width,
       const  CsvReadPolicy read,  bool  tryParseTemporal,  size_t rowLimit)

Walks up to `rowLimit` data rows, joining every cell's candidate into per-column accumulators. Every walked row must hold exactly `width` fields (the inferred contract): the first mismatch errors w...

private fn

CsvCandidate[] inferJoinBlocks(const  CsvTextBlock[]  blocks,
       size_t width,  const  CsvReadPolicy read,  bool  tryParseTemporal,
       bool  hasHeader,  ddn.data.csv.CsvDialect dialect)

Full-scan inference over the text blocks: each block joins its own per-column candidates (block 0 past the header), and the per-block results join in block order. The join is associative and commut...

Schema inferCsvSchema(const(char)[]  text,
       CsvInferPolicy policy =  CsvInferPolicy.init)

Infers the schema of CSV text: the opt-in alternative to the explicit schema (ADR-2026-10-24-19). Every sampled cell contributes a candidate; per column the candidates join in the lattice with STRI...

fnDataFrame readCsvText(const(char)[] text, CsvInferPolicy policy)Reads a frame from CSV text with an inferred schema (ADR-2026-10-24-19): `inferCsvSchema` composes the schema (full-scan by default, finite sample opt-in, overrides per column), then the existing r...

fnDataFrame readCsv(string path, CsvInferPolicy policy)Reads a frame from a CSV file with an inferred schema (ADR-2026-10-24-19); the text overload documents the contract.

private fnDataFrame emptyFrame(Schema schema)

void appendCell(ref  ColumnWriter w,  DataType dtype,  string column,
       ddn.data.csv.FieldView cell,  CsvReadPolicy policy,  size_t line)

private fn

bool isNullCell(ddn.data.csv.FieldView cell,
       const  CsvReadPolicy policy)

@safe pure nothrow

private fnconst(char)[] cellText(ddn.data.csv.FieldView cell) @safe

private fn

long parseLongChecked(ddn.data.csv.FieldView cell,  DataType dtype,
       string column,  size_t line)

@safe

private fnvoid failCell(size_t line, string column, const(char)[] what) pure @safe

private fnvoid ensureCapacity(ref char[] buf, size_t need) @safe pure nothrowGrows a render buffer to at least `need` bytes (amortised doubling, exact for bulk jumps).

private fnsize_t putInt(return scope char[] buf, size_t at, long v) @safe pure nothrowWrites the decimal text of a signed integer into `buf[at ..]` (`long.min` goes through the unsigned path).

private fnsize_t putUInt(return scope char[] buf, size_t at, ulong v) @safe pure nothrowWrites the decimal text of an unsigned integer into `buf[at ..]`.

private fnRenderedColumn renderColumn(Series col, string nullValue)Renders one column to cell text in one typed pass over its chunks: integers through the in-buffer decimal writer, temporals through the strict into-buffer formatters, floats through `to!string` (D'...

private fn

size_t appendCell(ref  RenderedColumn result,  size_t at,  size_t row,
       bool  isNull,  string nullValue,  scope  const(char)[]  text)

@safe pure nothrowAppends one rendered cell to `result`: `nullValue` at null slots, `text` otherwise, the offsets advancing one row.

private fn

void renderInts(Kind K,  T)(Series col,  ref  RenderedColumn result,
       ref  char[21]  buf,  string nullValue)

@safe pure nothrowRenders every element of the signed-integer column `col` with the in-buffer decimal writer.

private fn

void renderUInts(Kind K,  T)(Series col,  ref  RenderedColumn result,
       ref  char[21]  buf,  string nullValue)

@safe pure nothrowDitto, for unsigned-integer columns.

private fnRowRenderSpec rowRenderSpecOf(CsvDialect dialect) @safe pure nothrowBuilds the fast-path rendering tables from `dialect`.

private fn

void appendRowField(ref  RowBuffer buf,  scope  const(char)[]  cell,
       ref  const  RowRenderSpec spec,  size_t row)

@safeAppends one cell to the row buffer, applying the dialect's quoting exactly like `ddn.data.csv.CsvWriter`: ordinary bytes copy straight through; a cell holding the delimiter, the quote, CR or LF (or...

private fnconst(char)[] floatCell(T)(T v, return scope char[] tmp) @safeFormats one float into `tmp` through `formatValue` with the default `%s` spec - the shortest round-trip text, byte-identical to `to!string` (pinned by the differential sweep below).

private fn

const(char)[] signedCellText(long  v,  bool  valid,
       return  scope  char[]  tmp,  string nullValue)

@safe pure nothrowThe signed-integer cell text: the decimal writer's output, or `nullValue` at a null slot.

private fn

const(char)[] unsignedCellText(ulong  v,  bool  valid,
       return  scope  char[]  tmp,  string nullValue)

@safe pure nothrowDitto, for unsigned integers.

private fn

string[] writeRowsFast(DataFrame df,  size_t rows,
       string nullValue,  RowRenderSpec spec,  size_t segRows,  size_t nSegs)

Renders the rows of each segment straight into the segment's own buffer through the fast path: no `CsvWriter`, no `FieldView`, no intermediate `RenderedColumn` - every cell is formatted or copied e...

private fn

string[] writeRowsViaWriter(scope  RenderedColumn[]  rendered,
       size_t rows,  size_t width,  CsvDialect dialect,  size_t segRows,
       size_t nSegs)

The writer fallback: the original segment loop streaming every row through `ddn.data.csv.CsvWriter` over a per-segment buffer. Retained for dialects the fast path does not take (currently only the ...

void writeCsv(Sink)(ref  Sink sink,  DataFrame df,
       CsvWritePolicy policy =  CsvWritePolicy.init)

Writes the frame as CSV rows through ddn-df's own row renderer: row segments rendered straight into per-segment buffers (every cell formatted or copied exactly once, quoting rules mirrored from `dd...

void writeCsv(string path,  DataFrame df,
       CsvWritePolicy policy =  CsvWritePolicy.init)

Writes the frame to a file (the parent directory must exist).

void writeCsvFile(string path,  LazyFrame df,
       CsvWritePolicy policy =  CsvWritePolicy.init,
       size_t batchSize =  65_536)

@trustedWrites a deferred frame to a CSV file incrementally: batches pull through the row-wise chain one at a time (the input never materialises), the header writes once, and each pull appends its rows. Th...