ddn.df.io.scan

The lazy IO scan sources of ddn.df (ADR-2026-10-29-03, amended 2026-10-29 for the input modes): scanCsv reads CSV incrementally without loading a complete frame first. Two input modes: memory- mapped files (the fast path over seekable sources) and a buffered streaming mode (any source - pipes, stdin, sockets) that retains only the incomplete trailing record. Both modes parse serially with the eager reader's exact field handling, so errors surface in file order at the first failing batch.

Copyright

License

BSD-3-Clause

enum CsvScanMode

struct CsvBatchParser

class CsvScanBuffered CsvScanMmap

fn contains openCsvScan scanCsv

Types 4

enumCsvScanMode

The CSV scan's input mode (the author's 2026-10-29 amendment: non-seekable sources stay first-class).

AUTOmmap for regular files, buffered otherwise

MMAPmemory-mapped input only (regular file required)

BUFFEREDstreaming input (works for any source)

private structCsvBatchParser

Fields

Schema schema

CsvReadPolicy policy

size_t[] columnOf

size_t line

string[] projected

Methods

private

Nullable!DataFrame emitBatch(R)(ref  R rows,
          const(char) *  suspectEnd =  null)

@trusted

private void consumeHeader(R)(ref R rows) @trusted

private classCsvScanMmap : ScanSource

Fields

private MmFile mm

private CsvBatchParser parser

private string path

private bool closed

private bool started

private typeof(byRows(cast(const(char)[]) null,
          CsvReadPolicy.init.dialect))rows rows

Methods

ScanSource acquire() @safe

ColumnSchema[] schema() @safe

string description() const @safe

bool supportsProjection() @safe

ScanSource project(scope string[] columns) @safe

Nullable!DataFrame nextBatch() @trusted

void close() @trusted

Constructors

this(string path,  Schema schema,  CsvReadPolicy policy,
          string[]  projected =  null)

private classCsvScanBuffered : ScanSource

Fields

private File input

private CsvBatchParser parser

private string label

private string reopenPath

private bool closed

private bool started

private bool eof

private char[] buffer

private bool acquired_

Methods

ScanSource acquire() @safe

ColumnSchema[] schema() @safe

string description() const @safe

bool supportsProjection() @safe

ScanSource project(scope string[] columns) @safe

Nullable!DataFrame nextBatch() @trusted

private void fillBuffer() @trusted

private void advancePast(R)(ref R rows) @trusted

void close() @trusted

Constructors

this(File input,  Schema schema,  CsvReadPolicy policy,  string label,
          string reopenPath =  null,  string[]  projected =  null)

Functions 4

LazyFrame scanCsv(string path,  Schema schema,
       CsvReadPolicy policy =  CsvReadPolicy.init,
       CsvScanMode mode =  CsvScanMode.AUTO)

@trustedOpens a CSV file as a lazy scan source: the returned deferred frame pulls one batch per `policy.chunkRows` rows when collected or streamed, without loading the file first. The schema is explicit (i...

LazyFrame scanCsv(File input,  Schema schema,
       CsvReadPolicy policy =  CsvReadPolicy.init)

@trustedDitto, from an already-open handle (a pipe, stdin, a socket stream - the buffered mode by construction; the source owns the handle until close()).

private fnbool contains(string[] names, string n) @safe pure nothrow

private fn

ScanSource openCsvScan(string path,  Schema schema,
       CsvReadPolicy policy,  CsvScanMode mode)

@trusted