ddn.data.csv

Newline policy controls how record boundaries are detected and, for writer, how newlines are emitted.

Planned values match RFC 4180 defaults while allowing practical overrides.

import ddn.data.csv;

// Reader detects CRLF/LF by default
auto d1 = CsvDialect.init; // newlinePolicy = DETECT

// Force writer to emit LF newlines
auto d2 = CsvDialect.init; d2.newlinePolicy = NewlinePolicy.FORCE_LF;

Escape style for non‑RFC datasets (optional extension). none — only RFC 4180 double‑quote escaping inside quoted fields. backslash — treat backslash as escape for delimiter/quote/newline in unquoted fields.

Example:

import ddn.data.csv;
auto d = CsvDialect.init;
d.escapeStyle = EscapeStyle.NONE; // RFC 4180 behavior (default)

Reader error handling mode.

• PERMISSIVE (default): malformed rows are skipped and errors are counted; iteration continues. Optionally collect diagnostics.
• FAIL_FAST: stop iteration at the first error and surface it via CsvReader.stats.

Lightweight non‑owning view over a CSV field.

Header index providing fast name -> column index lookup.

Built once from a set of header fields (typically the first CSV record when CsvDialect.header == true) and attached to subsequent RowView instances via RowView.attachHeader to enable row.byName("col") without per-row overhead.

Policy for duplicates: the first occurrence wins; hasDuplicates is set to true when the header contains duplicate (normalized) names.

Lightweight non‑owning view over a parsed CSV row.

Error codes describing common CSV parsing and configuration issues.

Structured error information returned in CsvResult!T.

Exception type used by optional throwing helpers.

This exception wraps a CsvError and is thrown by convenience APIs such as CsvResult!T.valueOrThrow() for users who prefer exception‑driven control flow instead of explicit result checking. Hot paths should prefer CsvResult without throwing for performance.

Result container for error‑aware APIs without throwing.

Holds either a value of type T (when isOk is true) or an error. For compile‑time friendliness we store both value and err; users should consult isOk before accessing value.

CSV dialect configuration.

Represents the set of options that control how CSV is parsed and written. Defaults follow RFC 4180: comma delimiter, double‑quote for quoting, CRLF/LF detection on read, RFC‑only escaping, and no header row by default.

Fields (all public):

• delimiter: Single‑byte field separator (default: `,`).
• quote: Quote character (default: `"`).
• doubleQuote: When true (default), two consecutive quotes inside quoted fields represent a literal quote character.
• trimWhitespace: When true, trim leading/trailing whitespace for unquoted fields. Default: false.
• newlinePolicy: Controls record boundary handling and writer emission policy. Default: NewlinePolicy.DETECT.
• escapeStyle: Optional extension for non‑RFC datasets. Default: EscapeStyle.NONE.
• header: When true, the first record is interpreted as a header row.
• acceptUtf8BOM: When true (default), a leading UTF-8 BOM is skipped.

Construction:

• The type can be used with its .init value for RFC defaults, e.g. auto d = CsvDialect.init; and then mutate fields.
• A convenience constructor allows specifying any subset (positional with defaults) while other options fall back to defaults, e.g.: auto d = CsvDialect(';'); // semicolon delimiter

Validation:

• isValid() returns true if options are self‑consistent. At minimum it ensures delimiter != quote.

  Examples

  import ddn.data.csv;
  
  // Customize only the delimiter (semicolon-separated values)
  auto d1 = CsvDialect(';');
  
  // Fully configure a dialect
   auto d2 = CsvDialect(',', '\'', true, false, NewlinePolicy.FORCE_LF);
  assert(d1.isValid && d2.isValid);

High‑throughput CSV reader over an input range of bytes.

Models a forward input range over RowView.

Constructing from in-memory text/bytes (string, char[], const(ubyte)[]) parses eagerly to the first row when available; empty is false until the source is exhausted or an unrecoverable error occurs.

import ddn.data.csv;
// Parse two rows, sum first column lazily (no allocations on hot path)
const csv = "x,y\n10,20\n30,40\n";
long sum = 0;
auto r = byRows(csv);
// Skip header
r.popFront();
while (!r.empty)
{
    auto row = r.front;
    auto a = fromCsv!long(row[0]);
    auto b = fromCsv!long(row[1]);
    if (a.isOk && b.isOk) sum += a.value + b.value;
    r.popFront();
}
assert(sum == 100);

Efficient CSV writer to an output range of bytes.

Implements a buffered writer core that minimizes calls to the underlying sink by batching output into a large internal buffer. The buffer size is configurable via the constructor; a sensible default is used when not specified.

Notes:

• Fields are written as provided. Quoting rules per RFC 4180.
• Newline emission follows dialect.newlinePolicy: NewlinePolicy.DETECT and NewlinePolicy.FORCE_CRLF emit CRLF; NewlinePolicy.FORCE_LF emits LF.

  Examples

  import ddn.data.csv;
  static class Sink { string data; void put(const(char)[] s) { data ~= s; } }
  
  auto s = new Sink();
  auto w = CsvWriter!(Sink)(s); // default RFC dialect
  assert(w.writeRow(["id", "name"]).isOk);
  assert(w.writeRow(["1", "Doe, Jane"]).isOk); // quoted due to comma
  w.flush();
  assert(s.data.startsWith("id,name"));

Package-internal buffer manager that batches read() calls into large contiguous buffers to reduce syscall overhead and improve throughput.

Usage pattern (illustrative):

auto bm = BufferManager!MyReader(reader, 64 * 1024);
for (;;)
{
    auto chunk = bm.nextChunk();
    if (chunk.length == 0) break; // EOF
    // Consume data from `chunk` (do not hold past next call).
    bm.popFront(chunk.length);
}

Notes:

• Buffer size is configurable via the constructor; defaults to 64 KiB.
• nextChunk returns a slice of the internal buffer; its content becomes invalid after the next nextChunk call.
• Metrics readCalls and bytesRead are available for tests/diagnostics.

Package‑internal scan result for a single CSV row.

• row: A RowView referencing the provided buffer slice; valid until the next scan or buffer reuse.
• consumed: Number of bytes consumed from the start of the buffer, including the record terminator (CRLF/LF) when found.
• hasRow: true when a complete row was found; false otherwise (no terminator encountered in the provided chunk per newlinePolicy).

Zero‑copy row scanner that splits one CSV record into fields with dialect-aware quote and optional backslash-escape handling.

Notes:

• Handles CRLF and/or LF per CsvDialect.newlinePolicy.
• Delimiter is taken from CsvDialect.delimiter.
• trimWhitespace is applied to all fields (unquoted fields only are relevant at this stage).
• Produces FieldView slices that reference the input buffer; callers must ensure the buffer remains valid until they finish using the row.
• The scanner maintains an internal fixed array to avoid GC allocations in hot paths. If the number of fields exceeds the fixed capacity, a dynamic array fallback is used for that row (rare); @nogc tests should keep field counts below the fixed capacity to avoid allocations.

Reading statistics and optional diagnostics collected by CsvReader.

Fields:

• rows: number of successfully yielded rows.
• badRows: number of malformed rows encountered.
• errors: total error count (equals badRows in current implementation).
• lastError: the most recent error encountered.
• diagnostics: per-error list (populated only when collectDiagnostics is enabled).

Streaming CSV row reader used by CsvFile.reader().

This reader parses incrementally from either an in-memory byte slice (mapped path) or a file-backed chunk source (buffered path). It keeps only the current carry-over bytes between chunks instead of loading the entire file into memory.

Mode for opening a CsvFile.

Convenience wrapper for reading/writing CSV files.

import ddn.data.csv;
import std.file : remove, exists;

// Write a small CSV file
CsvFile out = CsvFile("./t23_sample.csv", CsvOpenMode.WRITE);
string[][] rows = [["a","b"],["1","x,y"],["2","z"]];
assert(out.writeRows(rows).isOk);
assert(out.close().isOk);

// Read it back (buffered path)
CsvFile f = CsvFile("./t23_sample.csv", CsvOpenMode.READ);
auto rr = f.reader();
size_t cnt = 0; while (!rr.empty) { ++cnt; rr.popFront(); }
assert(cnt == rows.length);
// Cleanup
if (exists("./t23_sample.csv")) remove("./t23_sample.csv");

Maximum file size to attempt memory mapping on 32‑bit platforms (heuristic).

Mapping very large files on 32‑bit OSes can fail due to limited virtual address space. We apply a conservative cap (≈1.5 GB) and fall back to buffered reading beyond this size.

Default delimiter (RFC 4180).

Default quote character (RFC 4180).

Default: interpret doubled quotes inside quoted fields.

Default: do not trim whitespace in unquoted fields.

Default newline policy: detect CRLF and LF on read.

Default escape style: RFC only.

Default: header row absent.

Default: skip a leading UTF-8 BOM when reading.

Trait that evaluates to true when R provides a read(ubyte[]) method returning the number of bytes read (size_t). This matches common D I/O patterns (e.g., file descriptors, custom sources) and enables our buffered reader to minimize read calls.