License
BSD-3-Clause
ddn.df.deferred
The deferred frame layer of ddn.df (ADR-2026-10-24-16, amended 2026-10-29): deferred execution over one inspectable plan-node tree. LazyFrame wraps an eager DataFrame (the in-memory source leaf); .filter/.select/.withColumn/.sort/.limit build plan nodes without executing; .collect() runs the tree through the eager kernels (an unoptimized tree walk); .schema() derives the output schema from metadata alone. The eager chain IS the correctness oracle: collect() must produce bitwise-identical output to the equivalent eager sequence, and repeated collections are equal.
One representation drives collect, explain, streaming and the optimizer. A source leaf is either an in-memory frame or a query-owned ScanSource; concrete CSV and Arrow IPC scan sources live in the IO package, while acquisition, projection and lifetime rules stay with this query layer.
Streaming (ADR-2026-10-24-18): .stream(batchSize) replays the plan per batch with bounded memory; blocking operations (sort, window expressions) fall back to the eager chain - disclosed by streamMode(), never silent.
Copyright
Copyright (c) 2026, DDN (D Developer Network) Members
enum PlanKind
iface ScanSource
Types 6
interfaceScanSource
A lazy batch source (ADR-2026-10-29-03): the leaf a SCAN node pulls through. nextBatch() returns batches in source order (Nullable null at exhaustion); close() releases the source deterministically (scope exit, exception or exhaustion; idempotent; the GC is only a backstop). description() feeds explain().
Methods
bool supportsProjection() @safeWhether the source can prune columns at the read (the projection-pushdown capability; ADR-2026-10-29-03).ScanSource project(scope string[] columns) @safeA source reading only `columns` (a proper subset of the schema's names); the error policy is the source's own (the approved rule: only error-free column reads may be skipped).ScanSource acquire() @safeAcquires the source for one collection: the first acquisition may return this live source, later ones return fresh iterators (rewindable sources) or throw DataFrameException (a non-rewindable sourc...enumPlanKind
The kind of one plan node.
SOURCEthe in-memory frame leaf
SCANthe lazy batch-source leaf (ADR-2026-10-29-03)
FILTERrow-wise: keep rows by a BOOLEAN expression
PROJECTrow-wise: project to expressions
WITH_COLUMNrow-wise: add/replace a column from an expression
SORTblocking: the global stable ordering
LIMITearly-exit: truncate to the first n rows
AGGREGATEblocking: group by keys and aggregate
JOINblocking: join two plans on a key
CROSSblocking: the Cartesian product (no keys)
structPlanNode
One node of the logical plan tree. A chain of single-child nodes (the only shape the row-wise builders produce; grouping and joins arrive as blocking multi-input nodes with the lazy query surface). The factories are package-visible so the optimizer can rebuild trees; users inspect through LazyFrame.explain and .schema.
Fields
private PlanKind _kindprivate DataFrame _sourceprivate ScanSource _scanprivate PlanNode[] _childrenprivate Expr[] _exprsprivate string _nameprivate bool _descendingprivate bool _nullsLastprivate size_t _nprivate string[] _keysprivate AggSpec[] _specsprivate JoinKind _joinKindprivate string[] _leftOnsprivate string[] _rightOnsprivate string _suffixMethods
Expr[] exprs() @safe pure nothrowThe node's expressions (FILTER's predicate, PROJECT's projections, WITH_COLUMN's value).PlanNode source(DataFrame df) static @safe pure nothrowBuilds a source leaf over an eager frame (shared, immutable).PlanNode scanNode(ScanSource src) static @safe pure nothrowBuilds a lazy batch-source leaf (ADR-2026-10-29-03).PlanNode filter(PlanNode child, Expr pred) static @safe pure nothrowBuilds a filter node over `child`.PlanNode project(PlanNode child, Expr[] exprs) static @safe pure nothrowBuilds a projection node over `child`.PlanNode withColumn(PlanNode child,
string name, Expr e) static @safe pure nothrowBuilds a withColumn node over `child`.PlanNode sortNode(PlanNode child, string key,
bool descending, bool nullsLast) static @safe pure nothrowBuilds a sort node over `child`.PlanNode limitNode(PlanNode child, size_t n) static @safe pure nothrowBuilds a limit node over `child`.PlanNode aggregateNode(PlanNode child,
string[] keys, AggSpec[] specs) static @safe pure nothrowBuilds a blocking aggregation node: group `child` by `keys` and aggregate per `specs`.PlanNode joinNode(PlanNode left,
PlanNode right, string[] leftOn, string[] rightOn,
JoinKind kind, string suffix) static @safe pure nothrowBuilds a blocking join node over the left and right plans.PlanNode withChildren(PlanNode[] children) @safe pure nothrowRebuilds this node with new children (the payload is copied).structLazyFrame
A deferred frame: a deferred computation over a plan tree. Build with df.deferred(), compose with .filter(expr), .select(exprs...), .withColumn(name, expr), .sort(by), .limit(n), then collect with .collect() or stream with .stream(batchSize). Nothing executes until collect or stream.
Fields
private PlanNode _rootMethods
LazyFrame fromPlan(PlanNode root) static @safe pure nothrowWraps a plan root as a deferred frame (the scan module's entry).DataFrame collect(bool optimize = true) @trustedCollects the deferred frame: optimizes the plan (bounded, proved-safe rewrites), then walks the tree through the eager kernels. The result equals both the unoptimized collect and the equivalent eag...string explain(bool optimize = true) @safePrints the plan: a rendering of the plan tree (application order) plus the streaming mode (ADR-2026-10-24-18: fallbacks are disclosed, never silent). The optimized view annotates which rewrite rule...string streamMode() @safeThe streaming mode of this plan (ADR-2026-10-24-18): "streamed: row-wise chain" when every operation is row-wise or an early-exit limit; "fallback: materialising (<op>)" for blocking operations (so...ColumnSchema[] schema() @safeThe plan's output schema, derived from metadata alone: the source's columns reshaped by each node (select projects the expressions' resolved dtypes under their labels; withColumn adds or replaces o...LazyFrame withColumn(string name, Expr e) @safeAdds or replaces a column from an expression (deferred).LazyFrame sort(string by, bool descending = false,
bool nullsLast = false) @safeSorts by a column (deferred).LazyGroupBy groupBy(string[] keys...) @safeGroups the deferred frame by key columns for aggregation (the deferred form of the eager `DataFrame.groupBy`): first- appearance group order, null keys forming their own group. Nothing executes unt...LazyFrame join(LazyFrame other, string on,
JoinKind kind = JoinKind.INNER, string suffix = "_right") @safeJoins this plan with another deferred frame on one key column present in both (the deferred form of the eager `DataFrame .join`; both children materialise, the join runs eagerly, the emission order...LazyFrame join(LazyFrame other, string leftOn, string rightOn,
JoinKind kind = JoinKind.INNER, string suffix = "_right") @safeDitto, with different key column names on the two sides.LazyFrame join(LazyFrame other, string[] leftOn, string[] rightOn,
JoinKind kind = JoinKind.INNER, string suffix = "_right") @safeDitto, the multi-column join (ADR-2026-10-30-03).LazyFrame join(LazyFrame other, string[] on,
JoinKind kind = JoinKind.INNER, string suffix = "_right") @safeDitto, the same key names on both sides.LazyFrame crossJoin(LazyFrame other, string suffix = "_right") @safeThe cross join (ADR-2026-10-30-03): the left-major nested loop, blocking (materialising fallback disclosed by `streamMode()`); the row-count check runs at collection.LazyFrame crossJoin(DataFrame other, string suffix = "_right") @safeDitto, joining an eager frame (wrapped as a source plan).LazyFrame join(DataFrame other, string on,
JoinKind kind = JoinKind.INNER, string suffix = "_right") @safeDitto, joining an eager frame (wrapped as a source plan).LazyFrame join(DataFrame other, string leftOn, string rightOn,
JoinKind kind = JoinKind.INNER, string suffix = "_right") @safeDitto, with different key column names on the two sides.DataFrame stream(size_t batchSize = 65_536) @trustedStreams the plan with bounded memory (ADR-2026-10-24-18): the source is pulled in fixed-size batches; row-wise operations (filter/select/withColumn) run per batch; the first limit early-exits the p...void forEachBatch(scope bool delegate(ref DataFrame) consumer,
size_t batchSize = 65_536) @trustedConsumes the deferred frame batch-by-batch without accumulating the output: each pulled batch (the row-wise prefix applied, the limit bound honored) is handed to `consumer`; returning `false` stops...structLazyGroupBy
A deferred grouping: .groupBy(keys) on a LazyFrame, aggregated by .agg(...). The aggregations are the eager AggSpec surface (including collect), or aggregate expressions (col("v").sum(), optionally .renamed) converted to their specifications - so the lazy output names its columns exactly like the eager GroupBy.agg (the source column's name, or the alias).
Methods
structExecContext
The per-collection execution context: the plan walker with the shared-subplan memo. Structurally identical deterministic subtrees evaluate once - the first evaluation in walk order owns the result and its error, later occurrences are served from the memo (bitwise-identical by construction: equal subtrees over the same immutable source storage feed deterministic eager kernels). The memo lives for one collection only: a context is created per collect call and stored nowhere, so no result can leak across collections. The counters are the execution-count evidence the optimizer's sharedSubplans prediction must match.
Functions 12
fn
bool planEq(PlanNode a, PlanNode b) @safe pureStructural equality of two plan subtrees: same node kind, same payload (expressions compared with `exprEq`, aggregation specifications field-wise) and recursively equal children. Source leaves comp...fn
size_t sharedSubplanCount(PlanNode root) @safe pureThe number of subtree evaluations one collection saves through shared-subplan reuse: every non-source subtree occurrence beyond the first of its structure. The executor's per-collection memo (see `...