License
BSD-3-Clause
ddn.columnar.parallel
The parallel-execution substrate of ddn.df (ADR-2026-10-19-02).
Owned std.parallelism.TaskPool generations, sized by the DDN_DF_MAX_THREADS environment variable (default: every core, clamped to the core count) or, at runtime, by setThreads (ADR-2026-10-23-05). Pools are created lazily, so single-threaded runs (DDN_DF_MAX_THREADS=1) and work below the row threshold never pay for threads they never get. A size-changing setThreads installs a new generation immediately: dispatches hold use-counted pool references, in-flight operations finish on their pool bit-identically, and a retired pool stops when its last user releases it - a control call never blocks on operations.
Determinism rule (the keystone of ADR-2026-10-19-02): parallel paths must produce bitwise-identical results to the serial path regardless of thread count or scheduling. Every parallel operation is shaped so the answer never depends on the scheduling - workers write disjoint result slots over immutable input, and reductions combine sequentially in chunk-index order on the calling thread. D's thread-local-by-default globals and implicitly shareable immutable chunks are what make this race-free by construction.
Copyright
Copyright (c) 2026, DDN (D Developer Network) Members
var cgroupReadFile generations_ guard_ MIN_ROWS_PARALLEL pool_ poolUses_ retired_ threads_ THREADS_ENV
struct RetiredPool
Module Initializers 1
shared static this
()Types 1
Functions 10
fn
void setThreads(size_t n) @trustedSets the process-wide `ddn.df` thread count for subsequent dispatches (ADR-2026-10-23-05).fn
size_t threads() @trustedThe effective `ddn.df` thread count: the policy a subsequent dispatch would use - `setThreads`' value when it has spoken, else the environment variable's, else the derived default (ADR-2026-10-23-0...fn
TaskPool acquirePool() @trustedAcquires the current pool generation for one dispatch: the pool is created lazily at `dfThreads()` and the caller's use is counted, so a concurrent `setThreads` can retire this generation without s...fn
void releasePool(TaskPool pool) @trustedReleases one dispatch's use of `pool`. When the last user of a retired generation releases it, the pool stops there and then - outside the guard (stop joins its worker threads, which are idle for t...fn
bool shouldParallel(size_t nChunks, size_t totalRows)Whether a chunked operation should take the parallel path: at least two chunks, at least `MIN_ROWS_PARALLEL` rows total, and more than one thread. The result of the operation never depends on this ...fn
void runSegments(size_t n, size_t totalRows,
scope void delegate(size_t) compute) @trustedRuns `compute(i)` for every independent unit, in parallel above the row threshold and serially below. This is the determinism-preserving pattern of ADR-2026-10-19-02: each call writes only its own ...fn
void overrideThreadsForTesting(size_t n)fn
size_t poolGenerationsForTesting()The number of pool generations installed so far (test evidence for one canonical executor across package boundaries).Variables 9
enumvar
THREADS_ENV = "DDN_DF_MAX_THREADS"The environment variable capping the ddn.df thread pool: a positive integer, clamped to the core count, read once on first use.
enumvar
MIN_ROWS_PARALLEL = 65_536Total rows below which a chunked operation stays single-threaded (ADR-2026-10-19-02, the serial-below-threshold rule): a handful of default chunks cannot feed every core, and small operations must not pay pool dispatch.
private var
Object guard_private var
size_t threads_private var
size_t poolUses_private var
size_t generations_var
string delegate(const(char)[] path) cgroupReadFile