Fan-out / map over a work list with join, under a concurrency cap
State: open/active · Genre: analysis
Use case: capped-fanout-join RFC mapping: UC6 Source: cross-repo survey of
sksizer/* (2026-07-05), companion to the workflow-constellation RFC (PR #682).
The need
Section titled “The need”Mapping a task over a work list (repos, files, images, matrix cells, VMs) with a join and per-item status is everywhere, and almost always in one of two broken forms: strictly sequential because hand-rolling parallelism was too costly, or unbounded (one bash background job per item, N simultaneous claude agents, no cost cap). Joins are done by PID arrays and grepping per-item log files; per-item failure handling is fail-fast-and-strand or count-and-lose.
Evidence
Section titled “Evidence”rust-ontogen
Section titled “rust-ontogen”Six copy-pasted *_all.sh fleet drivers: clone every repo from downstream.txt/cousins.json, launch per-repo scripts as background subshells with PIDS/REPOS/LOG_FILES arrays, wait-join, count failures, grep PR URLs from logs — no concurrency cap, each item spawning its own agent. Same suite vendored in rust-fluent-git, rust-agent-adapt, rust-markdown, kiosk, rust-evented-worker.
Where: scripts/bring_up_to_date_all.sh · scripts/cargo_update_all.sh ·
scripts/lib/pr_prompt.sh
Pain: ~400 lines of six-times-duplicated PID bookkeeping and log scraping; unbounded parallel agent spawning; a failed repo means rerunning the whole fleet.
DarkFactory
Section titled “DarkFactory”PRD-551 declares concurrency-capped parallel graph execution (—parallel-jobs N) as ‘the clearest value of DarkFactory’ — still draft because a parallel scheduler is expensive to hand-roll; the event log was redesigned just to prepare for it.
Where: .darkfactory/data/prds/PRD-551-parallel-graph-execution.md ·
python/darkfactory/event_log.py
Pain: The flagship feature is blocked on exactly the scheduler @flow/dag ships.
harness
Section titled “harness”shutdown_with_progress: one tokio JoinSet task per VM (emit → timed graceful stop → force fallback → DB update → emit), joined; force_shutdown_all is a second copy of the same fan-out; a 253-line frontend composable consumes the per-item events.
Where: prototypes/zoo/code2/src-tauri/src/vm/manager.rs ·
prototypes/zoo/code2/src-nuxt/app/composables/useShutdown.ts
Pain: UC6 verbatim, duplicated, with a bespoke per-item event schema.
agentpants
Section titled “agentpants”Skill indexer runs its per-skill fetch/parse/scan/score/upsert loop strictly sequentially (two GitHub API calls per skill, one at a time) beside a hand-rolled rate-limit budget; partial-failure resume is impossible because progress is journaled only at run granularity.
Where: src-indexer/src/orchestrator.ts · src-indexer/src/strategies/rate-limit.ts
Pain: Sequential-for-lack-of-a-cap; re-running only errored items impossible.
polish
Section titled “polish”LLM benchmark sweep: four nested for-loops over a model×mode×prompt×repeat matrix of slow live LLM calls, strictly serial, appending JSONL rows — an interrupted sweep re-runs every cell.
Where: crates/vaultpolish-agent/examples/bench_dispatch.rs
Pain: A capped fan-out with journal/resume would parallelize and make interruption cheap.
The orchestrator tick fans out one sub-agent per open PR concurrently and dispatches task-work up to per-project caps (max_implementations), with the cap accounting and join performed by the parent LLM following markdown instructions; multiple skills explicitly ask for ‘a pipeline/parallel fan-out of one agent per section via the Workflow tool’.
Where: plugin/skills/orchestrate/SKILL.md · plugin/skills/api-enhancement-scan/SKILL.md ·
plugin/skills/principle-review/SKILL.md
Pain: Concurrency-capped fan-out with join simulated by an LLM every tick; the demand for the library shape is written down verbatim.
Flowline fit
Section titled “Flowline fit”@flow/dag’s concurrency-capped fan-out with join is the exact shape. The cap is the load-bearing
feature: it is what every hand-rolled version lacks (unbounded) or was defeated by (sequential).
Typed per-item results must replace log-file scraping as the join channel, and per-item
continue-on-error with an aggregated report must replace fail-fast ? loops (monosync) and
count-and-lose batches (agent-dashboard).
Proposed API
Section titled “Proposed API”Builds on: 03 Introduces: fanOut()
Map a task (or a whole nested subloop) over a runtime-derived work list, under the concurrency
cap every script rebuilds with xargs -P, semaphores, or worker pools:
import { fanOut } from "flowline/dag";
const verifyAll = fanOut({ over: (ctx) => ctx.read(Findings).items, // the work list, derived at runtime do: (f) => verifyFinding(f), // any task / L2 subloop per item concurrency: 4, join: (rs) => Report.of({ results: rs }), // one barrier, one typed join});