Durable orchestration: journaling, idempotent re-entry, crash recovery

State: open/active · Genre: analysis

Use case: durable-journaled-resume RFC mapping: UC8 Source: cross-repo survey of sksizer/* (2026-07-05), companion to the workflow-constellation RFC (PR #682).

The need

Journaling + resume is the most expensively re-attempted capability in the survey: two Rust repos are literally abandoned from-scratch event-sourced workflow engines, one production engine needed a full rewrite because v1 kept execution state in local variables, and mature systems scatter durability across four or five parallel mechanisms — event logs that are written but never replayed, resume-safety predicates, cursor files, marker files, advisory locks, and reconcile-on-startup sweeps.

Evidence

rust-evented-worker

The entire crate is a hand-rolled event-sourced engine (ActivityEvent log as source of truth, fold-based restore, scheduler/controller loop) whose own defect docs record: events never persisted to disk, reducer panics on invariant violations, async unimplemented!.

Where: src/runner/controller.rs · src/runner/restore.rs · docs/todo.md

Pain: The whole @flow/durable value proposition attempted from scratch and never finished.

rust-worker-prototype

prototype_three: 1000-line single-file durable engine — EventEnvelope journal with intent/outcome correlation, fold recovery, snapshots, idempotent step skip on re-run, side-effect verification — the author’s third restart at the same library.

Where: src/bin/prototype_three.rs · src/event_store.rs · docs/todo.md

Pain: Three attempts at the same substrate; strongest possible prior-art evidence of the need.

DarkFactory

Durability hand-rolled four independent ways: per-PRD JSONL event journal (written, never replayed), is_resume_safe worktree re-entry classification, per-branch FileLock, and merged-PR status reconciliation duplicated in Python and as sed in a GitHub Action.

Where: python/darkfactory/event_log.py · python/darkfactory/checks.py · python/darkfactory/cli/reconcile.py · .github/workflows/prd-status-on-merge.yml

Pain: Four mechanisms where one journaled-run substrate should exist; the CI sed script is a workflow step that outgrew its medium.

dev

A full durability layer over git refs: CAS lease primitives with fencing tokens, TTL/2 heartbeat loops that a SKILL.md instructs the LLM to background via PID files, atomic handoff.md blobs for cross-session resume, plus five parallel gitignored state stores (PR cursors, quality baselines, watch state, tick journal) — heavy enough that a dedicated migration skill exists just to relocate them. task-work’s resume is a five-condition forensic predicate over commits/worktrees/branches/PR state surfaced to a human.

Where: plugin/lib/services/lease/README.md · plugin/lib/services/lease/heartbeat-service.ts · plugin/skills/task-work/SKILL.md · plugin/skills/migrate-runtime-state/SKILL.md

Pain: The sharpest version: markdown instructing an LLM to maintain a heartbeat process because no runtime owns step liveness, and resume-by-forensics instead of resume-by-journal-replay.

ontological

Persisted step cursor (current_step_index/attempt/failure_context) written after every step, startup recovery flipping orphaned Running executions to Suspended, suspend-on-shutdown — explicitly framed as ‘a pragmatic middle ground’ after Phase 4.5 rewrote v1, which lost all state on crash.

Where: src-tauri/src/engine/mod.rs · src-tauri/src/engine/session.rs · docs/design-work-execution.md

Pain: One full rewrite already paid; the open questions section wrestles with re-run-vs-skip — the idempotent re-entry problem a library should answer.

ralph

File-based durability by convention: prd.json ledger, append-only progress.txt journal, .last-branch cursor, archive rotation — with the checkpoint writes delegated to the LLM itself, so nothing guarantees work and ledger update land together.

Where: ralph.sh · prd.json.example · README.md

Pain: Journaling and idempotent re-entry reimplemented in bash + jq + prompt contract.

Flowline fit

@flow/durable (journaling + resume) is the shaped hole. Adjacent needs the evidence surfaces that the current design should either cover or explicitly scope out: mutual exclusion for concurrent runners (file locks, git-ref CAS leases with fencing), liveness (heartbeats/TTL so a crashed run’s claim expires), and reconcile-against-external-reality on startup (zoo’s reconcile_state, DarkFactory’s PR-status reconciliation) — the journal alone answers none of these.

Proposed API

Builds on: 11 Introduces: runDurable(), fileJournal()

Journal each completed step’s validated output; re-invoking with the same runId replays completed steps instead of re-running them:

import { runDurable, fileJournal } from "flowline/durable";

const result = await runDurable(g, {
  runId: "task-T-4WR3",
  journal: fileJournal(".sdlc/runs"),   // append-only step-output log
  concurrency: 8,
});
// kill -9 mid-run, re-invoke with the same runId: completed steps replay from
// the journal, the in-flight step re-runs as an idempotent re-attempt.

Boundary the evidence forces the RFC to name: the lease/heartbeat/fencing layer (this repo’s git-ref leases, DarkFactory’s FileLocks, zoo’s reconcile) is cross-process ownership and stays outside; the journal is intra-run dataflow. They compose; neither substitutes for the other.

---

← Back to Notes