Bounded retry with backoff and a recovery/fallback step

State: open/active · Genre: analysis

Use case: bounded-retry-with-recovery RFC mapping: UC3 Source: cross-repo survey of sksizer/* (2026-07-05), companion to the workflow-constellation RFC (PR #682).

The need

Attempt-counted retry loops with sleeps, timeout wrapping, and a recovery branch (graceful-then-force, reset-and-reapply, restart-with-backoff) are inlined by hand at every occurrence. Half-finished versions are common: backoff fields defined but never used, MAX_RETRIES declared but never read, retry-without-recovery that just re-runs the same failing thing.

Evidence

dev

commitToMainViaWorktree: 5-attempt loop classifying recoverable push rejects, then reset-to-fresh-tip, re-apply the idempotent mutation, re-commit, re-push — shared by three callers.

Where: plugin/lib/services/git/commit-to-main.ts

Pain: Recoverable-error classification and the reapply recovery are hand-coded inside a for-loop instead of a combinator.

harness

wait_for_ssh: 60-attempt 1s-interval probe loop with per-attempt progress events; shutdown paths hand-roll graceful stop under a 30s timeout with force-stop fallback folded into nested match arms.

Where: prototypes/zoo/code2/src-tauri/src/vm/provision.rs · prototypes/zoo/code2/src-tauri/src/vm/manager.rs

Pain: Each occurrence re-rolls attempt counting, sleeps, timeout wrapping, and fallback branching with its own ad-hoc progress emission.

deprecated-agent-visualizer

411-line sidecar supervisor: health-poll gate (200ms/10s deadline), bounded restart loop (max 3, 5s backoff), PID-file lease with SIGTERM→SIGKILL escalation, two-phase graceful shutdown.

Where: src-tauri/src/server_manager.rs · docs/vision.md

Pain: A textbook retry-with-recovery loop hand-coded with Arc state and sleeps.

2b

Bespoke RetryStrategy class (exponential backoff, max delay, jitter, reset-on-success) driving a Discord reconnect loop in a systemd service.

Where: sb_agent_service/src/sb_agent_service/service.py

Pain: A reusable-looking policy class that exists in exactly one repo and one call site.

rust-worker-prototype

prototype_three’s per-step RetryPolicy{max_attempts, backoff_ms} drives a hand-rolled attempt loop — but backoff_ms is defined and never used (the loop has no sleep).

Where: src/bin/prototype_three.rs

Pain: Exactly the gap a library-provided retry policy would not have; rust-evented-worker similarly got as far as counting attempts with no recovery hook.

agentpants

withRateLimitRetry: hand-rolled exponential backoff (1s→60s, max 5) on 403/429 for the GitHub indexer.

Where: src-indexer/src/strategies/rate-limit.ts

Pain: UC3 verbatim, re-implemented as a local helper.

Flowline fit

@flow/workflow retry combinator. The evidence says the combinator must include the recovery/fixer/compensation step and a real backoff policy as first-class parameters — bare bounded re-run is the version everyone builds first and it is never enough (rust-evented-worker’s own docs: ‘No retry or compensation API: failure is terminal’).

Proposed API

Builds on: 03 Introduces: onFailure: { task, retry }

DarkFactory’s onFailure, ported: on step failure run a recovery task, then re-try the failed step, bounded:

const wf = workflow("publish", "build, check, upload")
  .seed(Source.of(src))
  .add(build)
  .add(qualityCheck, { onFailure: { task: agentFix, retry: 3 } })
  // agentFix reads the failed QualityResult from the blackboard, edits, and
  // qualityCheck re-runs — at most 3 attempts, then a typed failure.
  .add(upload, { onFailure: { retry: { attempts: 5, backoff: "exponential" } } })
  .build();

The backoff policy object is an extension beyond the RFC sketch — harness and 2b both hand-roll exponential backoff around network steps.

---

← Back to Notes