B-QA7F-needs-definition-auto-fix-vs-human-triage

Status: promoted/task

Problem

task-ensure-ready currently emits a single ENSURE-READY-NEEDS-DEFINITION: verdict for any failure of the implementation-ready contract. Today’s orchestrator sessions surfaced that the failure space is heterogeneous — some failures are mechanical and obviously fixable, others require human judgment about the task’s intent. Treating them uniformly dumps every NEEDS-DEFINITION into a human-triage queue, even when the fix is deterministic and the task could otherwise proceed autonomously.

The pattern matters for the autonomous-orchestrator promise: every NEEDS-DEFINITION return removes a task from the dispatch pool until a human comes back and triages it. If half of those returns are mechanical (renamed paths, wrong Kind column, formatting), the loop’s effective throughput is bottlenecked on a human bandwidth that’s not actually needed.

Observed failure classes (2026-05-28 session)

Class	Example	Risk of auto-fix
Format / parser error	”row combines two files in one Location cell” — the touchpoint parser fails on the combined string	Very low — deterministic split
Kind mismatch	Files-to-touch row marked `modify` on a path that doesn’t exist (should be `new`)	Very low — reality contradicts the cell, flip it
Renamed / relocated path	`plugin/scripts/lease_cli/` cited; codebase moved it to `plugin/cli/lease_cli/`	Low if `git log --follow <old-path>` yields exactly one rename target; medium otherwise
Missing-path misname	`plugin/skills/new-skill/` cited; never existed (commit message misled the task author)	High — multiple plausible replacements (use existing `skill-creator`? create a new `skill-refine`? abandon as obsolete?)
Materially-out-of-date framing	lease-library v1→v2 task assumes a JSON-schema-versioned architecture; the actual codebase uses Pydantic models with no JSON schema at all	Very high — full rewrite changes the contract; auto-rewriting would silently retarget the implementation
Unresolved TBD placeholder	Approach step 4 contains a literal “TBD” the author forgot to fill	Low — the gap is named, but the fix needs human input on what to put there

The first three are deterministic-or-near-deterministic. The last three need human judgment. Today’s flow treats them identically.

Proposed fixes (assortment — not commitments)

A. Classify-and-escalate

Add a classifier between ensure-ready’s failure and the --cleanup-on-fail path. The classifier reads the definition_gap text + the task body and emits one of auto-fixable:<class> / human-required:<reason>. Only the auto-fixable classes attempt repair; everything else flows to the existing NEEDS-DEFINITION cleanup + human triage.

The classifier itself could be:

A rules engine matching gap text patterns (“non-existent path” / “Kind mismatch” / etc.). Pure regex over the gap text.
An LLM call with strict structured output. More flexible but reintroduces the LLM-improvisation risk we just spent today fixing.
A small Pydantic model + a hand-written matcher. Probably the right size.

B. Per-class auto-fixers (script tier)

Each opt-in auto-fix class is a small deterministic script. Sketch of the three lowest-risk:

Class	Script	Logic
`cell-format`	`plugin/skills/task-ensure-ready/fix_cell_format.py`	Split combined Location cells; backtick known-code patterns (`<basename>` placeholders); normalize markdown table whitespace
`kind-mismatch`	`plugin/skills/task-ensure-ready/fix_kind_mismatch.py`	For each Files-to-touch row where path doesn’t exist + Kind=modify, change Kind to `new`. Inverse for paths that exist + Kind=new
`renamed-path`	`plugin/skills/task-ensure-ready/fix_renamed_path.py`	For each Today row whose path doesn’t resolve, run `git log --follow --diff-filter=R <cited-path>`. If exactly one rename target and it resolves today, suggest the substitution. Commit the change only if `--auto-renames` is set; otherwise emit a structured suggestion and exit non-zero

Each script returns a structured “what would change” preview the orchestrator can log before committing.

C. sdlc.yaml config gate

Opt-in per project, classes individually configurable:

auto_define:
  enable: true                  # global on/off
  classes:
    - cell-format               # safe
    - kind-mismatch             # safe
    - renamed-path-unambiguous  # safe-ish (only if git log yields one target)
  max_attempts: 1               # one shot; if re-ensure-ready still fails, escalate
  preview_only: false           # if true, print the would-be diff but don't commit

Defaults enable: false. Conservative projects stay opted-out; aggressive projects can opt into specific classes. The max_attempts cap prevents auto-fix loops where the fixer keeps producing edits that don’t satisfy ensure-ready.

D. Surface gap to the operator inline (lighter alternative to A+B+C)

Instead of building a full auto-fix tier, keep the existing NEEDS-DEFINITION flow but enrich the definition_gap text with a “suggested fix” line per known class. The orchestrator log gets a more actionable summary; the human triaging can apply the suggestion with a copy-paste rather than diagnosing from scratch. Cheaper to build, no risk of wrong auto-fix, but doesn’t reduce human-triage load.

E. Splitting NEEDS-DEFINITION into multiple verdicts at the protocol level

Change the marker shape so the orchestrator branches on the failure class:

ENSURE-READY-NEEDS-DEFINITION-MECHANICAL: <basename>
  classes: cell-format,kind-mismatch
  auto_fixable: true

ENSURE-READY-NEEDS-DEFINITION-SEMANTIC: <basename>
  reason: framing-out-of-date
  auto_fixable: false

The orchestrator’s dispatch logic decides per verdict whether to attempt auto-fix-then-re-dispatch or surface to human. More protocol surface area; cleaner separation than option A which classifies in-script.

F. `/sdlc:task-define --auto` mode

Don’t introduce a new skill. Add a --auto flag to the existing /sdlc:task-define that:

Reads the definition_gap from ensure-ready’s last fail
Attempts the auto-fixable classes
Re-runs ensure-ready
Returns either TASK-DEFINE-AUTO-FIXED (re-ensure-ready passes) or TASK-DEFINE-HUMAN-REQUIRED (still failing, hand off to interactive task-define or operator)

task-work’s Step 5a NEEDS-DEFINITION path optionally invokes this before cleanup-on-fail.

Cross-cutting considerations

Audit trail: every auto-fix should commit on main with a distinct subject (e.g. chore(tasks): auto-fix <class> for <basename>) so the human can git log --grep to see what the auto-fixer changed without their review. Same pattern as PR #167 established for status flips.
Test fixtures: each auto-fix class needs a canonical input → output fixture. Failure regressions are catastrophic (silently changes task contracts).
Backstop: even with auto-fix enabled, the FIRST 3-5 auto-fixes on each project should be surfaced to the operator for confirmation before silently applying. After confidence builds, switch to silent.
What about Step 2’s relevance check? The lease-library case today wasn’t caught by ensure-ready — it was caught by task-work’s Step 2 LLM-driven relevance check. Either fold that check into ensure-ready (so all NEEDS-DEFINITION paths run through one gate) OR extend the classify-and-escalate pattern to Step 2 as well. Worth resolving before designing the full auto-fix tier.

Discovery context

Surfaced 2026-05-28 after the post-#167 / post-#169 orchestrator run pattern was stable enough to expose the next reliability layer. Of 6 task-work dispatches in the final session, 5 hit NEEDS-DEFINITION; of those 5, 4 had failure classes that map to the deterministic side of the table above. The human-triage queue is filling with work that scripts could resolve.

← Back to Backlog