T-F8BP-dogfood-baseline-smoke-test

Status: closed/done · Impact: medium · Complexity: small

Auto-generated from a /sdlc:task-work post-mortem. Review and promote to open/ready before picking up.

Goal

A task that ships a new quality-check verb or extends plugin/scripts/run_quality_checks.ts needs a cheap pre-PR sanity check that the verb behaves correctly against origin/main’s tree before the sub-agent declares done. Today the only available signal is the full quality-check gate at Step 7, which (a) runs late, after every Edit, and (b) doesn’t exercise the new verb’s end-to-end flow against unmodified baseline content — so authoring bugs that only surface against chatty real-world output get caught only after the PR is open. Surfaced by T-H69K-run-quality-checks-isolates-pre-existing-drift: the v1 “capture every stdout line as a finding” rule didn’t survive contact with chatty passing verbs (test runners, audit count summaries) and was only caught during ad-hoc post-implementation dogfooding.

Today

Location	Role today
`plugin/scripts/run_quality_checks.ts`	The runner the task-work gate invokes; new verbs / flags land here without a dedicated smoke-test affordance
`plugin/scripts/quality_baseline.ts`	Capture/diff/prune helpers shipped by the originating task; no in-repo dogfood-it-end-to-end script
`plugin/skills/task-work/SKILL.md` Step 7	Runs the full gate against the worktree at end-of-task; this is the only built-in dogfood and it’s too late to catch authoring bugs cheaply
(none)	No `plugin/scripts/dogfood_baseline.sh` (or equivalent) exists today; the implementer of the originating task ran the capture-then-gate sequence ad-hoc

The originating post-mortem made the gap concrete: a single-line semantic refinement (only capture findings from verbs that exited non-zero) would have been caught in seconds by a pre-commit dogfood eval, but instead surfaced via manual capture-then-gate experimentation after the sub-agent had already declared the task done. The friction shape is: authoring a verb’s stdout shape correctly requires running it against real-world chatty output, but no scripted affordance exists to do that quickly.

Proposed

Add a small dogfood smoke-test script that runs the capture-then-gate sequence against the current worktree’s origin/main SHA and reports delta — false-positive new-drift: lines (the verb is too eager) and spammy pre-existing: lines (the verb’s output is too chatty to gate on usefully) are surfaced as smoke-test failures even when the formal gate would pass. The implementer of a quality-check verb runs this script before opening the PR; if the smoke-test fails, they iterate on the verb’s output shape until it’s clean.

The script lives at plugin/scripts/dogfood_baseline.sh (or .py — implementer’s call), is referenced from plugin/skills/task-work/SKILL.md under a new “Pre-PR dogfood (when verb authoring is in scope)” subsection, and is invoked manually — not gated. The gate stays at Step 7.

Approach

Write plugin/scripts/dogfood_baseline.sh that:
- Captures a fresh baseline against origin/main’s tree (checkout, quality_baseline.capture(...), return to feature branch).
- Runs bun run run_quality_checks.ts --diff-against-baseline <sha> against HEAD.
- Surfaces both pre-existing: and new-drift: lines with a summary line (“X pre-existing, Y new”); exits non-zero only if Y
  
  0 (false positives) OR if X is suspiciously high (heuristic: more than 50 lines, configurable via --max-pre-existing N).
Document the script in plugin/skills/task-work/SKILL.md under a “Pre-PR dogfood” subsection of Step 7 (or a new Step 6.5), explicitly scoped to tasks whose Files to touch include plugin/scripts/run_quality_checks.ts or any verb under plugin/scripts/.
Add an eval case at plugin/scripts/tests/dogfood_baseline.test.ts (a Bun test, the house convention bun test discovers — landed here instead of the originally-proposed test_dogfood_baseline.sh so the eval actually runs in the gate rather than rotting unexercised) that builds an ephemeral two-commit git fixture and asserts the summary-line shape.

Files to touch

Location	Kind	Change
`plugin/scripts/dogfood_baseline.sh`	new	The smoke-test script described above
`plugin/skills/task-work/SKILL.md`	modify	Document the script under a “Pre-PR dogfood” subsection
`plugin/scripts/tests/dogfood_baseline.test.ts`	new	Bun eval against a constructed git fixture (house convention; supersedes the proposed `test_dogfood_baseline.sh` so the gate runs it)

Acceptance criteria

AC-1: plugin/scripts/dogfood_baseline.sh exists, is executable, and runs the capture-then-gate sequence end-to-end against the current worktree’s origin/main SHA without manual operator input.
AC-2: The script exits non-zero when HEAD introduces a finding absent from the baseline; exits zero when HEAD’s findings are a subset of the baseline.
AC-3: plugin/skills/task-work/SKILL.md documents the script in a named subsection scoped to verb-authoring tasks, with an example invocation that exists on disk.
AC-4: plugin/scripts/tests/dogfood_baseline.test.ts exercises the script against a constructed git fixture and asserts the summary-line shape; the test passes. (Landed as a Bun .test.ts — the house convention bun test discovers — rather than the originally-proposed test_dogfood_baseline.sh, which no gate would have run.)

Out of scope

Making the dogfood smoke-test mandatory at Step 7. The formal gate stays the gate; this is an authoring affordance, not a second gate.
Generalising to non-quality-check verbs. The motivating case is run_quality_checks.ts-shaped tooling; other scripts can adopt the shape later if they show the same authoring friction.

Dependencies

none

Discovery context

Spawned by /sdlc:task-work post-mortem of T-H69K-run-quality-checks-isolates-pre-existing-drift on 2026-05-23.

Dedup search (spawn-from-post-mortem)

Bullet: The v1 design’s capture every stdout line as a finding rule didn’t survive contact with chatty passing verbs (test runners, the audit_skill_runtime count summary). The dogfood caught it, but the spec authoring should have predicted it. A pre-commit dogfood eval (capture-then-gate against origin/main before opening the PR) would have caught the issue without a sub-agent round trip. Worth adding: a plugin/scripts/dogfood_baseline.sh (or similar) that the implementer runs locally as a smoke test before declaring done. Keywords searched: audit_skill_runtime, capture-then-gate, dogfood_baseline, pre-commit, authoring, predicted, sub-agent, declaring Excluded: 2026-05-21-run-quality-checks-isolates-pre-existing-drift Top candidates (score / status / headline):

33 / planning/draft / 2026-05-23-subagent-invariant-preflight-lint — Run skill-prose invariant linter inline during sub-agent edits to catch forbidden-phrase violations pre-flight
22 / closed/done / 2026-05-20-task-work-sub-agent-verdict-contract-clarity — Tighten task-work sub-agent verdict contract so ensure-ready’s READY marker isn’t mistaken for task-work’s final verdict
15 / planning/draft / 2026-05-21-orchestrated-sub-agent-design-call-gap — Surface design questions before orchestrated /sdlc:task-work commits to an approach
13 / closed/done / 2026-05-21-worktree-scope-guard-pre-commit — Pre-commit guard rejects edits to active-worktree-owned paths on main
12 / closed/done / 2026-05-21-skill-md-runtime-drift-audit — Audit /sdlc:task-work SKILL.md against per-project runtime conventions Decision: SPAWNED Rationale: Override script’s LINKED-EXISTING → subagent-invariant-preflight-lint. That task is about a prose invariant linter (forbidden-phrase patterns in SKILL.md), not about smoke-testing newly-authored quality-check verbs against real chatty output. The bullet asks for a runnable smoke-test affordance that the implementer invokes before declaring done — a distinct shape from inline prose linting. No existing task covers this affordance.

Post-mortem

Captured by /sdlc:task-work on 2026-06-04. PR: pending.

Acceptance criteria coverage

AC-1: agent-manual — ran bash plugin/scripts/dogfood_baseline.sh --max-pre-existing -1 end-to-end against the live worktree; it captured a baseline against origin/main via a throwaway worktree and printed dogfood: 354 pre-existing, 2 new with no operator input. Script is -rwxr-xr-x.
AC-2: auto — plugin/scripts/tests/dogfood_baseline.test.ts covers both directions: the “new drift” case asserts exit 1 when HEAD’s output gains a finding, the “clean subset” case asserts exit 0 when findings are a subset.
AC-3: agent-manual — added the “Pre-PR dogfood (when verb authoring is in scope)” subsection to plugin/skills/task-work/SKILL.md Step 7, scoped to verb-authoring tasks, citing bash ${CLAUDE_PLUGIN_ROOT}scripts/dogfood_baseline.sh (exists on disk).
AC-4: auto — bun test plugin/scripts/tests/dogfood_baseline.test.ts → 6 pass / 0 fail; asserts the dogfood: <X> pre-existing, <Y> new summary shape across clean / new-drift / chattiness-ceiling / passing-verb / bad-arg cases.

What worked

The existing quality_baseline.ts capture + run_quality_checks.ts --diff-against-baseline surfaces composed cleanly; the dogfood script is a thin shell wrapper over the same two helpers the real gate uses, so it exercises the genuine end-to-end path.
The plugin/scripts/tests/quality_baseline.test.ts fixture pattern (ephemeral project root, stub verbs, sentinel files for drift) ported directly to a git-fixture shape for the new eval.

Friction and automation gaps

The Step 7 baseline gate flagged 2 new-drift: lines (bun test: PR: https://github.com/example/repo/pull/{1,7}) that are demonstrably pre-existing — the exact strings appear in the baseline JSON’s bun test findings. Root cause: bun test is currently failing (273 pre-existing failures in a “validator golden suite”) AND its failing-suite stdout is non-deterministic between runs, so the per-line finding set shifts and the order-sensitive diff classifies stable lines as new. The baseline-isolation mechanism silently assumes each verb’s findings are deterministic; a flaky failing verb defeats it. Gap: quality_baseline/run_quality_checks should either (a) sort+dedupe each verb’s findings before diffing, (b) treat a verb whose pass/fail status is unchanged vs baseline as non-gating regardless of line churn, or (c) the gate should warn when a “new-drift” line is byte-identical to a baseline finding under any verb. Without one of these, the dogfood script (and the real gate) raise false positives on any project with a flaky failing verb. → T-TWZD-normalize-baseline-diff-nondeterministic-output
The originating spec named the eval plugin/scripts/test_dogfood_baseline.sh, but no gate discovers *.sh tests — only bun test (via plugin/scripts/tests/*.test.ts) runs in CI. Landed the eval as dogfood_baseline.test.ts and corrected the spec. Gap: /sdlc:task-ensure-ready’s readiness gate could lint proposed test-file paths against the project’s actual test-discovery globs, catching “this test would never run” at definition time rather than implementation time.

Spawned follow-up tasks

T-TWZD-normalize-baseline-diff-nondeterministic-output (open/ready) — LINKED-EXISTING. Bullet 1 (flaky-verb findings defeat the baseline diff) is the sixth post-mortem to surface this exact gap; T-TWZD already tracks the findingsFromStdout normalization fix. Appended T-F8BP to its related: and the dedup trail to its ## Discovery context.
Bullet 2 (ensure-ready lints proposed test-file paths against the project’s test-discovery globs) — classified Upstream-plugin/sdlc, SPAWNED, no existing tracker. NOT dispatched as an independent PR this run: the Agent//sdlc:spawn-task-pr tool surface was unavailable in this Agent-less task-work context (Step 6/8 inline fallback), and the procedure forbids opening a second independent PR from inside this flow without that boundary. Recorded here so a follow-up
```
/sdlc:spawn-task-pr --slug ensure-ready-lints-test-discovery-globs --classification Upstream-plugin --tag sdlc-meta --originating-task T-F8BP-dogfood-baseline-smoke-test
```
can open it.

← Back to Tasks