T-H69K-run-quality-checks-isolates-pre-existing-drift

Status: closed/done · Impact: medium · Complexity: medium

Goal

plugin/scripts/run_quality_checks.py returns a hard FAIL whenever any declared verb’s exit code is non-zero, even when that failure is caused entirely by pre-existing drift unrelated to the current branch (e.g. plugin/scripts/audit_entities.py flagging a task file that was already broken on origin/main). /sdlc:task-work Step 7 then has to manually triage every failure to decide whether to proceed — friction the runner shouldn’t have to absorb on every task pickup.

Today

Location	Role today
`plugin/scripts/run_quality_checks.py`	Runs each verb from `sdlc.yaml`’s `quality_checks:` list in order; surfaces `FAIL <first-failed-cmd>` on the first non-zero exit
`plugin/scripts/audit_entities.py`	One of the declared verbs; emits a per-file `DRIFT` listing across the entire entity corpus, not just files the branch touched
`plugin/skills/task-work/SKILL.md`	Step 7 invokes the executor and treats any non-zero exit as a hard gate-failure
`sdlc.yaml`	Declares the verb list; no concept of “baseline” or “what the branch introduced” anywhere in the schema today

There is no notion of “what the current branch changed vs. what was already broken on main”; failures are absolute. Observed twice in back-to-back /sdlc:task-work post-mortems:

[T-TZD2-task-work-spawn-fast-path-no-op](/planning/tasks/task-work-spawn-fast-path-no-op/) — audit_entities.py flagged 10 pre-existing manual-review items unrelated to the PR; runner had to confirm via diff and “Proceed anyway.”
[T-5VN7-pr-check-cursor-bootstrap-misses-existing-comments](/planning/tasks/pr-check-cursor-bootstrap-misses-existing-comments/) — 5 pre-existing drift items unrelated to the PR; required spawning a separate chore/audit-drift-fix PR (#105) as a sidequest before the gate would pass.

The friction shape is the same both times: the operator must triage every failure to decide whether the current branch caused it, which forces a context-switch mid-implementation or mid-PR.

Proposed

Instead of comparing HEAD against the merge-base on every run (the prior sketch — cost: re-run every verb twice, complexity: per-verb output diffing), memoize a baseline per origin/main SHA.

Capture the baseline at the start of /sdlc:task-work (informational, non-gating). Store one JSON file per SHA at .sdlc/quality-baselines/<origin-main-sha>.json (gitignored). At the end-of-task gate, compare the current run’s findings against the baseline file for the SHA the branch was based on. The gate fails only on findings present in HEAD but absent from the baseline — i.e. drift the branch introduced. Pre-existing drift surfaces on stderr (visible to the operator) but does not gate the exit code.

Properties this gets us:

No mid-implementation surprise. The baseline is captured before any code is written, so the gate’s behavior is deterministic by Step 7.
Cheap. One extra full run at the start; the gate at the end is a set diff, not a re-run.
No shared state. Local cache per developer; no committed baseline file accumulating noise in the repo.
Self-healing on main movement. If origin/main advances and someone fixes drift, the next task-work invocation captures a fresh baseline at the new SHA. Stale baselines auto-prune.

Severity (some findings being advisory rather than hard-fail-worthy) is an orthogonal axis and deferred — listed under “Out of scope.” This task is exclusively about the origin axis: “did the branch introduce this, or was it already there?”

Approach

Add a baseline capture/read module — likely a new file at plugin/scripts/quality_baseline.py (co-located with run_quality_checks.py; promote to its own module only if a second caller emerges). Exposes two functions:
- capture(sdlc_yaml_path, project_root, baseline_dir, sha) -> Path — runs every verb against the current tree, collects each verb’s output line-by-line, writes a JSON file at <baseline_dir>/<sha>.json shaped as
```
{"sha": <sha>, "captured_at": <iso8601>, "verbs": {<verb>: {"exit": <int>, "findings": [<line>, ...]}}}
```
  Returns the written path.
- diff(current_findings, baseline_findings) -> list[str] — returns the list of <verb>: <line> strings present in current but absent from baseline. Verb-aware (per-verb line-set diff).
Extend run_quality_checks.py with two new flags:
- --baseline-dir <path> — defaults to <project-root>/.sdlc/quality-baselines/. The directory is auto-created on first capture.
- --diff-against-baseline <sha> — gates the run’s exit code on the diff against the named baseline. Without this flag, behavior is unchanged (every non-zero verb still fails the gate). With the flag, pre-existing findings (present on both HEAD and the baseline) surface on stderr as pre-existing: <verb>: <line> but don’t contribute to the exit code; new findings (present on HEAD but not on the baseline) surface on stderr as new-drift: <verb>: <line> and do flip the exit to non-zero.
Add a baseline-prune helper invoked by every capture call: keep the 5 most recent JSON files in <baseline_dir> (sort by file mtime, delete the rest). Five is a soft default — covers typical in-flight parallelism without growing unbounded.
Update /sdlc:task-work Step 3 (post-flight, before worktree creation) to capture a baseline against the current origin/main SHA and surface the count to the operator as informational (“Baseline captured: N pre-existing findings”). Step 7’s invocation of run_quality_checks.py passes --diff-against-baseline <sha> with the SHA captured in Step 3.
Update plugin/conventions/sdlc-yaml.md to document the .sdlc/quality-baselines/ location and the new flags. Confirm .sdlc/ is already gitignored (it is — 2026-05-22-move-plugin-runtime-state-to-sdlc-dir shipped that).
Add regression tests at plugin/scripts/test_quality_baseline.py covering: (a) capture writes a well-formed file; (b) diff returns [] when current==baseline; (c) diff returns the new lines when current is a superset of baseline; (d) --diff-against-baseline exits 0 when only pre-existing findings are present and exits 1 when at least one new finding is present; (e) prune keeps 5, deletes older.

Files to touch

Location	Kind	Change
`plugin/scripts/quality_baseline.py`	new	Capture / read / diff / prune helpers; one JSON file per `origin/main` SHA at `<project-root>/.sdlc/quality-baselines/<sha>.json`
`plugin/scripts/run_quality_checks.py`	modify	Add `--baseline-dir` and `--diff-against-baseline <sha>` flags; route output through the diff helper when the latter is supplied
`plugin/skills/task-work/SKILL.md`	modify	Step 3: capture baseline against current `origin/main` SHA, surface count. Step 7: pass `--diff-against-baseline <captured-sha>` to the executor
`plugin/conventions/sdlc-yaml.md`	modify	Document the baseline directory location and the two new flags
`plugin/scripts/test_quality_baseline.py`	new	Regression tests for capture / diff / gate / prune

Acceptance criteria

AC-1: plugin/scripts/quality_baseline.py exposes capture(...) and diff(...) functions; capture writes a JSON file at <baseline_dir>/<sha>.json with the documented shape; diff returns the per-verb set difference of findings.
AC-2: run_quality_checks.py --diff-against-baseline <sha> exits 0 when every finding on HEAD is also in the named baseline file; pre-existing findings surface on stderr prefixed pre-existing:.
AC-3: run_quality_checks.py --diff-against-baseline <sha> exits non-zero when HEAD introduces at least one finding absent from the named baseline; new findings surface on stderr prefixed new-drift:.
AC-4: A capture call prunes <baseline_dir> to the 5 most recently mtimed JSON files.
AC-5: /sdlc:task-work Step 3 captures a baseline against the current origin/main SHA before the worktree is created and surfaces the pre-existing finding count to the operator.
AC-6: /sdlc:task-work Step 7’s documented invocation example passes --diff-against-baseline <sha> with the SHA captured in Step 3.
AC-7: plugin/scripts/test_quality_baseline.py exercises capture / diff / gate (both directions) / prune; all tests pass.
AC-8: plugin/conventions/sdlc-yaml.md documents the .sdlc/quality-baselines/ location and both new flags.

Out of scope

Severity classification of findings. Some findings (e.g. scratch notes lacking frontmatter) are stylistically noisy; others (e.g. a closed task with no completion_note) are real bugs. This task treats all findings uniformly — the gate fires if the branch introduced one. A future task can layer a per-verb (or per-finding) severity that downgrades some new findings to advisory-only. That belongs at the verb level, not the runner level.
Committing baselines. Each developer’s .sdlc/quality-baselines/ is local. No coordination across machines is needed; the cache rebuilds on first use against any new origin/main SHA.
Reformatting existing audit output. audit_entities.py’s line shape is treated as opaque by the diff helper. If we later want structured per-finding metadata, that’s a separate audit-side change.
Backfilling baselines for historical SHAs. Only the current origin/main (and whatever’s already cached) is captured. There’s no need to rebuild “what main looked like a week ago.”

Dependencies

none

Discovery context

Spawned by /sdlc:task-work post-mortem of T-TZD2-task-work-spawn-fast-path-no-op on 2026-05-22.

Dedup search (spawn-from-post-mortem)

Bullet: Pre-existing audit drift (10 manual-review items unrelated to this PR) gated run_quality_checks.py. Tracked separately as T-H69K-run-quality-checks-isolates-pre-existing-drift — proceeded past the gate by user choice; no new drift introduced. Keywords searched: run-quality-checks-isolates-pre-existing-drift, run_quality_checks, manual-review, pre-existing, separately, introduced, unrelated, proceeded Excluded: 2026-05-22-restructure-task-touchpoints-as-a-table-with-symbol-dir-glob Top candidates (score / status / headline):

30 / planning/draft / 2026-05-21-audit-entities-baseline-allow — audit_entities.py: distinguish pre-existing drift from PR-introduced drift
15 / planning/draft / 2026-05-21-run-quality-checks-isolates-pre-existing-drift — run_quality_checks.py only fails on drift the current branch introduced
9 / closed/done / 2026-05-21-task-work-spawn-fast-path-no-op — Inline fast-path for empty post-mortem friction lists in /sdlc:task-work
6 / closed/done / 2026-05-20-sdlc-yaml-json-schema-and-validator — Add JSON Schema + validator for sdlc.yaml
5 / closed/done / 2026-05-19-task-work-uses-per-project-quality-checks — Make /sdlc:task-work quality-check commands per-project configurable Decision: LINKED-EXISTING Rationale: Bullet contains an explicit wikilink to this very task. The originating task author explicitly identified this task as the tracking item (“Tracked separately as …”). Override script’s LINKED-EXISTING→audit-entities-baseline-allow recommendation to honor the explicit author signal. The audit-entities task is adjacent but distinct (it’s about audit_entities.py per-file baselines); this task is the right home for the runner-level isolation gap. Linked to: 2026-05-21-run-quality-checks-isolates-pre-existing-drift

Dedup search (spawn-from-post-mortem)

Bullet: audit_entities.py quality-check failed on 5 pre-existing drift items unrelated to this branch — required a separate chore/audit-drift-fix PR before this PR could pass the gate. The task 2026-05-21-run-quality-checks-isolates-pre-existing-drift would close this loop by making the gate fail only on drift the current branch introduced. Keywords searched: run-quality-checks-isolates-pre-existing-drift, audit-drift-fix, audit_entities, quality-check, pre-existing, introduced, unrelated, required Excluded: 2026-05-21-pr-check-cursor-bootstrap-misses-existing-comments Top candidates (score / status / headline):

45 / planning/draft / 2026-05-21-audit-entities-baseline-allow — audit_entities.py: distinguish pre-existing drift from PR-introduced drift
31 / planning/draft / 2026-05-21-run-quality-checks-isolates-pre-existing-drift — run_quality_checks.py only fails on drift the current branch introduced
20 / closed/done / 2026-05-19-implement-entities-migrate — Implement /sdlc:entities-migrate to apply mechanical schema-drift fixes
13 / closed/done / 2026-05-19-skill-prose-invariant-linter — Doc-linter that asserts required invariants in skill prose
12 / closed/done / 2026-05-19-clarify-out-of-scope-requirement — implementation-ready contract disambiguates whether Out of scope is required or conditional Decision: LINKED-EXISTING Rationale: Bullet contains an explicit reference to “the task 2026-05-21-run-quality-checks-isolates-pre-existing-drift” as the tracking item that would close this loop. Override script’s top recommendation (audit-entities-baseline-allow, score 45) to honor the explicit author signal pointing at this task (score 31). The audit-entities task is adjacent but distinct (per-file baseline allow-list at the audit_entities layer); this task is the right home for the runner-level isolation gap that the originating bullet describes. Linked to: 2026-05-21-run-quality-checks-isolates-pre-existing-drift Originating task: 2026-05-21-pr-check-cursor-bootstrap-misses-existing-comments

Post-mortem

Captured by /sdlc:task-work on 2026-05-23. PR: pending.

Acceptance criteria coverage

AC-1: auto — plugin/scripts/quality_baseline.py exposes capture(...), diff(...), prune(...); capture writes JSON at <baseline_dir>/<sha>.json; pinned by case_capture_writes_wellformed_json.
AC-2: auto — --diff-against-baseline <sha> exits 0 when HEAD findings are all in the baseline; pre-existing findings prefixed pre-existing: on stderr; pinned by case_gate_exits_zero_when_no_new_drift + dogfood demo against origin/main (e9de4dc).
AC-3: auto — same flag exits non-zero when HEAD introduces a finding; new entries prefixed new-drift:; pinned by case_gate_exits_nonzero_when_new_drift_introduced.
AC-4: auto — capture() calls prune(baseline_dir, keep=5); pinned by case_capture_invokes_prune + case_prune_keeps_n_and_deletes_older.
AC-5: auto — /sdlc:task-work Step 3a documents the baseline capture against the current origin/main SHA and the operator-facing count line; verifiable via grep on SKILL.md.
AC-6: auto — Step 7’s documented invocation appends --diff-against-baseline "$ORIGIN_MAIN_SHA"; Step 9’s post-rebase re-run does the same; verifiable via grep.
AC-7: auto — plugin/scripts/test_quality_baseline.py covers 11 cases (capture / diff / gate-zero / gate-nonzero / missing-baseline / prune / capture-invokes-prune / dropping-passing-verb-stdout / chatty-passing-verb-silence / shifting-passing-verb-silence); all pass.
AC-8: auto — plugin/conventions/sdlc-yaml.md “Baseline-gated mode” subsection documents .sdlc/quality-baselines/, both flags, the JSON shape, and the prune policy.

What worked

The design conversation at task-define time scoped the work cleanly. Defining the cache layout (one file per SHA, gitignored, ~5 retained), the flag names, and the verb-output shape upfront meant the implementer had no design decisions left to make at code time.
Dogfood testing of the end-to-end feature flow (capture against origin/main, gate against the captured SHA) caught a real false-positive before opening the PR. The first implementation wave passed all 8 ACs but produced ~30 lines of pre-existing: spam from passing test-runner verbs and a single false-positive new-drift: from a count-summary line that incremented because the branch added a new script. A single-line semantic refinement (only capture findings from verbs that exited non-zero) cleanly resolved both. Pre-emptive dogfood beat reactive bug-fix.
The start_task.py flip-and-rebase conflict was the same one I hit on the prior task; resolution took under a minute because I’d seen it before. Once is friction; twice is a pattern (see below).

Friction and automation gaps

start_task.py rebase conflict on the task file’s frontmatter is now confirmed as a recurring failure mode — happened on both the pr-check task last turn and this task. The conflict arises when ensure-ready’s verify-stamp commit and start-commit on main both touch the frontmatter near readiness_verified_at/last_reviewed and three-way merge can’t resolve cleanly. The fix should be in start_task.py itself: detect the verify-stamp commit on the task branch, see that its only delta is readiness_verified_at, and absorb the value into the start-commit’s frontmatter edit so the verify-stamp commit’s content is already present in the rebase target. (Tracked separately, but this is the second consecutive task hit by it — promote priority.) → T-2QXZ-start-task-handles-frontmatter-rebase-cleanly
The v1 design’s “capture every stdout line as a finding” rule didn’t survive contact with chatty passing verbs (test runners, the audit_skill_runtime count summary). The dogfood caught it, but the spec authoring should have predicted it. A pre-commit dogfood eval (capture-then-gate against origin/main before opening the PR) would have caught the issue without a sub-agent round trip. Worth adding: a plugin/scripts/dogfood_baseline.sh (or similar) that the implementer runs locally as a smoke test before declaring done. → T-F8BP-dogfood-baseline-smoke-test
The --project-root positional was missing from the SKILL.md Step 3a example in the first wave; the implementing sub-agent’s audit_skill_runtime.py caught it as a script-reference drift and prompted a fixup commit (d7ab434). That gap exists because the SKILL.md example was authored without running the actual script first. Worth adding: a SKILL.md-example linter that extracts shell-shaped lines (script + flags) and dry-runs them against the actual script’s --help to confirm the flag set matches. Out of scope here, but file as a follow-up. → T-CTTD-skill-md-shell-example-lint
The plugin path ${CLAUDE_PLUGIN_ROOT}scripts/run_quality_checks.py resolves to the installed plugin, not the worktree under development, which makes the new flags invisible to invocations that use that path until the PR merges. This is the known issue tracked by [T-ZFE9-task-work-uses-worktree-skill-md](/planning/tasks/task-work-uses-worktree-skill-md/). Not a new gap, but it bit me again during dogfood (my first capture attempt against ${CLAUDE_PLUGIN_ROOT} failed because the installed plugin didn’t know about --baseline-dir). Reinforces priority of that task. → T-ZFE9-task-work-uses-worktree-skill-md
v1 line-level diff still produces false-positive new-drift: on chatty FAILING verbs. The wave-2 refinement (“only capture findings from verbs that exit non-zero”) solved chatty passing verbs. But audit_entities.py exits non-zero whenever ANY file drifts, and its stdout includes - OK lines for the passing files plus a summary count line — both of which shift whenever the corpus grows. Self-gating this PR against origin/main produced 3 false-positive new-drift: entries: two - OK lines for the new spawned task files and the summary Audited N file(s) line where N changed (144→149). Real DRIFT lines (- DRIFT ...) correctly classified as pre-existing:. This is a meaningful v1 limitation — line-level diff doesn’t know which lines are “findings” vs “noise” in a verb’s output. The proper fix is verb-aware filtering (e.g. only consider lines matching ^- DRIFT for audit_entities.py, or a per-verb finding-extractor declaration in sdlc.yaml). Captured here so a follow-up can pick it up; do NOT block this PR on it — the gate is materially less noisy than the absolute-pass/fail it replaces. → spawn a follow-up: per-verb finding-extractor (line patterns) so the diff sees only real drift, not corpus-shape lines

Spawned follow-up tasks

T-2QXZ-start-task-handles-frontmatter-rebase-cleanly — auto-resolve the recurring start_task.py rebase frontmatter conflict (linked)
T-F8BP-dogfood-baseline-smoke-test — pre-PR dogfood smoke-test script for new quality-check verbs (created)
T-CTTD-skill-md-shell-example-lint — lint SKILL.md shell examples against actual script --help (created)
T-ZFE9-task-work-uses-worktree-skill-md — bootstrap task-work from the worktree’s SKILL.md/scripts when the task modifies them (linked)

← Back to Tasks