Skip to content
sdlc

T-E69Y-skill-md-runtime-drift-audit

Status: closed/done · Impact: medium · Complexity: small

Auto-generated from a /sdlc:task-work post-mortem. Review and promote to open/ready before picking up.

Goal

Section titled “Goal”

The canonical procedure documented in plugin/skills/task-work/SKILL.md Step 7 calls ${CLAUDE_PLUGIN_ROOT}scripts/run_quality_checks.py --config <project-root>/sdlc.yaml --line. In practice, this project’s own runtime briefs an implementation sub-agent to use a different incantation, and a parent skill’s preflight may pre-resolve verbs inline. Both paths converge in behavior, but the prose drift makes the canonical SKILL.md a misleading map. A doc-audit pass would catch cases where the canonical procedure isn’t actually how a project runs. Surfaced by post-mortem of T-6HFR-orchestrator-categorized-in-flight-limits.

Today

Section titled “Today”

Each SKILL.md in plugin/skills/<name>/ describes a procedure in prose. Some procedures bottom out at scripts under plugin/scripts/ (which themselves carry their own help text); others reference sdlc.yaml keys or commit-message conventions. There is no audit that asserts the per-skill commands actually exist, that referenced scripts accept the flags as documented, or that the per-project sdlc.yaml lines up with the canonically-documented usage. Drift is caught only when a human spots it during a run.

Proposed

Section titled “Proposed”

A project-local auditor at .claude/scripts/audit_skill_runtime.py that walks every plugin/skills/*/SKILL.md in this repo, extracts shell-out invocations (regex against backtick-fenced commands), runs each invocation with --help, and reports drift between the documented flag set / sub-command name and the script’s actual interface. The script lives under .claude/ (project-local) rather than plugin/scripts/ (general) because the audit only makes sense inside the SDLC plugin’s own dev repo — downstream consumers don’t have a plugin/skills/ tree of their own. Out of scope: validating semantics — this is a surface-area check, not a behavior check.

Approach

Section titled “Approach”
  1. Inventory shell-out patterns across this repo’s plugin/skills/ SKILL.md files. Pin a regex (\${CLAUDE_PLUGIN_ROOT}scripts/(\S+)\.py etc.) that captures script invocations.
  2. For each captured invocation, exec it with --help in a subprocess and diff the SKILL.md’s flag set against the parser’s. Report missing-from-help and missing-from-skill diffs.
  3. Wire it into .claude/scripts/audit_skill_runtime.py and add to quality_checks: in sdlc.yaml (with the project-local path).
  4. Add a fixture-based regression suite under .claude/scripts/tests/audit_skill_runtime/ that locks in the auditor’s heuristic decisions (fenced-vs-prose flag attribution, missing-script detection, backslash-continuation handling). Each fixture is a self-contained mini-repo with synthetic plugin/skills/<name>/SKILL.md (and real Python stub scripts where needed for --help introspection). The driver shells out to audit_skill_runtime.py --repo-root <fixture> --json and asserts on drift_count plus per-finding status. Wire the driver into sdlc.yaml’s quality_checks:.

Files to touch

Section titled “Files to touch”
  • .claude/scripts/audit_skill_runtime.py (new, project-local) — the auditor.
  • .claude/scripts/tests/audit_skill_runtime/run_evals.py (new) — fixture-based regression suite driver.
  • .claude/scripts/tests/audit_skill_runtime/fixtures/ (new) — per-case synthetic mini-repos exercising specific heuristic properties.
  • sdlc.yaml — add the auditor and the eval runner to quality_checks: after verifying the audit runs in CI time.

Acceptance criteria

Section titled “Acceptance criteria”
  • AC-1: audit_skill_runtime.py exits non-zero when a SKILL.md cites a script flag that the script’s --help does not advertise.
  • AC-2: Adding the script to sdlc.yaml’s quality_checks: causes the gate to fail on intentionally-introduced drift.
  • AC-3: .claude/scripts/tests/audit_skill_runtime/run_evals.py exits 0 against the current auditor with every shipped fixture passing.
  • AC-4: Loosening the auditor to re-enable indented-flag peek-ahead (i.e. attributing flag-shaped tokens from outside the invocation’s fenced block) causes the flag-shaped-prose/ or slash-command-flag-near-script/ fixture to fail, demonstrating the regression guard works.
  • AC-5: A fixture whose SKILL.md references a script path that does not exist under plugin/ makes the auditor emit a missing finding and run_evals.py asserts that exact status.

Out of scope

Section titled “Out of scope”
  • Behavior assertions (the script will run a thing X way) — that belongs to eval harnesses.
  • Auto-fixing drift — this is a reporter.

Dependencies

Section titled “Dependencies”
  • none

Discovery context

Section titled “Discovery context”

Spawned by /sdlc:task-work post-mortem of T-6HFR-orchestrator-categorized-in-flight-limits on 2026-05-21.

Absorbed 2026-05-21-audit-skill-runtime-fixture-suite per user review comment on PR #75 (2026-05-21): the fixture suite was spawned as a follow-up but the reviewer asked it be rolled into this PR’s scope and the spawned task removed.

Post-mortem

Section titled “Post-mortem”

Captured by /sdlc:task-work on 2026-05-21. PR: pending.

Acceptance criteria coverage

Section titled “Acceptance criteria coverage”
  • AC-1: auto — introduced --completely-fake-flag into a fenced invocation in plugin/skills/task-work/SKILL.md; auditor reported drift and exited 1.
  • AC-2: auto — with the same intentional drift in place, ran plugin/scripts/run_quality_checks.py --config sdlc.yaml --line and observed FAIL .claude/scripts/audit_skill_runtime.py, exit 1. After restoring SKILL.md, full gate prints OK 8/8.

What worked

Section titled “What worked”
  • The fenced-code-block-only attribution policy cleanly eliminated the four false-positive drifts the first prose-aware heuristic surfaced (--detect, --auto, -F from git commit -F, etc.) while still catching real drift inside \```-fenced shell examples — which is exactly where SKILL.md actually documents intended invocations.
  • ${CLAUDE_PLUGIN_ROOT} resolves to <repo>/plugin/ in the dev tree, which fell out cleanly once recognized; no further path normalization was needed.

Friction and automation gaps

Section titled “Friction and automation gaps”
  • .claude/ was gitignored with .claude/* and a small allowlist (settings.json, skills/, hooks/, etc.). Adding a new project-local script under .claude/scripts/ required also un-ignoring that subtree — a step the task spec did not flag, caught only when git status showed the file missing after staging. A pre-implementation lint that “any path the task lists under Files to touch resolves to something git will track” would surface this earlier. (See also the related .claude/scripts/ un-ignore precedent.) → T-CWLT-files-to-touch-gitignore-resolution-lint
  • The first-cut flag-association heuristic (peek at indented flag-shaped continuation lines after a fenced invocation) produced four false positives against the live repo. The fix — attribute flags only when both the invocation and the flag live inside the same fenced block — was straightforward once observed, but the cycle (write → run → see false positives → tighten) cost several iterations. A small fixture suite under .claude/scripts/tests/audit_skill_runtime/ with both true-drift and look-alike-prose cases would make future heuristic tweaks regression-safe. → 2026-05-21-audit-skill-runtime-fixture-suite

Spawned follow-up tasks

Section titled “Spawned follow-up tasks”
  • T-CWLT-files-to-touch-gitignore-resolution-lint — ensure-ready disqualifier that flags Files to touch paths matched by .gitignore without an un-ignore rule (created).
  • 2026-05-21-audit-skill-runtime-fixture-suite — synthetic SKILL.md fixtures that lock in the auditor’s heuristic decisions. Absorbed into this PR per reviewer request on 2026-05-21; the standalone task file was deleted and the work rolled into AC-3, AC-4, AC-5 above.

← Back to Tasks