T-BLFH-extend-project-check-with-skill-doc-lint-suite

Status: closed/done · Impact: high · Complexity: medium

Goal

Three separate skill-doc lint tasks were filed independently for the same surface: plugin/skills/<slug>/SKILL.md paired with docs/skills/<slug>.md. Each adds one new check to project-check. Collapsing them into one umbrella task lets the implementer extend project-check once (one PR, one wave of fixture trees, one wiring into SKILL.md) instead of three round-trips against the same script. The result is a hard gate on three classes of skill-doc drift that today rely on per-edit hooks or eyeballing.

Today

.claude/skills/project-check/ ships two checks today:

check_entities.py — validates entity schemas + frontmatter shape.
check_skill_prose.py — runs the prose-invariant linter from PR #19 across every SKILL.md.

Neither asserts anything about the per-skill docs/skills/<slug>.md companions, even though there are now 15 of them. The only existing defenses are:

.claude/hooks/warn-skill-doc-drift.py — per-Edit advisory, can’t block, doesn’t run on already-merged code.
Manual review when a SKILL.md changes.

Three known holes:

No Mermaid parse check. Every docs/skills/<slug>.md carries a
```
is "GitHub renders it / my eyes say it looks right."
```
No pipe-tail antipattern check. Skill prose that shows validate_frontmatter.py | tail -... masks the validator’s exit code (Bash returns the last pipeline stage’s exit code by default). Section “Don’t pipe commands you gate on” was added to plugin/skills/CLAUDE.md:88 but nothing enforces it on new prose.
No doc-coverage check. Nothing asserts that every skill in plugin/skills/ has a matching docs/skills/<slug>.md whose flowchart cites each numbered step. The next skill added without a doc only gets caught when someone tries to edit its SKILL.md.

Proposed

/sdlc:project-check grows three new deterministic checks under .claude/skills/project-check/, each invoked from the project-check entry point and following the same <path>:<line>: <message> / exit-code 0-or-1 contract as check_entities.py / check_skill_prose.py. Running /project-check exercises the suite; CI gates on it.

Approach

Land as a single PR; the three checks share scaffolding (fixture-tree layout, runner shape) and adding them together is cheaper than three round-trips.

Scaffold the suite: add .claude/skills/project-check/check_skill_docs.py and a tests/skill-doc-fixtures/ tree under the existing project-check tests directory. Steal structure from check_entities.py.
Mermaid parse check (absorbed from T-LHO1-lint-mermaid-blocks-in-skill-docs): a Python helper in .claude/skills/project-check/check_skill_docs.py extracts each docs/skills/*.md ```mermaid block and shells out to mmdc --input - --output /dev/null --quiet. Aggregate; non-zero on any parser failure with a per-file report (file path, block index, parser error excerpt). No separate bash sub-helper — the Python script owns the full check end-to-end. Pre-requisite: mmdc on PATH. Add a deterministic plugin/scripts/check_dependencies.sh (or equivalent) that /sdlc:setup invokes advisorily so a clean checkout gets a clear install message rather than a confusing parser error.
Pipe-tail antipattern check (absorbed from T-XPG7-validator-skill-examples-no-pipe-tail): new standalone script .claude/skills/project-check/check_pipe_tail.py (NOT folded into check_skill_prose.py — keeps each check independent, easier to reason about and toggle). The script greps every plugin/skills/*/SKILL.md for validate_frontmatter.py[^\n]*\| patterns and allowlists explicit-pipefail forms (set -o pipefail nearby, or ${PIPESTATUS[0]} capture). AC-1 of the superseded task already verified the corpus is currently clean — this check prevents regression. Convention doc at plugin/skills/CLAUDE.md:88 already exists.
Doc-coverage check (absorbed from T-5CVT-project-check-asserts-skill-docs-match-skill-md): walk plugin/skills/*/SKILL.md, confirm a matching docs/skills/<slug>.md exists, parse each SKILL.md’s ## N. … numbered steps, parse the doc’s ```mermaid block as a string, and substring-match each step number against the flowchart text. Best effort — do NOT parse Mermaid semantically (that’s step 2’s job).
Wire all three into .claude/skills/project-check/SKILL.md as ordered steps; verify /project-check runs them and surfaces failures verbatim.
Fixture coverage per check: at least one clean case and one deliberately-broken case per failure mode.

Files to touch

.claude/skills/project-check/check_skill_docs.py (new) — Mermaid parse check (Approach step 2) AND doc-coverage check (Approach step 4). One Python script owning both since they share the per-skill walk; emits separate failure-class prefixes (MERMAID: / COVERAGE:) so failures are still distinguishable.
plugin/scripts/check_dependencies.sh (new) — advisory mmdc / uv / python3 presence check used by /sdlc:setup so the Mermaid lint has a clear install path on first run.
.claude/skills/project-check/check_pipe_tail.py (new) — pipe-tail antipattern check (Approach step 3). Standalone, not folded into check_skill_prose.py.
.claude/skills/project-check/SKILL.md — add steps invoking the three new checks.
plugin/skills/setup/SKILL.md + plugin/scripts/setup_planning.py — invoke check_dependencies.sh advisorily after the structure step.
.claude/skills/project-check/tests/skill-doc-fixtures/... (new) — fixture trees for each check, clean + broken cases.
.claude/skills/project-check/tests/run_skill_doc_evals.py (new) — runner that walks fixtures and asserts per-check expected exit codes.
plugin/skills/README.md — add “Validation” subsection referencing the lint suite.

Acceptance criteria

Mermaid parse (absorbed from lint-mermaid-blocks-in-skill-docs):

AC-1: The mermaid parse check exits 0 against every current docs/skills/*.md.
AC-2: A deliberately-broken Mermaid block (e.g. missing --> arrow target) makes the check exit non-zero with a file-and-block-index pointer.
AC-3: plugin/scripts/check_dependencies.sh exists and reports presence/absence of mmdc, uv, python3 (+ any other plugin hard requirements) with concrete install instructions.
AC-4: /sdlc:setup invokes the dependency check advisorily — setup completes regardless, but missing tools are surfaced.

Pipe-tail antipattern (absorbed from validator-skill-examples-no-pipe-tail):

AC-5: grep -rEn 'validate_frontmatter\.py[^\n]*\|' plugin/skills/*/SKILL.md returns no results that aren’t allowlisted as pipefail-safe; the check enforces this in /project-check.
AC-6: A fixture SKILL.md with a raw validate_frontmatter.py | tail -… pattern makes the check exit non-zero with a path:line citation.
AC-7: A fixture SKILL.md that wraps the same pattern in set -o pipefail (or captures ${PIPESTATUS[0]}) passes the check.

Doc coverage (absorbed from project-check-asserts-skill-docs-match-skill-md):

AC-8: check_skill_docs.py exits 0 against the current corpus (every skill has a doc whose flowchart text cites every numbered step).
AC-9: A fixture missing docs/skills/<slug>.md makes the check exit non-zero with a path and missing-doc message.
AC-10: A fixture where the doc’s Mermaid block omits step N makes the check exit non-zero with a path:line citation naming the missing step.

Wiring:

AC-11: .claude/skills/project-check/SKILL.md runs all three new checks; a fresh /project-check invocation surfaces failures from each one verbatim.

Out of scope

Validating Mermaid semantics against SKILL.md control flow (that the decision diamonds line up with actual branches). That’s a harder problem and likely needs LLM judgment — see T-RXKE-eval-harness-for-doc-authoring-skills.
Linting Mermaid blocks outside docs/skills/ (arbitrary docs).
Auto-formatting Mermaid blocks.
Auto-installing missing dependencies.
Generalizing the pipe-tail check to other scripts beyond validate_frontmatter.py (allowed as a follow-up once the shape is proven on the motivating example).

Dependencies

none

Discovery context

Three sibling tasks all targeting /sdlc:project-check for new skill-doc asserts were filed independently on 2026-05-19. Collapsed into this umbrella on 2026-05-20 during /sdlc:task-review cleanup; the three originals are closed/superseded with completion_notes pointing here.

Post-mortem

Captured by /sdlc:task-work on 2026-05-21. PR: pending.

Acceptance criteria coverage

AC-1: auto — check_skill_docs.py exits 0 against the current docs/skills/*.md corpus (run end-to-end during implementation).
AC-2: auto — tests/skill-doc-fixtures/broken-mermaid/ exercises the dangling-arrow case; run_skill_doc_evals.py asserts exit 1 with a MERMAID: …:block-1 citation.
AC-3: auto — plugin/scripts/check_dependencies.sh reports python3 / uv / mmdc / git with concrete install hints; verified by running with a stripped PATH=/usr/bin:/bin to confirm missing-tool output.
AC-4: auto — plugin/skills/setup/SKILL.md now runs check_dependencies.sh as Step 4 (advisory). Verified by inspecting the SKILL.md change; the script’s exit-0 contract means setup never fails on missing tools.
AC-5: auto — check_pipe_tail.py exits 0 against the current plugin/skills/*/SKILL.md corpus. The same scan would now block any future SKILL.md adding a raw validate_frontmatter.py | tail.
AC-6: auto — tests/skill-doc-fixtures/bad-pipe-tail/ exercises the raw pipe; eval asserts exit 1 with the file:line citation.
AC-7: auto — allowlisted-pipefail and allowlisted-pipestatus fixtures pass (exit 0); the allowlist window looks 5 lines back for set -o pipefail and 3 lines forward for ${PIPESTATUS[0]}.
AC-8: auto — check_skill_docs.py (coverage step) exits 0 against the current corpus. Required 5 companion-doc fixes (find-quality-checks added; 4 docs had step-number drift).
AC-9: auto — missing-companion-doc fixture exits 1 with COVERAGE: …: missing companion doc for skill <slug>.
AC-10: auto — missing-step fixture exits 1 with COVERAGE: …:<line>: step 3 not cited.
AC-11: auto — .claude/skills/project-check/SKILL.md now invokes Step 1a (check_pipe_tail.py) and Step 1b (check_skill_docs.py). Both scripts emit verbatim file:line citations on failure, matching the existing surface contract.

What worked

The per-check structure (one script per concern, shared fixture layout, single run_skill_doc_evals.py) made it cheap to add the third check after the first two — copy a stanza, add a fixture, add a CASES_* tuple. The friction stayed flat as scope grew.
mmdc’s --quiet --output <tmp> surface returns a clean non-zero exit on parse failure, so the Python wrapper stayed tiny — just subprocess + stderr-first-line capture.
The coverage check’s substring-match strategy caught real drift on the first run (4 stale docs + 1 missing companion). No false positives across the 20 currently-shipped skills.

Friction and automation gaps

mmdc wasn’t installed at task pickup — had to npm install -g @mermaid-js/mermaid-cli mid-run, taking ~20s. The task spec anticipated this with check_dependencies.sh, but no automation ran the check at task start. /sdlc:task-work could shell out to plugin/scripts/check_dependencies.sh (or a project-declared equivalent) as a Step 4.5 once the worktree is initialised, so a task whose ACs need a particular binary fails fast instead of failing mid-implementation. → 2026-05-21-task-work-runs-check-dependencies-preflight
Pre-existing mermaid drift in the corpus: 9 docs had unquoted parens / single quotes / brackets inside […] node labels that the check refused. Each fix was mechanical (wrap label in "…"), but the cleanup ballooned the PR diff. A plugin/skills/README.md house-style rule + an update-skill-doc pre-flight that runs the mermaid check before writing would have prevented this drift from accumulating in the first place.
Coverage check identified 4 stale flowcharts where SKILL.md grew a step but docs/skills/<slug>.md didn’t. The warn-skill-doc-drift.py hook fires on Edit but is advisory and doesn’t run in CI. A /sdlc:project-check gate in pre-commit (lefthook) would close that loop. Tracked elsewhere — this PR fixes the symptom, not the source of drift. → T-A5H1-lefthook-project-check-pre-commit
The eval runner skips mermaid-requiring cases when mmdc is absent (counted as pass, with a SKIP line). That keeps CI on hosts without Node from breaking, but it also means a regression could silently land if every CI host happens to lack mmdc. A --require-mmdc flag for CI would tighten the contract; the default-skip stays useful for local dev. → T-CPAH-check-skill-docs-require-mmdc-flag

Spawn follow-ups note

Three follow-ups worth tracking are surfaced above. A fourth — generalising the pipe-tail check beyond validate_frontmatter.py — is already enumerated in this task’s Out of scope; defer until a second gating example demands it.

Spawned follow-up tasks

2026-05-21-task-work-runs-check-dependencies-preflight — /sdlc:task-work should shell check_dependencies.sh after worktree_init: so missing binaries surface before implementation starts. Created.
T-A5H1-lefthook-project-check-pre-commit — wire the four deterministic project-check scripts into lefthook.yml pre-commit so doc/skill drift cannot land silently. Created.
T-CPAH-check-skill-docs-require-mmdc-flag — add --require-mmdc so CI runners can demand the binary instead of falling through to the local-dev SKIP path. Created.

← Back to Tasks