T-9CI4-skill-prose-invariant-linter

Status: closed/done · Impact: medium · Complexity: medium

Auto-generated from a /sdlc:task-work post-mortem. Review and promote to ready before picking up.

Goal

Several ACs on T-DHGL-build-import-planning-skill (AC-5, AC-6, AC-7) had to be marked agent-manual because they assert properties of the skill’s prose (e.g. “Step 5 dispatches a single sub-agent in one message”, “body sections are populated, not left as placeholders”). Today these are verified by a human reading the SKILL.md. A small deterministic linter that scans SKILL.md files for required phrases / sections would let these ACs run as auto, shrinking the agent-manual band that LLM-orchestration skills inherently carry.

Today

LLM-orchestration skills (import-planning, milestones-from-file, task-work, project-cleanup) cannot exercise their end-to-end behavior under the eval harness — sub-agents only exist at skill-invocation time. So ACs that assert “Step 7 shells out to new_task.py” or “Step 5 fans out in one message” are checked by the orchestrator reading the prose during task-work Step 7.
This is recorded in T-DHGL-build-import-planning-skill’s post-mortem:

“AC-5 / AC-6 / AC-7 are orchestration claims that can only be fully exercised by a live sub-agent fan-out, which the eval harness can’t simulate (sub-agents only exist at skill-invocation time). Marked agent-manual against SKILL.md prose. A future affordance — a --self-test mode that uses fixture sub-agent responses, or a doc-linter that asserts required invariants are present in skill prose — would make these auto.”
There is no existing tooling that lints skill prose.

Proposed

A plugin/scripts/lint_skill_prose.py (PEP-723 uv run) that:

Takes one or more SKILL.md paths.
Reads a co-located <skill-dir>/invariants.yaml (new) declaring required phrases / required H2 sections / required tool references for that skill.
Reports missing invariants with a precise file:line citation, exits non-zero on any failure.
Is callable from /project-check (added as a new check script alongside check_entities.py) so drift is caught project-wide.

ACs of the form “Step N dispatches sub-agents in a single message” then become auto-verifiable by a single grep-style invariant in invariants.yaml.

Approach

Draft the invariants.yaml schema: required-phrase, required-h2, forbidden-phrase, required-tool-ref. Start with the smallest set that covers the agent-manual ACs already on import-planning (Steps 5, 7c).
Implement lint_skill_prose.py with that schema. Plain markdown parsing — no LLM. Output one violation per line in <path>:<line>: <message> format.
Wire it into .claude/skills/project-check/check_skill_prose.py (new) so /project-check runs it across every SKILL.md under plugin/skills/.
Backfill invariants.yaml for import-planning, milestones-from-file, task-work, project-cleanup — the four current LLM-orchestration skills.
Flip the affected ACs on [T-DHGL-build-import-planning-skill](/planning/tasks/build-import-planning-skill/) (and any other task whose post-mortem points here) from agent-manual to auto, with a brief note linking to the linter.

Files to touch

plugin/scripts/lint_skill_prose.py (new) — the linter.
plugin/skills/import-planning/invariants.yaml (new) — first user.
plugin/skills/milestones-from-file/invariants.yaml (new).
plugin/skills/task-work/invariants.yaml (new).
plugin/skills/project-cleanup/invariants.yaml (new).
.claude/skills/project-check/check_skill_prose.py (new) — /project-check entry point.
.claude/skills/project-check/SKILL.md — document the new check.

Acceptance criteria

AC-1: lint_skill_prose.py plugin/skills/import-planning/SKILL.md exits 0 against the unmodified skill and non-zero when a required invariant phrase is deleted in a tmp copy.
AC-2: /project-check runs the new linter and reports failures with file:line citations.
AC-3: At least three previously agent-manual ACs on T-DHGL-build-import-planning-skill (AC-5, AC-6, AC-7) can be re-classified auto because the linter mechanically asserts the same property.
AC-4: A targeted test case demonstrates a violation is caught (mutate a tmp copy of a SKILL.md, run the linter, assert non-zero exit with the expected file:line citation).

Out of scope

Linting body prose of docs/planning/<type>/ entities — that’s check_entities.py’s domain.
Semantic understanding of prose. The linter is a glorified phrase/section presence checker.

Dependencies

none

Discovery context

Spawned by /sdlc:task-work post-mortem of T-DHGL-build-import-planning-skill on 2026-05-19. Closes the “future affordance” the post-mortem proposed for making LLM-orchestration ACs auto-verifiable.

Post-mortem

Captured by /sdlc:task-work on 2026-05-19. PR: pending.

Acceptance criteria coverage

AC-1: auto — plugin/scripts/lint_skill_prose.py plugin/skills/import-planning/SKILL.md exits 0 against the unmodified skill; mutating a tmp copy (deleting one sub-agent in a single message) makes it exit 1 with a <path>:104: citation. Verified inline.
AC-2: auto — .claude/skills/project-check/check_skill_prose.py shells the linter across every plugin/skills/*/SKILL.md and streams its <path>:<line>: <message> output verbatim. /project-check’s SKILL.md adds a Step 0 that runs it. Manually verified the script exits 0 on the current corpus.
AC-3: auto — AC-5/AC-6/AC-7 on T-DHGL-build-import-planning-skill reclassified from agent-manual to auto, each citing the specific invariants.yaml clause that pins the claim.
AC-4: auto — .claude/skills/project-check/tests/run_prose_evals.py reports 6/6 eval case(s) passed. The missing-phrase fixture asserts a specific file:line citation (missing-phrase/sample-skill/SKILL.md:11:) that mutates against a deliberately-deleted invariant phrase, satisfying the AC-4 mutation contract directly.

What worked

The existing check_entities.py + run_evals.py pattern was a strong donor — same PEP-723 self-bootstrapping shape, same fixture-as-self-contained-tree model, same (name, exit, [substrings]) tuple expectations. The new prose linter slid into the same idiom with no invention.
Opt-in-per-skill (linter is silent when no sibling invariants.yaml) sidestepped the “must everyone adopt this immediately” problem. The 10 skills without invariants stay valid; the 4 that need it get it now.
Keeping invariants as literal-phrase / required-H2 / required-tool-ref checks (no regex, no semantic understanding) kept the linter ~300 LOC and the per-skill invariants.yaml files readable. Anything fancier would creep toward LLM territory and lose the deterministic value.

Friction and automation gaps

Markdown fenced-code blocks tripped the heading splitter — task-work/SKILL.md embeds a ## Post-mortem H2 inside a ``` template, which the naive H2 regex picked up as a real section start. Fixed by adding a fenced-code-block detector to the splitter. Worth flagging because the same trap exists for any Markdown-aware tooling we write next; a small shared utility might pay off.
The first cut of section-hint matching only used H3-or-narrower bounds, so section: "8." would end at the first H3 child rather than spanning the whole H2 region. Fix: prefer H2 matches, fall back to H3-only matches. Documented in the linter source — if this idiom shows up again we should extract _split_into_subheading_sections to a shared helper. Not blocking.
Two fixture SKILL.md files accidentally included the phrase the fixture was meant to assert absent — once in the description: frontmatter (“Skill prose forgot to mention new_task.py”) and once as the literal word “TODO” describing the scenario. Forced two re-runs of the eval before the cases passed. Worth a check that fixture descriptions don’t contain literal-phrase assertions the fixture intends to test against — but cost/benefit feels low; the runner caught both with one-line edits.

Spawned follow-up tasks

None this run. The remaining agent-manual ACs on import-planning (AC-1, AC-4) are not phrase-pinnable: AC-1 asserts the skill loads, AC-4 asserts --dry-run behavior end-to-end. Both still require a live invocation, so the prose linter cannot help.

← Back to Tasks