T-RXKE-eval-harness-for-doc-authoring-skills

Status: planning/backlog · Impact: medium · Complexity: large

Auto-generated from a /sdlc:task-work post-mortem. Review and promote to ready before picking up.

Goal

Skills like /update-skill-doc (added by T-AV6J-add-update-skill-doc-skill) author markdown files from other inputs and claim properties — idempotency, output shape, section presence — that can only be verified by running the skill. Today those ACs are scored agent-manual because there’s no harness that drives a real invocation under deterministic conditions. A fixture-based eval harness for doc-authoring skills would let claims like “second invocation produces no diff” run as auto.

Today

Location	Role today
`apps/sdlc/skills/update-skill-doc/SKILL.md`	doc-authoring skill whose idempotency/output-shape ACs are scored `agent-manual`; no harness exercises them
`apps/sdlc/skills/import-planning/tests/run_evals.test.ts`	fixture-based `bun:test` eval suite driving a deterministic CLI verb; the per-skill eval precedent
`apps/sdlc/skills/entities-audit/tests/run_evals.test.ts`	same pattern — `Case` objects (`name`, `expectedExit`, required substrings) asserted over fixture trees
`apps/sdlc/skills/entities-migrate/tests/entities_migrate.test.ts`	further per-skill test precedent under `apps/sdlc/skills/<skill>/tests/`

Proposed

A harness that can drive a doc-authoring skill against a fixture input tree and assert observable properties of the output:

Idempotency. The harness runs the skill twice in succession; the second run’s git diff against the first must be empty.
Output shape. The written doc has the required H1, the **Trigger:** line, exactly one ```mermaid block, and a ## Notes section. (Mirrors the apps/sdlc/skills/README.md contract.)
Diff containment. The skill writes only docs/skills/<slug>.md and commits exactly that path.

The harness lives at apps/sdlc/skills/update-skill-doc/tests/run_evals.test.ts, mirroring the entities-audit / import-planning eval suites — but where those drive a deterministic CLI verb, this one must drive a sub-agent invocation, since the skill’s logic is LLM-driven.

Approach

Sketch the contract: what observable properties can a non-LLM script verify about a doc the skill wrote? Idempotency (textual diff) and shape (regex over the file) are both straightforward; semantic correctness (“the flowchart accurately reflects the SKILL.md”) is not in scope.
Build a minimal fixture tree under apps/sdlc/skills/update-skill-doc/tests/fixtures/<case>/ — each fixture is a fake repo with an apps/sdlc/skills/<slug>/SKILL.md and (optionally) an existing docs/skills/<slug>.md.
Decide on the invocation surface: either (a) the harness shells out to claude code and runs the skill against the fixture (heavy dep), or (b) the harness drives an Agent tool sub-agent inline. Pick whichever is cheaper to keep deterministic across model versions.
Write the runner as a bun:test suite with the same Case shape (name, expectedExit, required substrings) as the existing eval suites so the report format stays consistent.
Reclassify the affected ACs on T-AV6J-add-update-skill-doc-skill post-mortem from agent-manual to auto once the eval is green.

Files to touch

Location	Kind	Change
`apps/sdlc/skills/update-skill-doc/tests/run_evals.test.ts`	new	eval runner.
`apps/sdlc/skills/update-skill-doc/tests/fixtures/`	new	fixture trees, one per case.
`docs/planning/tasks/T-AV6J-add-update-skill-doc-skill.md`	modify	reclassify ACs in the post-mortem.

Acceptance criteria

AC-1: apps/sdlc/skills/update-skill-doc/tests/run_evals.test.ts exists and passes under bun test against the current /update-skill-doc skill, with at least one fixture asserting idempotency (second run produces no diff).
AC-2: A deliberately-broken /update-skill-doc (e.g. one that always appends a timestamp) makes the harness exit non-zero with a precise reason.
AC-3: Where the new harness automates an AC previously classified agent-manual in any task post-mortem, the post-mortem is updated to mark that AC auto with a citation to the harness.

Out of scope

A general eval framework for every LLM-driven skill — this task is scoped to doc-authoring skills.
Semantic validation (does the Mermaid flowchart actually match the SKILL.md steps?). That’s T-5CVT-project-check-asserts-skill-docs-match-skill-md’s territory.

Dependencies

none

Discovery context

Spawned by /sdlc:task-work post-mortem of T-AV6J-add-update-skill-doc-skill on 2026-05-19. Closing this gap moves AC-2/AC-3/AC-4 of that task from agent-manual to auto.

← Back to Tasks