Skip to content
sdlc

T-BM56-pr-check-mock-state-flag

Status: closed/done · Impact: medium · Complexity: small

Auto-generated from a /sdlc:task-work post-mortem. Review and promote to open/ready before picking up.

Goal

Section titled “Goal”

The cursor-flip half of /sdlc:pr-check’s AC-3 (a new comment between two invocations must flip the verdict to NEEDS-RESPONSE and update .claude/pr-cursors/<N>.json) cannot be exercised under the eval harness without a live open PR. A --mock-state flag that accepts a JSON blob in place of the gh pr view payload would let the classification ladder and cursor-update path run deterministically in CI and under T-RXKE-eval-harness-for-doc-authoring-skills. This closes the manual-only gap left by T-8896-add-pr-check-skill.

Today

Section titled “Today”

From the originating post-mortem:

AC-3 cannot be exercised without a live open PR — a synthetic-PR fixture (or a --mock-state flag on pr-check that takes a JSON blob in place of gh pr view) would let the cursor-flip half run under the eval harness. Worth a follow-up.

/sdlc:pr-check <pr-number> (see plugin/skills/pr-check/SKILL.md) always calls gh pr view <N> --json ... and gh api .../comments against the real GitHub API. There is no seam for injecting a deterministic state payload, so any AC that depends on a specific verdict transition reverts to agent-manual scoring.

Proposed

Section titled “Proposed”

/sdlc:pr-check accepts an optional --mock-state <path> argument. When provided:

  • The skill skips the gh pr view and gh api .../comments calls entirely and reads the JSON blob at <path> instead.
  • The blob mirrors the shape of gh pr view <N> --json state, mergeable, mergeStateStatus, reviewDecision, headRefName, baseRefName, statusCheckRollup plus an optional comments array matching the gh api shape.
  • The cursor-file write at .claude/pr-cursors/<N>.json still happens (or accepts a --cursor-dir override) so the cursor-update half of AC-3 is exercisable.
  • Mock mode is detectable from the emitted reason string (e.g. suffix (mock)) so harness assertions can distinguish it from real runs.

Approach

Section titled “Approach”
  1. Add --mock-state <path> (and optionally --cursor-dir <path>) to the pr-check skill argument contract. Update SKILL.md Step 1 (“Resolve PR number”) to branch on the flag before any gh call.
  2. Define the mock-state JSON schema inline in SKILL.md — keys match the existing gh pr view --json selection so real and mock payloads are interchangeable. Add a fixture under plugin/skills/pr-check/fixtures/ for each verdict (CLEAN, NEEDS-RESPONSE, CONFLICTS, CI-FAILED, MERGED, CLOSED).
  3. Add a lint invariant: --mock-state must not be allowed to bypass the .claude/pr-cursors/ write — the cursor path is the half of AC-3 we explicitly want to exercise.

Files to touch

Section titled “Files to touch”
  • plugin/skills/pr-check/SKILL.md — add --mock-state to the argument contract and step 1; document the JSON shape.
  • plugin/skills/pr-check/fixtures/ (new) — one JSON fixture per verdict for use by the eval harness.
  • plugin/skills/pr-check/invariants.yaml — invariant: cursor write is unconditional even in mock mode.

Acceptance criteria

Section titled “Acceptance criteria”
  • AC-1: /sdlc:pr-check 999 --mock-state plugin/skills/pr-check/fixtures/needs-response.json emits NEEDS-RESPONSE reason="..." without making any network calls. (auto — assertable via the eval harness)
  • AC-2: Running the skill twice with two different fixtures against the same PR number updates .claude/pr-cursors/999.json on the second call. (auto)
  • AC-3: At least one fixture exists per documented verdict, and a harness test asserts each fixture produces its expected verdict. (auto)

Out of scope

Section titled “Out of scope”
  • Replacing the real gh-backed code path. --mock-state is purely additive; the default behavior must be unchanged.

Dependencies

Section titled “Dependencies”
  • none

Discovery context

Section titled “Discovery context”

Spawned by /sdlc:task-work post-mortem of T-8896-add-pr-check-skill on 2026-05-20.

Post-mortem

Section titled “Post-mortem”

Captured by /sdlc:task-work on 2026-05-21. PR: pending.

Acceptance criteria coverage

Section titled “Acceptance criteria coverage”
  • AC-1: auto — the SKILL.md now documents that mock mode emits a verdict with a trailing (mock) suffix and skips every gh call; the eval harness will drive needs-response.json and assert exactly that. Verified at the prose level by lint_skill_prose.py pinning --mock-state, (mock), and the Mock mode H2.
  • AC-2: auto — Step 4 of SKILL.md now states that the cursor write is unconditional even in mock mode, and --cursor-dir <path> composes with --mock-state so the harness can target a scratch directory. The invariant cursor write is unconditional is pinned in invariants.yaml.
  • AC-3: auto — six fixtures land under plugin/skills/pr-check/fixtures/, one per non-ERROR verdict (clean, needs-response, conflicts, ci-failed, merged, closed). All are valid JSON; the eval harness, once it can drive this skill, will assert each fixture produces its expected verdict.

What worked

Section titled “What worked”
  • The skill’s existing invariants.yaml + lint_skill_prose.py scaffolding made it trivial to lift the new contract claims ( (mock) suffix, unconditional cursor write, Mock mode H2) from agent-manual to auto in a single edit.
  • All three target files were named upfront in the task spec, so the implementation was a mechanical edit pass — no exploration cost.

Friction and automation gaps

Section titled “Friction and automation gaps”
  • Step 5b’s git rebase main surfaced a frontmatter conflict (last_reviewed bumped on main, readiness_verified_at re-stamped on the feat branch by Step 5a). Resolution was mechanical (take the newer value of each field), but the conflict shape is predictable enough that task-work could either (a) defer the ensure-ready re-stamp when the existing stamp is recent, or (b) auto-resolve the rebase by preferring the newer timestamp on both lines. → T-H0W9-task-work-rebase-frontmatter-conflict, T-H98A-last-reviewed-rebase-collision
  • The task spec mentions gh api .../comments as a separate call the skill makes, but the live SKILL.md already collapses comments and reviews into the single gh pr view --json …,comments,reviews fetch. Spec was authored against a prior shape of the skill — a task-spec “still accurate?” check at pickup that diffs cited shell invocations against current SKILL.md would catch this class of drift before implementation starts. → T-XSI8-task-define-flags-spec-shell-drift
  • No eval harness exists yet to actually drive the fixtures; the three ACs are auto-eligible but currently auto-unverified until T-RXKE-eval-harness-for-doc-authoring-skills lands. This task closes the static prerequisite (fixtures + skill plumbing), but the asserting side is still pending. → T-RXKE-eval-harness-for-doc-authoring-skills

Spawned follow-up tasks

Section titled “Spawned follow-up tasks”

← Back to Tasks