T-ZWJQ-review-run-analytics

State: planning/proposed · Impact: medium · Complexity: medium

Goal

Every review-pr pass now stamps a run record on the review it posts: when it ran, on which host and session, the model and effort, the skill build, and the head and base commits (#2226). Two of those fields, model and effort, are self-reported by the session running the skill, and the record lives only inside GitHub review bodies. Nothing reads it back. This task makes the record trustworthy and queryable, so questions like “which model produced the reviews the author acted on” or “how many findings per pass, per host, this week” have an answer.

Today

Location	Role today
`content/agent-vault/marketplaces/sksizer/plugins/pr-tools/skills/review-pr/review-pr.md`	Records the run once per pass; opens each posted review with a run table and appends the record to the marker as `key=value` pairs. `model` and `effort` are whatever the session says about itself, with `unknown` as the fallback.
`content/agent-vault/marketplaces/sksizer/plugins/pr-tools/skills/check-pr/check-pr.md`	Sibling loop skill; ends with a report line per PR and records nothing about the run that produced it.
`.sdlc/orchestrator-log.md`	The only run log the loop keeps, one prose line per orchestrate tick, gitignored, never parsed.
`docs/planning/decisions/github-ref-leases/`	The lease decision the claim ref in `review-pr` step 3 follows; the claim is deleted when the pass ends, so no run history survives it.

Proposed

The run record’s model and effort come from a source the skill can check, not from the session’s own account: an environment variable or a harness-exposed file that the claude and codex CLIs set, read by one shell command the skill names. When neither harness exposes the value, the record says unreported, distinct from unknown, so a dashboard can tell “not measurable” from “not filled in”.
A sdlc op reads the run records back: it walks the reviews on a repo’s PRs, parses each review-pr marker, and emits one row per pass with the record’s fields plus the PR, the review kind, and the finding count. The op is read-only against GitHub and emits text, json, and jsonl like every other op.
check-pr stamps the same record on the replies it posts, so the two loop skills share one shape and one reader.
A report kind renders the rows as a table by pass, by host, and by model, so a person can see at a glance which sessions are producing reviews and how they are being used.

Approach

Find what each harness exposes about the running model and effort (claude CLI environment, codex exec environment, MCP server context) and write the one command per harness that reads it. Record the answer in the skill; where a harness exposes nothing, note it and use unreported.
Extend the marker grammar note in review-pr step 2 to name the full field set and the two sentinel values, so the reader op and the skill agree on one contract.
Add a review runs op under the ontological lib that lists passes from GitHub review markers: --repo, --since, --pr, and the usual output modes. Reuse the existing gh command seam and the PR snapshot shape pr-check already fetches.
Add a report kind that renders the rows grouped by pass, host, and model, following the existing report-kind pattern under the entity packages.
Stamp the same record on check-pr replies, reusing the review-pr wording by reference rather than a second copy.
Rebuild the marketplace and confirm no drift.

Areas

Area	Note
`content/agent-vault/marketplaces/sksizer/plugins/pr-tools/skills/review-pr/review-pr.md`	Verified `model` and `effort` source per harness; `unreported` sentinel; full marker field list in step 2.
`content/agent-vault/marketplaces/sksizer/plugins/pr-tools/skills/check-pr/check-pr.md`	Stamp the run record on posted replies.
`published/marketplaces/sksizer/`	Rebuilt by `agent-pants build`.
`solutions/ontological/lib/`	A `review runs` op that reads markers back into rows, and its report kind.
`docs/planning/decisions/github-ref-leases/`	Note that the posted marker, not the claim, is the durable record of a pass.

Acceptance criteria

AC-1: A review posted by review-pr under the claude CLI carries a model value that matches what the harness reports through the command the skill names, not a value the session typed from memory.
AC-2: A harness that exposes no model or effort produces unreported in the marker, and unknown no longer appears in new markers.
AC-3: sdlc review runs --repo sksizer/dev --since <date> --json returns one row per pass with every marker field, the PR number, the review kind, and the finding count, and returns nothing for a PR with no review-pr marker.
AC-4: The report kind renders the same rows grouped by pass, host, and model, and reads correctly for reviews posted before this task (missing fields show as empty, not as an error).
AC-5: check-pr replies carry the run record in the same shape, and the reader op lists them alongside review-pr passes.
AC-6: agent-pants check reports no drift after the rebuild.

Out of scope

Changing what a review finds or how it is written; only the record around it.
Storing run records anywhere other than the posted GitHub review and the reader’s output. A local database of passes is a later decision.
Cost or token accounting per pass; neither harness exposes it today.

Dependencies

#2226 merged, so the run record exists to verify and read.

Discovery context

Raised while reviewing the first review-pr sweep on 2026-09-19. The run table was added in #2226 with model and effort self-reported, accepted as good enough for now, with this task to make them verified and to build the reader.

← Back to Tasks