T-B2VW-agent-pants-e2e-gates-pull-requests
State: planning/draft · Impact: medium · Complexity: small
Agent-pants has two real Playwright suites — twelve tests asserting
against bytes on disk, including the publish refusal guard — and they
run only when someone dispatches them by hand. A break in the desktop
experience, app-frame, or the shell can merge on a green PR and only
surface at release-candidate time. The survey’s new console-watch
fixture (every spec now fails on a stray console error, pageerror, or
failed request) makes a green run mean something again, which is what
makes gating worth deciding now.
| Location | Role today |
|---|---|
.github/workflows/agent-pants-ci.yml:41-43 | on: workflow_dispatch: / pull_request: — but its paths: filter (lines 44-49) covers only solutions/agent-pants/** and packages/rust/agent-adapt/**; it never runs the determined product’s Playwright suites |
.github/workflows/release-candidate.yml:21-22 | on: workflow_dispatch: only — no push or pull_request trigger at all |
.github/workflows/release-candidate.yml:42-154 | The agent-pants-desktop-e2e job: builds the SPA, runs both Playwright projects (install, authoring), 12 tests total |
solutions/agent-pants/docs/vault/tasks/ci-covers-determined-agent-pants.md | The existing tracked task for this gap; still status: open |
solutions/determined/products/agent-pants/tests/e2e/consoleWatch.ts | New: every spec fails on a console error, pageerror, requestfailed, or an HTTP ≥ 400, unless named benign |
Proposed
Section titled “Proposed”A pull request that touches the agent-pants product, its experience,
app-frame, or the shell it depends on runs both Playwright projects and
fails the PR on a red or console-watch-flagged spec — the same job
release-candidate.yml already runs, moved onto a trigger that gates
merges instead of running only on demand.
Trade-off, left to the reader: this job costs roughly 45 minutes per run against the rest of the suite’s 20, so gating it on every PR buys real coverage at a real CI-time and cost increase — whether that trade is worth it, or whether a narrower trigger (e.g. only PRs touching the listed paths, as already scoped in the acceptance criteria below) is the right compromise, is the decision this task exists to make, not to assume.
Approach
Section titled “Approach”- Decide the trigger scope: full
pull_requestgating vs. path-filtered (solutions/determined/products/agent-pants/**,solutions/determined/experiences/agent-pants/**,packages/ts/app-frame/**,solutions/determined/packages/shell/**). - Move (or duplicate with the decided trigger) the
agent-pants-desktop-e2ejob fromrelease-candidate.ymlonto a workflow that runs onpull_request, keeping the SPA-build-then-test ordering so a spec never runs against a stale.output/public. - Confirm
consoleWatch.ts’s per-spec failure behavior is in effect for the CI run, not just local runs. - Update
ci-covers-determined-agent-pants.md’s acceptance criteria against what actually shipped, and close it if this task satisfies all of them.
| Area | Note |
|---|---|
.github/workflows/release-candidate.yml | Remove or narrow agent-pants-desktop-e2e once it moves |
.github/workflows/agent-pants-ci.yml | Or: add the e2e job here instead, per the decided trigger scope |
solutions/agent-pants/docs/vault/tasks/ci-covers-determined-agent-pants.md | Update acceptance criteria against what shipped; close if satisfied |
Acceptance criteria
Section titled “Acceptance criteria”- AC-1: A pull request touching the decided trigger paths runs both
Playwright projects (
install,authoring) and fails the PR on a red or console-watch-flagged spec. - AC-2: The job builds the product’s SPA in the same run that tests
it, never reusing a stale
.output/public. - AC-3:
release-candidate.ymlno longer duplicates a job that a PR-gating workflow already runs.
Out of scope
Section titled “Out of scope”- Writing new end-to-end coverage; this task runs what already exists.
- The CLI/wasm gate in
agent-pants-ci.yml’s existing job, which is unaffected.
Dependencies
Section titled “Dependencies”- none
Discovery context
Section titled “Discovery context”Found by the agent-pants bug survey, N-5LD4-bug-sweep-2026-09-12-agent-pants
(row AP-39), which found the tracked vault task
ci-covers-determined-agent-pants.md partially done: Playwright runs,
but only on manual dispatch.