T-92QH-fixture-isolation-lint-for-test-scoring
Status: closed/superseded · Impact: low · Complexity: small
Auto-generated from a /sdlc:task-work post-mortem. Review and
promote to open/ready before picking up.
When a test suite scores candidates against a shared fixture
corpus (the pattern in
plugin/skills/task-work/test_dedup_search.py — every test
case sees every fixture, the case asserts on which fixture wins),
a fixture seeded for one case can silently outscore the
expected winner in a sibling case. This was hit during the
originating task: a keyword-stuffed closed fixture
intended for case_closed_top_candidate_spawns_new outscored
the live overlap pair in case_known_overlap_fires, causing
AC-1 to red. The interference is not visible at write time —
each case looks fine in isolation. A lint or fixture-population
helper that flags cross-case fixture interference would catch
this class of bug before red-test time. Cited by
T-A2C6-spawn-from-post-mortem-stronger-dedup.
test_dedup_search.py populates fixtures once via
_seed_fixtures(tasks_dir) and runs every case against the same
corpus. Cases are independent in their assertions but coupled
through the shared fixtures. Nothing surfaces the coupling at
write time; the red test does, but only after a
debugging round-trip.
Proposed
Section titled “Proposed”Either:
(a) A small lint that, for each test case asserting “candidate X wins for bullet Y”, re-runs the scoring with X removed and asserts the next-best winner is still consistent with the intended discrimination — i.e. the assertion relies on X being uniquely-best, not coincidentally-best.
OR
(b) A per-case fixture-population helper that lets each case declare its own minimal fixture subset, so cases don’t share a global corpus by default. Higher rigor, more boilerplate.
Pick whichever fits the existing harness style better and delivers the catch-at-write-time property.
Approach
Section titled “Approach”- Survey other scoring-based test suites in the plugin
(
plugin/skills/import-planning/tests/,plugin/skills/entities-audit/tests/) — do any share the global-fixture pattern and have the same exposure? - Decide between (a) and (b) above. The lint approach generalizes; the per-case fixtures are clearer per-test.
- Implement against
test_dedup_search.pyfirst; if it pays off, generalize.
Files to touch
Section titled “Files to touch”plugin/skills/task-work/test_dedup_search.py— refactored to whichever approach wins.- Possibly
plugin/scripts/if a generalized helper emerges.
Acceptance criteria
Section titled “Acceptance criteria”- AC-1: Re-introducing the original keyword-stuffed closed-placeholder fixture into the suite either fails at write time (lint approach) or fails only in the case that needs that fixture (per-case approach), instead of silently red-failing AC-1.
Out of scope
Section titled “Out of scope”- Rewriting every scoring-based test suite in the plugin in one PR. Validate on the one that motivated the task first.
Dependencies
Section titled “Dependencies”- none
Discovery context
Section titled “Discovery context”Spawned by /sdlc:task-work post-mortem of T-A2C6-spawn-from-post-mortem-stronger-dedup on 2026-05-21.
Dedup search (spawn-from-post-mortem)
Section titled “Dedup search (spawn-from-post-mortem)”Bullet: First closed-fixture (keyword-stuffed with placeholder vocabulary) outscored the live overlap pair and failed AC-1 in test. A fixture isolation lint or a per-case fixture-population helper would have caught the cross-test interference at write time Keywords searched: keyword-stuffed, interference, fixture-population, vocabulary, placeholder, outscored, cross-test, isolation Top candidates (score / status / headline):
- 11 / in-progress / 2026-05-21-spawn-from-post-mortem-stronger-dedup — Strengthen spawn-from-post-mortem dedup and record the search trail (self-match; excluded from decision)
- 10 / in-progress / 2026-05-20-task-ensure-ready-flags-spec-placeholders — task-ensure-ready flags TBD and ‘final name’ placeholders in spec body
- 2 / closed/done / 2026-05-19-templates-html-comment-after-frontmatter — Move template HTML comment block after frontmatter Decision: SPAWNED Rationale: After excluding the self-match, the next-best (score 10) is the placeholder-flagging task — adjacent vocabulary but a different gap (readiness-gate placeholder detection, not test-fixture isolation). Below an unambiguous link threshold; SPAWNED is correct.