T-A2C6-spawn-from-post-mortem-stronger-dedup

Status: closed/done · Impact: high · Complexity: small

Goal

plugin/entities/task/spawn-from-post-mortem.md Step 3 already does a keyword-grep dedup check before scaffolding a new task, but its scope is too narrow (only task headlines + Goal sections) and it leaves no audit trail. A bulk /sdlc:task-review run on 2026-05-21 surfaced 44 unfinished tasks and uncovered five clear near-duplicate / partial-overlap clusters (e.g. task-ensure-ready-flags-spec-placeholders vs task-spec-flags-schema-rejecting-placeholders, task-new-flags-cascading-schema-edits vs new-scripts-derive-patterns-from-schema) that the existing Step 3 should have caught at spawn time. Widening the search scope and recording the search outcome inline in the spawned task makes the dedup gate auditable and reduces the rate of overlap reaching /sdlc:task-review for cleanup.

Today

plugin/entities/task/spawn-from-post-mortem.md Step 3 reads:

For each kept bullet, grep <worktree>/docs/planning/tasks/ for any existing task that already covers this gap (keyword-search the bullet text against task headlines and Goal sections). If one exists: do not create a new task. Edit that existing task’s frontmatter to append the originating task’s basename to its related: array (de-duplicated). Validate the edited file (see step 5). Record this outcome as LINKED-EXISTING, not SPAWNED.

Three concrete weaknesses observed during the 2026-05-21 review:

1. Scope too narrow. The grep is documented as “headlines and Goal sections” only. A bullet that uses different wording than the existing task’s headline / Goal but matches its Today / Approach / AC text slips through.
2. No audit trail. When Step 3 returns “no match” the sub-agent just proceeds to Step 4 and scaffolds a new file. There is no record of what was searched, what was found, or why the result was “create new” instead of “link existing”. A reviewer running /sdlc:task-review later has no way to tell whether the dedup fired or was skipped.
3. Exact-keyword matching misses paraphrases. A bullet that says “ensure-ready misses TBD placeholders” doesn’t hit an existing task whose Goal says “spec-drift fragments slip past the implementation sub-agent” — they’re the same gap, but the words don’t overlap.

Proposed

Step 3 is rewritten to (a) search the full body of every unfinished task, not just headline + Goal, (b) emit a structured search-trail block recording the keywords searched and the top ranked candidates, and (c) embed that block in the new task’s ## Discovery context section so the result is preserved inline. The sub-agent makes the create-vs-link call from a ranked candidate list rather than a binary grep hit.

If the top candidate’s score crosses a “likely duplicate” threshold (operational definition: keyword overlap above N% AND the candidate’s status: is not closed/*), the sub-agent treats the bullet as a LINKED-EXISTING outcome and appends the originating basename to the candidate’s related: array instead of creating. If the score is below threshold, it proceeds to scaffold, but the search trail goes into the new task’s body so a later reviewer can second-guess the call without rerunning the search.

Approach

1. Replace plugin/entities/task/spawn-from-post-mortem.md Step 3 with a stronger procedure:

   • For each kept bullet, extract a small keyword set (3-5 significant nouns/verbs from the bullet, dropping stopwords).
   • For each unfinished task under <worktree>/docs/planning/tasks/ (status not starting with closed/), score the candidate by counting keyword occurrences across the entire file body (not just headline + Goal). Optionally weight headline matches higher.
   • Build a ranked list of the top 5 candidates with their scores. Decide create-vs-link from that list, not from a binary grep hit.

2. Extend the procedure to emit a structured search-trail block. Suggested format:

   ### Dedup search (spawn-from-post-mortem)
   
   Bullet: <verbatim post-mortem bullet>
   Keywords searched: foo, bar, baz
   Top candidates (score / status / headline):
     - 12 / open/ready / 2026-05-19-foo-task
     - 8 / planning/draft / 2026-05-18-bar-task
     - 3 / closed/done / 2026-05-15-baz-task
   Decision: <SPAWNED|LINKED-EXISTING <basename>|SKIPPED>
   Rationale: <one-line reason if the decision is non-obvious>

3. In Step 5 (body fill-out), embed the search-trail block at the end of the new task’s ## Discovery context section (under a ### Dedup search (spawn-from-post-mortem) H3). When the outcome is LINKED-EXISTING, append the same block as a comment in the candidate task’s ## Discovery context instead.

4. Add a test fixture under plugin/skills/task-work/tests/ (or in spawn-from-post-mortem’s own test harness, if one exists) that exercises a known-overlap pair — a post-mortem bullet that should link to an existing task — and asserts the dedup fires.

5. (absorbed) Add --exclude-basename <BASENAME> (repeatable, action="append") to dedup_search.py so the dedup search drops the originating task from candidates instead of self- linking. Wire spawn-from-post-mortem.md Step 3’s invocation to pass the originating basename. Render an Excluded: ... line in the search-trail block when the exclude set is non-empty. Add case_excludes_named_basename_from_candidates to test_dedup_search.py exercising the flag against a keyword-stuffed self-match fixture.

6. (absorbed) Add --emit-telemetry-line <PATH> and --worktree <PATH> to dedup_search.py. Each invocation appends one JSON line documenting decision, link_to, top_score, keyword_count, ratio, worktree, and excluded. Wire Step 3 to pass <worktree>/.claude/dedup-telemetry.jsonl (covered by .gitignore’s .claude/* rule). Ship companion script plugin/skills/task-work/summarize_dedup_telemetry.py that reads the JSONL log and prints a decision histogram bucketed by top-candidate score plus top-score / ratio / keyword-count percentiles.

7. (absorbed) Refactor test_dedup_search.py from a shared global fixture corpus to per-case fixture-population (option (b) from the fixture-isolation-lint follow-up). Each case builds its own minimal tempfile.TemporaryDirectory corpus so a fixture seeded for one case cannot interfere with another at scoring time.

8. (absorbed) Extract load_module(name, path) to a shared helper at plugin/skills/_test_support/load_module.py. The helper registers the module in sys.modules BEFORE exec_module so Python 3.14’s dataclass decoration finds it during exec — the registration is the load-bearing detail the next contributor would otherwise re-debug. Migrate test_dedup_search.py and test_start_task.py to use the helper. Helper ships its own self-test exercising a dataclass-bearing target.

Files to touch

• plugin/entities/task/spawn-from-post-mortem.md — rewrite Step 3, extend Step 5 with the search-trail emission, pass --exclude-basename and --emit-telemetry-line in the invocation example.
• plugin/skills/task-work/SKILL.md — Step 8’s prose may reference Step 3’s outputs; update if needed.
• plugin/skills/task-work/dedup_search.py — add the --exclude-basename and --emit-telemetry-line flags, the telemetry emit helper, and the Excluded: line in the rendered block.
• plugin/skills/task-work/summarize_dedup_telemetry.py (new) — read the JSONL log and print the histogram + percentiles.
• plugin/skills/task-work/test_dedup_search.py — known- overlap regression case, plus the new case_excludes_named_basename_from_candidates case, plus per-case fixture population, plus migration to the shared loader helper.
• plugin/skills/task-work/test_start_task.py — migrate to the shared loader helper.
• plugin/skills/_test_support/__init__.py (new) — package marker for the shared test-support module.
• plugin/skills/_test_support/load_module.py (new) — shared importlib loader that registers in sys.modules before exec_module.

Acceptance criteria

• AC-1: Re-running spawn-from-post-mortem against a known overlap (e.g. the task-ensure-ready-flags-spec-placeholders + task-spec-flags-schema-rejecting-placeholders pair) produces LINKED-EXISTING rather than SPAWNED, with the linked task named in the report.
• AC-2: Every task spawned by the rewritten procedure contains a ### Dedup search (spawn-from-post-mortem) block under ## Discovery context recording keywords, top candidates, and decision rationale.
• AC-3: A reviewer scanning a newly-spawned task can reconstruct the dedup call without re-running the search, using only the inline search-trail block.
• AC-4 (absorbed): dedup_search.py search --exclude-basename <X> ... returns a candidate list that contains no entry with basename == <X>; the rendered block names excluded basenames on an Excluded: ... line when the exclude set is non-empty; case_excludes_named_basename_from_candidates in test_dedup_search.py asserts on both shapes.
• AC-5 (absorbed): Each invocation of dedup_search.py search --emit-telemetry-line <path> ... appends exactly one valid JSON line to <path> with the documented fields (decision, link_to, top_score, keyword_count, ratio, worktree, excluded).
• AC-6 (absorbed): summarize_dedup_telemetry.py <path> prints a histogram of decisions bucketed by top-candidate score plus top-score / ratio / keyword-count percentiles.
• AC-7 (absorbed): A test file that loads a target script containing @dataclass(frozen=True) via the shared load_module helper does not raise AttributeError on Python 3.14; existing tests (test_start_task.py, test_dedup_search.py) keep passing after migration.
• AC-8 (absorbed): test_dedup_search.py cases each build their own minimal fixture corpus; re-introducing the original keyword-stuffed closed-placeholder fixture into one case cannot regress AC-1 in a sibling case.

Out of scope

• Migrating already-spawned tasks to retroactively include the search-trail block. Only new spawns gain it.
• Replacing keyword grep with semantic embedding search. A better grep + structured audit trail is enough; if the next bulk review still surfaces overlap clusters, that’s the follow-up.
• Auto-merging detected duplicates. The new gate still creates or links — it doesn’t restructure existing task bodies.

Dependencies

• none

Discovery context

Filed 2026-05-21 during a /sdlc:task-review walk-through of 44 unfinished tasks. The user observed that the post-mortem flow was filing too many near-duplicates and asked for an audit-trail dedup. The fix is here rather than in /sdlc:task-review because by the time tasks reach review the duplicate is already in the corpus — Step 3 is the only chokepoint that can stop a near-dupe from being filed in the first place.

Originally spawned 4 follow-up tasks 2026-05-21-{dedup-search-emits-telemetry,dedup-search-excludes-originating-task,fixture-isolation-lint-for-test-scoring,test-scaffold-sys-modules-registration}; user requested rollup into this PR via review comments on PR #76; follow-up tasks marked closed/superseded.

Post-mortem

Captured by /sdlc:task-work on 2026-05-21. PR: pending.

Acceptance criteria coverage

• AC-1: auto — case_known_overlap_fires in plugin/skills/task-work/test_dedup_search.py plus a live run of dedup_search.py against the real corpus (top score 31 for task-ensure-ready-flags-spec-placeholders → LINKED-EXISTING).
• AC-2: auto — case_block_contains_keywords_and_decision asserts the rendered block carries the H3 header, keyword list, top-candidate lines, and decision; the rewritten spawn-from-post-mortem.md Step 5 mandates appending the verbatim block under ## Discovery context.
• AC-3: auto — same test case asserts the block contains the bullet, keywords, candidate scores with status + headline, and decision — everything needed to reconstruct the dedup call without re-running.

What worked

• The known-overlap pair scored 31 vs 21 against the live corpus, well above both threshold guards — no tuning iteration needed.
• The closed-top-candidate guard fell out cleanly from the decision rule; the negative test fixture isolates that branch without coupling it to placeholder vocabulary.
• Mirroring test_start_task.py’s importlib-load pattern made the unit-test scaffolding nearly mechanical.

Friction and automation gaps

• importlib.util.spec_from_file_location + @dataclass raised a confusing AttributeError: 'NoneType' object has no attribute '__dict__' on Python 3.14 because the dynamically-loaded module wasn’t registered in sys.modules before exec — fix was a one-liner sys.modules[name] = mod, but the error surface was opaque. The sibling test_start_task.py doesn’t hit this only because start_task.py has no dataclasses. Per-skill test scaffolding should standardize the sys.modules registration step so the next contributor doesn’t re-debug it. → T-B1Y3-test-scaffold-sys-modules-registration
• The fixture corpus needed two iterations: the first closed-fixture (keyword-stuffed with placeholder vocabulary) outscored the live overlap pair and failed AC-1 in test. Splitting the closed-top-candidate case onto its own non-overlapping vocabulary (streaming-ingest) made each test case independent. A “fixture isolation” lint — or a per-case fixture-population helper — would have caught the cross-test interference at write time rather than at red-test time. → T-92QH-fixture-isolation-lint-for-test-scoring
• The dedup script ships defaults (threshold-min-score 6, threshold-ratio 0.5) tuned against a tiny calibration set; there’s no telemetry hook to record how often Step 3 returns LINKED-EXISTING vs SPAWNED in production runs so the thresholds can be revisited empirically. A --emit-telemetry-line flag (or append-only log) would close the calibration loop. → T-5A82-dedup-search-emits-telemetry
• The dedup_search.py script does not exclude the originating task from its candidate corpus, so a post-mortem bullet inevitably scores the originating task as the top match (because the bullet’s keywords come from text that now lives in that task). All three spawned bullets here returned LINKED-EXISTING <self> and required manual override to SPAWNED. The script needs an --exclude-basename flag passed by the spawning sub-agent. → T-HC03-dedup-search-excludes-originating-task

Spawned follow-up tasks

• T-B1Y3-test-scaffold-sys-modules-registration — standardize the sys.modules registration loader pattern, created.
• T-92QH-fixture-isolation-lint-for-test-scoring — cross-fixture interference detection for scoring-based test suites, created.
• T-5A82-dedup-search-emits-telemetry — append-only telemetry log so dedup thresholds can be recalibrated empirically, created.
• T-HC03-dedup-search-excludes-originating-task — add --exclude-basename flag so the dedup search drops the originating task from candidates instead of self-linking, created (discovered during this run’s own spawn step).

---

← Back to Tasks