T-ZU6V-new-scripts-derive-patterns-from-schema

Status: closed/done · Impact: low · Complexity: small

Auto-generated from PR #18 review feedback (DRY concern on TASK_OR_EPIC_RE regex duplication). Review and promote to ready before picking up.

Goal

plugin/scripts/new_*.py scripts hard-code regex patterns (e.g. TASK_BASENAME_RE, TASK_OR_EPIC_RE, RELATED_RE) that mirror patterns already declared in the corresponding plugin/entities/*/schema.json. The duplication means a schema change has to be hand-propagated to every script that validates user input — and the originating PR (#18) introduced exactly this pattern when it added TASK_OR_EPIC_RE to new_milestone.py. Closing this gap makes the schema the single source of truth for these patterns and removes a class of cascading-edit bugs.

Cited by T-J2CW-add-epic-entity-task-depends-on-dependencies (PR review comment on new_milestone.py regex addition).

Today

plugin/scripts/new_task.py, new_milestone.py, and new_epic.py each define a handful of module-level regex constants (SLUG_RE, DATE_RE, TASK_BASENAME_RE, TASK_OR_EPIC_RE, RELATED_RE, SEMVER_RE) used to validate CLI flag values before writing the output file. The same patterns appear in plugin/entities/*/schema.json under properties.*.pattern. Adding a new accepted shape (as PR #18 did for epic ids in milestone membership) requires editing both places; missing one means the CLI rejects legitimate input or the schema accepts illegitimate input — they drift.

Reference example (PR #18):

# plugin/scripts/new_milestone.py:49
TASK_OR_EPIC_RE = re.compile(r"^(E\d{4}|\d{4}-\d{2}-\d{2}-[a-z0-9]+(?:-[a-z0-9]+)*)$")

The same regex appears in plugin/entities/milestone/schema.json under properties.tasks.items.pattern (wrapped in ...).

Proposed

Each new_*.py script resolves its validation patterns from the corresponding schema.json at startup rather than hard-coding them. Approach: direct schema load (option a). Each script reads its own ../entities/<type>/schema.json at import time, walks the JSONPath of interest (e.g. properties.tasks.items.pattern), and compiles the resulting regex. Strips the wikilink \[\[...\]\] wrapping when the schema uses wikilink-wrapped patterns but the CLI takes bare values.

Rejected alternatives:

(b) Shared helper module at plugin/scripts/_patterns.py. Worth revisiting when a third script (e.g. an audit helper) wants the same regex; today it’d be premature abstraction.
(c) Code-generation of a _patterns.py constants module via plugin/scripts/regen_patterns.py. Heaviest option; adds a CI step for a problem direct schema load already solves.

Approach

For each new_*.py script, replace each hard-coded re.compile(...) constant that mirrors a schema pattern with a small helper that loads the matching entities/<type>/schema.json, extracts properties.<field>.pattern (or properties.<field>.items.pattern), unwraps \[\[(.*?)\]\] if present, and re.compile()s the inner pattern. Cache by path so each schema loads at most once per process.
Walk every _RE constant in new_task.py, new_milestone.py, new_epic.py (at plugin/skills/epic-new/new_epic.py), and new_backlog.py. Categorise: schema-derived, script-internal (e.g. HTML_COMMENT_AT_TOP_RE), or unclassifiable. Refactor the schema-derived ones. new_backlog.py confirmed to carry schema-derived constants (TASK_WIKILINK_RE, MILESTONE_WIKILINK_RE, BACKLOG_WIKILINK_RE) — included unconditionally.
Add a regression test: load each schema’s pattern fields and the corresponding script-loaded regex; assert identical pattern strings (modulo wikilink unwrapping).
Smoke test procedure for AC-3:
- Temporarily edit plugin/entities/milestone/schema.json to widen TASK_OR_EPIC_RE (e.g. add F\d{4} for fictional “feature” ids).
- Run plugin/scripts/new_milestone.py --task F0001 ... and confirm the CLI now accepts the new shape WITHOUT touching new_milestone.py.
- Revert the schema edit; confirm the CLI rejects again.

Files to touch

plugin/scripts/new_task.py — replace schema-derived _RE constants with schema loader.
plugin/scripts/new_milestone.py — same; specifically remove TASK_OR_EPIC_RE and TASK_BASENAME_RE in favor of a schema lookup.
plugin/skills/epic-new/new_epic.py — same; TASK_BASENAME_RE, RELATED_RE. (Script is co-located with its skill; NOT at plugin/scripts/new_epic.py.)
plugin/scripts/new_backlog.py — same treatment for TASK_WIKILINK_RE, MILESTONE_WIKILINK_RE, BACKLOG_WIKILINK_RE.

Acceptance criteria

AC-1: No new_*.py script defines a regex constant whose pattern duplicates a string already present in a sibling schema.json (modulo wikilink wrapping). Verifiable by grep + manual inspection.
AC-2: Each script still rejects malformed inputs at CLI time (e.g. new_milestone.py --task not-a-real-slug exits non-zero). Manual smoke tests or fixture-driven.
AC-3: Per the smoke procedure in Approach step 4: a temporary schema widening (add F\d{4} to plugin/entities/milestone/schema.json’s TASK_OR_EPIC pattern) makes new_milestone.py --task F0001 succeed without any source edit to the script. Revert restores rejection.

Out of scope

Refactoring validate_frontmatter.py itself (it already uses the schema directly; this task is only about the new_*.py scripts catching up).
Pushing the pattern-derivation pattern out to other Python helpers (audit_entities.py, migrate_entities.py) — those load schemas via a different code path and don’t have the same duplication risk.
Adopting a richer schema-binding library (e.g. pydantic models generated from JSON Schema) — out of scope for a duplication-removal refactor.

Dependencies

none

Discovery context

Spawned from PR #18 review feedback. The reviewer flagged TASK_OR_EPIC_RE in new_milestone.py as duplicating the schema’s pattern: “Should these be hardcoded into the python or can these patterns be derived from the schema so we avoid duplication?” The duplication was introduced by the sub-agent as a CLI-side guard during epic-id support; fixing the underlying DRY issue is a clean follow-up rather than a blocker for PR #18.

Post-mortem

Captured by /sdlc:task-work on 2026-05-21. PR: pending.

Acceptance criteria coverage

AC-1: agent-manual — command grep "re.compile" across all four scripts; only SLUG_RE remains hard-coded, and it’s a filename-component validator with no schema source (no entity schema declares a bare slug field with a pattern).
AC-2: agent-manual — ran each script against malformed inputs in /tmp/test-cli-rejection/. Each rejected with exit 2 and a human-readable error message; new_milestone.py --task F0001 rejected pre-widening.
AC-3: agent-manual — temporarily added F\d{4} to plugin/entities/milestone/schema.json’s tasks.items.pattern, ran new_milestone.py --task F0001 --title T and got exit 0 pointing at the freshly-created M0001.md. Reverted schema, re-ran, got rejection back.
(also added) auto: plugin/scripts/test_schema_patterns.py — loads each schema-derived regex through the helper and asserts 10 groups’ worth of accept / reject cases. Run via uv run.

What worked

The helper’s unwrap_wikilink flag cleanly separated “schema stores wikilink form, CLI takes bare” from “schema stores bare form” — one parameter covered the three callers that needed it.
Hoisting the conditional result.pattern walker into a separate schema_conditional_result_pattern helper kept the simple schema_pattern callers from carrying conditional-status noise they don’t need.
Confirming with a real-world schema widening (the F0001 smoke test) caught zero issues — the regex layer was right on first try.

Friction and automation gaps

Wikilink-unwrap regex got two false starts. The schema’s JSON-escaped \\[ becomes \[ in the in-memory Python string, but my first two regex attempts treated the escaped form differently. A small unit test for the unwrap helper alone (separately from end-to-end pattern tests) would have surfaced this faster than python3 -c REPL-style debugging.
TASK_BASENAME_RE in new_milestone.py was defined but never referenced — dead code. A lint pass that flags module-level-defined-but-unused names would have caught this before review. ruff F841 covers function-scope unused; module-scope needs vulture or similar. → T-LQQL-lint-unused-module-level-names
Cross-directory module loading via importlib.util.spec_from_file_location is boilerplate-heavy (5 lines + an assert). If a third script outside plugin/scripts/ needs this, the boilerplate should move into a tiny one-liner helper. (Skipped — current call sites: one. Revisit on third caller per “co-locate first, promote when shared”.)

Spawned follow-up tasks

T-LQQL-lint-unused-module-level-names — created. Closes the gap that let TASK_BASENAME_RE ship dead.

← Back to Tasks