Lossless markdown CST core

Status: open/draft

Goal

Build the syntax core, staged splice-first. Phase 1: a byte-accurate event parser (TM-0018-byte-accurate-parser), a span-level splice edit engine (TM-0020-splice-write), and a comment-preserving yaml-splice frontmatter editor, all behind the corpus gate (TM-0019-corpus-gate) — this alone unblocks reshape link-rewrites and every frontmatter consumer. Phase 2: the rowan-style lossless CST sink (TM-0016-concrete-syntax-tree, TM-0017-red-green-tree) with typed DOM and mdast view (TM-0024-mdast), gated on a named consumer for structured synthesis (insert-section / wrap-in-list class APIs) — the same demand test applied everywhere else. Home: the markdown-contract workspace (default per the open question in D-7VMX-markdown-ecosystem-strategy), implementing its D-0016 per-node source-fidelity design.

Success criteria

• Phase 2 opens with a feasibility gate (review R-3/PE-4): prototype and compare a full rowan CST against the event parser + source-backed structured edits on mutation fidelity, semantic differentials, malformed-input recovery, position equivalence, fuzzing, and named performance/memory budgets. Stop-or-narrow is a legitimate outcome that keeps the phase-1 wins.
• Parse→serialize with no edits is byte-identical across the combined corpus: S-0007 fixtures, the rumdl 660-file corpus, determined-vault’s frontmatter fixtures, and a sampled real vault.
• CommonMark + GFM + YAML frontmatter + wikilinks/anchors/transclusions (D-0008-markdown-standard dialect) parse with positions; fault-tolerant on malformed input.
• Splice edits emit minimal text edits from exact spans; YAML comments and key order inside frontmatter survive value edits via yaml-splice (the failure ontogen’s markdown-store documents).
• Phase 2: typed DOM + read-only mdast-compatible JSON projection derived from the CST, exercised by the contract engine reading through it (one parse, no re-parse).
• canonical write mode exists and is formatter-stable for machine-owned files.
• Named consumer runs on it end-to-end (sdlc entity read path or determined-vault frontmatter ops) before the milestone closes.

Deliverables

• Syntax crate: one grammar, event-emitting parser with pluggable sinks — the seam that later admits a read-only AST sink without a second grammar.
• Splice edit engine over exact spans (phase 1).
• Consume the pure yaml-splice crate defined by D-7N7I-pure-functional-yaml-editing; its extraction may land before this milestone. Complete any CST-integration work and missing conformance cases (phase 1).
• Consume the tier-0 markdown-util crate extracted by M-6TQL-markdown-util-extraction (its byte-exact frontmatter split and fidelity types); that extraction may land before this milestone. Add any lexing helpers the CST paths demand beyond what it already provides (phase 1).
• CST sink (tokenizer per markdown-rs/CommonMark rules, green/red tree, error recovery) — phase 2, gated on a named structured-synthesis consumer; AST sink deferred until profiling justifies it.
• Typed DOM + mdast projection (phase 2).
• Splice/canonical writer with atomic+CAS file discipline.
• Byte-identity corpus gate in CI.

Risks / open questions

• Tokenizer scope creep: CommonMark’s edge cases are the schedule risk; the corpus gate, not spec-completeness, defines done.
• Annado’s line-splice write-back is the naive baseline: lossless around the touched line, lossy on it, fragile line-number addressing. The CST splice must beat exactly those two failure modes or it isn’t earning its complexity.
• Whether determined-vault’s byte-span splice remains as a fast path or is replaced outright.

---

← Back to Milestones