Lossless round-trip markdown IO
Status: open/planned · Kind: subsystem · Audience: system
Summary
Section titled “Summary”- A rowan-style lossless CST (TM-0016-concrete-syntax-tree, TM-0017-red-green-tree) in which every byte — whitespace, markers, comments, frontmatter — is a token, so untouched content reserializes byte-identically.
- One event-emitting parser with pluggable sinks: the CST is the fidelity sink; an mdast-shaped read-only sink and a frontmatter-only byte-scan serve read paths where fidelity is wasted work — one grammar, tiered materialization.
- A four-operation write model over one atomic+CAS storage seam: guarded TM-0021-swap-write / TM-0020-splice-write / TM-0022-canonical-write, plus TM-0023-external-change (observed, reconciled); splice and canonical never interchangeable.
- The corpus gate (TM-0019-corpus-gate: S-0007 fixtures, rumdl’s 660-file precedent) guards every release.
Statement
Section titled “Statement”Comment- and line-preserving write-back is the requirement no existing markdown tool
meets: comrak, pulldown-cmark, and markdown-rs all normalize on re-serialization.
Losslessness is a property of the tree, not the parser — so the core is a concrete
syntax tree in the red-green pattern (rust-analyzer, taplo), built over a
byte-accounted tokenizer, from which semantic views are derived. Edits replace CST
nodes; serialization concatenates leaves; a document read and written without semantic
change is byte-identical, including YAML comments inside frontmatter — the failure mode
ontogen’s markdown-store documents as unsolved today.
What it provides
Section titled “What it provides”- Parse to lossless CST; reserialize byte-identically.
- Typed document DOM and a read-only mdast-compatible view derived from the CST.
- Read tiers under one grammar: frontmatter-only byte-scan, event→AST fast path, event→CST fidelity path.
- Splice writes: minimal text edits from CST node replacement, atomic write with CAS
(etag) semantics per
determined-vault’s discipline. - Canonical writes for machine-owned files, formatter-stable.
- Swap writes: full-content replacement from editor buffers, CAS-guarded, validation advisory (a human save is never blocked).
- External-change reconciliation: self-echo classification via the write-hash journal; bulk invalidation for git-scale change.
- Position/anchor fidelity for
^block-id, wikilinks, transclusions per D-0008-markdown-standard.
Underlying implementation
Section titled “Underlying implementation”New build (M-C6XN-lossless-cst-core) — no off-the-shelf crate exists. Inputs:
rowan/taplo (tree pattern), markdown-rs (tokenization rules), determined-vault
(frontmatter.rs byte-span splice, write.rs atomic+CAS, the splice/canonical
regression test), markdown-contract D-0016 (per-node raw/mdast/typed three-view
design), rumdl corpus gate (D-0011).