Lossless round-trip markdown IO
State: open/building · Kind: subsystem · Audience: system
Summary
Section titled “Summary”- A rowan-style lossless CST (TM-0016-concrete-syntax-tree, TM-0017-red-green-tree) in which every byte — whitespace, markers, comments, frontmatter — is a token, so untouched content reserializes byte-identically.
- One event-emitting parser with pluggable sinks: the CST is the fidelity sink; an mdast-shaped read-only sink and a frontmatter-only byte-scan serve read paths where fidelity is wasted work — one grammar, tiered materialization.
- A four-operation write model over one atomic+CAS storage seam: guarded TM-0021-swap-write / TM-0020-splice-write / TM-0022-canonical-write, plus TM-0023-external-change (observed, reconciled); splice and canonical never interchangeable.
- The corpus gate (TM-0019-corpus-gate: S-0007 fixtures, rumdl’s 660-file precedent) guards every release.
Statement
Section titled “Statement”Comment- and line-preserving write-back is the requirement no existing markdown tool
meets: comrak, pulldown-cmark, and markdown-rs all normalize on re-serialization.
Losslessness is a property of the tree, not the parser — so the core is a concrete
syntax tree in the red-green pattern (rust-analyzer, taplo), built over a
byte-accounted tokenizer, from which semantic views are derived. Edits replace CST
nodes; serialization concatenates leaves; a document read and written without semantic
change is byte-identical, including YAML comments inside frontmatter — the failure mode
ontogen’s markdown-store documents as unsolved today.
What it provides
Section titled “What it provides”- Parse to lossless CST; reserialize byte-identically.
- Typed document DOM and a read-only mdast-compatible view derived from the CST.
- Read tiers under one grammar: frontmatter-only byte-scan, event→AST fast path, event→CST fidelity path.
- Splice writes: minimal, span-preserving edits to top-level YAML frontmatter fields, byte-identical outside the edited span.
- Canonical writes for machine-owned files, formatter-stable.
- Swap writes: full-content replacement from editor buffers, CAS-guarded, validation advisory (a human save is never blocked).
- External-change reconciliation: self-echo classification via the write-hash journal; bulk invalidation for git-scale change.
- Position/anchor fidelity for
^block-id, wikilinks, transclusions per D-0008-markdown-standard.
Shipped: the frontmatter byte-scan (markdown-util) and YAML splice/canonical writes
(yaml-splice). The rest waits on the body CST (M-C6XN-lossless-cst-core).
Underlying implementation
Section titled “Underlying implementation”Shipped foundation: packages/rust/markdown-util/ (the frontmatter-only byte-scan
tier) and packages/rust/yaml-splice/ (span-preserving splice writes and the
canonical YAML value emitter), each with a TypeScript peer
(packages/ts/markdown-util/, packages/ts/yaml-splice/) and, for yaml-splice, a
WASM boundary (packages/rust/yaml-splice-wasm/).
The rowan-style body CST is not built (M-C6XN-lossless-cst-core). Its planned
inputs: rowan/taplo (tree pattern), markdown-rs (tokenization rules),
determined-vault (frontmatter.rs byte-span splice, write.rs atomic+CAS, the
splice/canonical regression test), markdown-contract D-0016 (per-node
raw/mdast/typed three-view design), rumdl corpus gate (D-0011).
Locations
Section titled “Locations”packages/rust/markdown-util/packages/ts/markdown-util/packages/rust/yaml-splice/packages/rust/yaml-splice-wasm/packages/ts/yaml-splice/packages/rust/atomic-write/packages/rust/atomic-write/src/lib.rs#write_atomicpackages/rust/atomic-write/src/lib.rs#write_cas