Markdown dialect and compatibility profile

Status: open/proposed

Summary

• One normative dialect and compatibility profile for the markdown ecosystem: the construct set, malformed-input policy, position model, and document framing (BOM, line endings). The profile — not any tree materialization — is what implementations must agree on.
• The shared conformance corpus is this decision’s executable appendix: current consumer behavior is captured as fixtures before any consumer is migrated.
• Extracted from D-7VMX-markdown-ecosystem-strategy per review remarks R-1 and EW-2 (see the ADR’s review-discussion.md); the ADR keeps the architecture, this decision owns the semantics.

Context

Consumers do not share one behavior today. apps/sdlc/lib/util/frontmatter.ts uses a regex fence boundary and absorbs trailing whitespace; determined-vault’s frontmatter.rs handles BOM/CRLF and treats malformed YAML as a recoverable diagnostic — but only a malformed payload: a malformed fence yields no diagnostic at all, which is the asymmetry item 6 exists to close; vault-graph resolves link suffixes case-insensitively with shallow-first ambiguity handling; obsidian-astro’s remark plugin recognizes a narrower link set with a different dangling-link policy. Parser consolidation must not decide these policies by accident.

Decision

1. Compatibility before consolidation. Before a consumer is migrated onto the shared core, its current parse, position, finding, and write behavior is captured in the conformance corpus. Tier-0 byte scans, event/AST reads, and CST reads may coexist only where they pass the same relevant semantic fixtures.

2. UTF-8 BOM policy (relocated from the strategy ADR). A byte-order mark is document framing, not a frontmatter feature. When EF BB BF / U+FEFF occurs at byte zero, every read tier recognizes it before parsing markdown or a leading frontmatter fence. swap and splice preserve an existing BOM byte-for-byte and never add or remove one; newly generated canonical documents omit it by default. Byte offsets remain file-relative, so the first authored token begins at byte 3 in a UTF-8-BOM file. U+FEFF anywhere else is ordinary document content. YAML-payload BOM rules inside frontmatter belong to D-7N7I-pure-functional-yaml-editing.

3. Corpus scope. The first corpus pins: frontmatter recognition and raw spans (BOM, CRLF, empty/malformed blocks); parsed frontmatter values and diagnostics; wikilink/embed enumeration and resolution including ambiguity; byte positions and Rust-byte ↔ TS-position conversion; no-op and edited output bytes; finding IDs, severities, locations, and intentionally stable messages.

4. Line endings. Existing mixed endings are never normalized by read or splice paths; inserted material uses the document’s first observed ending (LF when none).

5. Framing convergence is decided, not deferred. The BOM, line-ending, and leading-whitespace framing above is one normative behavior every consumer targets, Rust or TypeScript. The splitters that trim framing away diverge by accident, not intent, and converge onto it. apps/ontological-legacy’s split_frontmatter and determined-polish’s extract_frontmatter have since converged onto markdown_util::split_lenient; vaultpolish-core’s extract_frontmatter is the last hand-rolled splitter outstanding. For those consumers the conformance corpus proves the migration reaches the unified framing (an intentional, reviewed behavior change) rather than freezing today’s trimming. TypeScript consumers reach it through the WASM surface or a corpus-matched port, not for free. This settles the framing axis only; link identity (below) stays open.

6. Malformed input is two questions, not one (answers Q-2). Framing malformation is a broken --- fence — unterminated block, missing terminator, stray whitespace on a fence line — where no YAML parse is attempted. Payload malformation is a well-formed fence whose YAML fails to parse or has a non-mapping root. The two get different answers. Conflating them is what produces today’s most misleading output: determined-contracts::check_note reads a padded closing fence as no frontmatter and then reports every required key as missing, for a note whose frontmatter is intact.

   operation	framing malformed	payload malformed
   read	body-only, plus a diagnostic	empty mapping, plus a diagnostic
   validate	one frontmatter/unterminated finding	one frontmatter/unparsable finding
   index	body-only; diagnostic reaches the report	body-only; diagnostic reaches the report
   splice	refuse	refuse
   reshape	refuse the file, record it, run goes on	refuse the file, record it, run goes on

   Validate emits one diagnostic finding and suppresses the required-key cascade behind it: a note whose frontmatter could not be read has not failed its shape, it has failed to parse, and reporting both buries the cause. Four invariants hold across every cell:

   1. No single note aborts a bulk operation. A projection over thousands of notes degrades; it never throws. This is the tier-0 rule (markdown-util’s “a projection over thousands of notes must degrade, never abort”) lifted to every operation and both languages.
   2. Every degradation is observable. A silent fallback is prohibited. Code that recovers from malformed input records why, and the record survives to the operation’s output — a diagnostic that is computed and dropped does not satisfy this.
   3. “Absent” and “broken” are distinct states. A note with no frontmatter and a note whose frontmatter is unreadable must never be represented identically. Every degraded value carries which one it was.
   4. A write never regenerates a block it could not parse. Refusal is the default. Regeneration requires an explicit opt-in at the call site and a validating gate before the bytes land — determined-notes’ compose_frontmatter is the one sanctioned instance and the shape others follow.

   Diagnostics on read and index paths are Option<String>-grade today and carry no id, severity, or position. Structured findings are required only where a finding is the operation’s product (validate); elsewhere the diagnostic must be present and propagated, and upgrading it to a Finding is left to the corpus work in item 3.

Compliance gaps

Item 6 is not descriptive of today. These sites violate it, surveyed at e879bc9e4; the two marked live are user-visible defects, the two marked latent are hazards in pub APIs with no production callers yet.

site	violates	behavior
markdown-contract dist/runner/corpus.js:121, via vault-contracts/src/check.ts:113	1	live — doc.toJS() throws on an unresolved or circular YAML alias, uncaught, so one note ends the whole vault:lint run with a stack trace
apps/sdlc/lib/services/lease/ops/task/sweep.ts:151 → lib/model/read.ts:185	1	live — same throw aborts the lease sweep; readEntity’s docstring at read.ts:168 claims it “never throws on document problems”
markdown-contract dist/core/projection.js:254	2, validate row	doc.errors is never read anywhere in the package; malformed YAML is silently repaired and validated as authored, with no finding id for it
determined-index/src/walk.rs:134	2	WalkedNote.parse_error is set and never read outside its own test, while the sibling unreadable list is plumbed to the report
determined-vault/src/frontmatter.rs:49	3	framing malformation yields yaml: None with parse_error: None — indistinguishable from a note that genuinely has no frontmatter
determined-contracts/src/validate.rs:126	3, validate row	strict split in a file that is otherwise a byte-parity port of a remark-based node arm; a padded fence validates as {} in Rust and as real data in TypeScript
apps/sdlc/lib/util/frontmatter.ts:226	3	parseFrontmatter collapses four distinct outcomes to null; every bulk TypeScript reader consumes the collapsed form
determined-vault/src/frontmatter.rs:87	4, splice row	latent — unwrap_or("") conflates “no block” with “broken block”, prepending a second fenced block onto an unterminated one
packages/rust/yaml-splice/src/splice.rs:202	4	latent — unwrap_or_default() drops an unparseable existing block and regenerates from the incoming mapping; the adjacent line 203 propagates the incoming parse error

The conformance corpus of items 1 and 3 does not exist yet. The only conformance artifact is packages/rust/markdown-util/tests/conformance.rs — Rust-only, framing-only. Nothing pins payload-malformation behavior or finding ids across the two arms, which is why the divergences above went unnoticed.

Open questions

• Q-1 (dialect): the exact CommonMark, GFM, YAML-frontmatter, and Obsidian constructs normative in v1, and how extensions are versioned.
• Q-3 (positions): the public position model across Rust and TypeScript — UTF-8 byte offsets internally is settled; the boundary representation (UTF-16 code units vs converted structure) and CRLF/invalid-UTF-8 handling are not.
• Link identity: case sensitivity, suffix resolution, and ambiguity policy — today’s divergent implementations need one adjudicated answer.

References

D-7VMX-markdown-ecosystem-strategy and its review-discussion.md (R-1, EW-2, Q-1..Q-3); divergence evidence cited in Context.

---

← Back to Decisions