fix: keep the form a code block was authored with - #379
Merged
Conversation
The parse gives both code forms one node carrying a value and an info string, and `mdast-util-to-markdown` picks one form for the whole document, so every block was written fenced. The two forms agree on everything the node carries, so the form survives only in the slice of the file the node was built from. The head of that slice names it. An indented block's position opens on the line its indentation is written on, while a fence's opens at the fence itself, past whatever indentation the file gave it, so indented code can never stand on a fence run: its own four spaces stand there first. Inside a container the position opens past the container's prefix, which leaves the same reading. The form reaches the serializer through the `fences` option rather than being written directly, because the handler's indented branch also holds the conditions CommonMark puts on that form. A block carrying an info string, opening or closing on a blank line, or holding nothing but whitespace cannot be written indented, and giving way to a fence there is what keeps an edit inside the block from writing a file that reads back as something else. The rewrite did not stop at the block either: indented content holding a fence of its own forced a longer outer run, rewriting the lines around a construct the author never wrote.
`mdast-util-to-markdown` spells every fence from one option for the whole document and sizes each run from the content it just wrote, so a tilde fence came back as a backtick fence, indentation and info-string spacing were dropped, and a fence the file left open gained a closing run. The tilde is the spelling that carries content the backtick cannot: an info string may hold a backtick only when the fence is spelled with tildes, so rewriting the fence forced the info string to be written as ``` and changed what another tool reads. Setting the option to the recorded character is what puts the literal backtick back, because the serializer marks a backtick unsafe only inside a grave-accent fence. The length is kept as the surplus over the shortest run that can hold the content, not as the run the file spelled. A fence has to outrun anything inside it, so the length is a floor the content can raise at any time, and recording the number itself made the record shift on the save that raised it. `corpus/commonmark/code.md` reaches that case without an edit: a backtick in a backtick info string opens no fence, so the rest of that file parses as a fence holding runs of its own, and the run has to widen. Measured against the same floor on the way back in, the surplus survives it. A fence is recorded open only where the block ends the document, which is the only place one can be written open. Recorded anywhere else it names a form the file can never hold, and the record flips on the save that closes it, which is what a fence left open inside a blockquote does. Indentation is measured against the document root, where the column the fence opens at is the indentation. Inside a container that column also counts the prefix the container wrote, and mdast names neither separately, so the prefix is taken from the narrowest line the block holds: CommonMark strips up to the fence's own indentation from each content line, which leaves that line spelling the prefix alone wherever one line was written without the indentation. A block indented uniformly reads the indentation as narrower and is written with less of it, which costs bytes rather than content, because CommonMark strips whatever the fence is written with back off on the way in. An indented fence no longer opens the line it is written on, so the pair of blocks it belongs to is measured against a line that can carry up to three spaces before the run.
A fence is ended by whatever ends the block it stands in, not only by the end of the file, so one the file leaves open at the end of a blockquote, a list item, or a footnote definition can be written open and the blocks after that container still read outside the code. Closing it there spent a rewrite on a form the file already held, and it was the one code-block difference Typora 1.14.9 does not produce. Deriving a container's prefix from the block's own lines is withdrawn. It assumed the container writes the same prefix on the line the block opens on as on the lines under it, which a footnote definition does not: it writes its label on the first and indents the rest by four, so the difference read as indentation the file never wrote and was written into the code the block holds. The corpus reaches no fence inside a footnote definition, so nothing guarding it caught this. A form axis that can reach the content is narrower than one that cannot, so a fence keeps its indentation only at the document root, where the column it opens at is the indentation and nothing else. That leaves a quoted indented fence written flush, which costs bytes rather than content.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A save rewrote every code block into one shape. Indented code became a fenced block, a tilde fence became a backtick fence, the space between an opening fence and its info string was dropped, a fence indented up to three spaces moved to column zero, and a fence the author left unclosed gained a closing run. The tilde rewrite is the damaging one: an info string may hold a backtick only when the fence is spelled with tildes, so changing the fence forced the info string to change too and
~~~ language+ backtick saved as a backtick fence carrying`. The block still read back as the same code, but the info string another tool reads was no longer the one the author wrote.Both forms parse into one
codenode carrying a value and an info string, andmdast-util-to-markdownpicks one form and one fence spelling for the whole document, so nothing on the node answered for either. The form survives only in the slice of the file the node was built from, and the head of that slice names it: an indented block's position opens on the line its indentation is written on, while a fence's opens at the fence itself, past whatever indentation the file gave it, so indented code can never stand on a fence run — its own four spaces stand there first. That holds inside a container too, where the position opens past the container's prefix rather than at the line, which is what #321 asked to settle before the fence axes were worth attempting.Both choices reach the serializer through
mdast-util-to-markdown's own options rather than being written directly, because its indented branch also holds the conditions CommonMark puts on that form. A block carrying an info string, opening or closing on a blank line, or holding nothing but whitespace cannot be written indented and gives way to a fence, which is what keeps an edit inside a block from writing a file that reads back as something else. Setting the fence option to the recorded character is also what puts the literal backtick back, because the serializer marks a backtick unsafe only inside a grave-accent fence — no unescaping pass is involved.The length is kept as the surplus over the shortest run that can hold the content rather than as the run the file spelled. A fence has to outrun anything inside it, so the length is a floor the content can raise at any time, and recording the number itself made the record shift on the save that raised it.
corpus/commonmark/code.mdreaches that without an edit: a backtick in a backtick info string opens no fence, so the rest of that file parses as one fence holding backtick runs of its own and the run has to widen from three to four. Measured against the same floor on the way back in, the surplus survives it.An open fence is ended by whatever ends the block it stands in rather than only by the end of the file, so one left open at the end of a blockquote, a list item, or a footnote definition is written open and the blocks after that container still read outside the code. It is recorded open only where it stands last in its container, which is the only place one can be written open — recorded anywhere else it names a form the file can never hold and the record flips on the save that closes it.
Indentation is kept only at the document root, where the column the fence opens at is the indentation and nothing else. Inside a container that column also counts the prefix the container wrote and mdast names neither separately, so a fence inside a container keeps no indentation of its own. Deriving the prefix from the block's own lines was implemented, measured, and withdrawn: it assumes the container writes the same prefix on the line the block opens on as on the lines under it, which a footnote definition does not — it writes its label on the first line and indents the rest by four, so the difference read as indentation the file never wrote and was written into the code the block holds. A form axis that can reach the content is narrower than one that cannot.
Related Issue
Closes #320
Closes #321
Verification
markdownCompatibility.test.tsgains aCode block formblock of 34 cases. Seven cover the indented form — at the top level, holding a blank line, holding a fence of its own, and inside a list item, a blockquote, and a list item holding a blockquote — and each was confirmed to fail without the change, reproducing #321's reported output including the```` cascade. Ten cover the fence axes the same way: both fence characters, a run longer than the content needs, both spacings before an info string, a tilde info string holding a literal backtick, and one- and three-space indentation.The rest are guards. A block made by
insert.codeBlockand byformat.codeBlockwrites the default fence; an indented block whose content grows a leading blank line gives way to a fence; a recorded run rises when content outgrows it and a recorded surplus stays above the run the content needs; a fence left open at the end of a blockquote, a list item, or a list item holding a blockquote stays open; an unclosed fence is closed once a block follows it inside its container; a fence standing alone with no final newline is not read as its own closing run; and a fence inside a footnote definition keeps clear of the label's own width, which is the case that withdrew the container prefix derivation. The container cases assert the reopened document alongside the written bytes, because byte equality cannot see a form record that flips or indentation pushed into content.corpusRoundTrip.test.tsmovesisolated/end-of-file/unclosed-code-fence.mdtobyteIdenticalFiles, completing that directory at four of four, which is what #320 named as the reason to separate the unclosed case.Two existing cases in
markdownCompatibility.test.tsasserted the old behavior — an indented-code first child inside a list item written back as a fence. They are folded into the list of non-paragraph first children that now round-trip unchanged, which keeps the coverage they existed for (the block stays a first child inside its item) under a stricter assertion.Measured across all 82 Markdown files under
corpus/, mounting each, saving, and remounting: againstorigin/mainthe only status change isisolated/end-of-file/unclosed-code-fence.mdmoving from rewritten to byte-identical. Every other file converges and preserves its document exactly as it did before, including the hand-verified subtrees the automated guard does not cover.extensions/frontmatter/yaml.mdandpractical/extension-survey.mdfail document preservation onorigin/mainas well and are untouched by this change.Compared against Typora 1.14.9, the editor
Preserve the form a file was written incites, on three fixtures covering twelve form cases. Manually verified on Windows 11:pnpm check:frontendpasses.Not verified: no manual pass in the assembled application, and
pnpm check:backendwas not run. The change is parse-time and serializer behavior with no rendered surface of its own, and it touches no Rust or Tauri code.Notes
language+one© ``` is written back decoded — that is the character-reference class under A character reference is decoded and saved as the literal character #262.Four spaces do not interrupt an existing paragraph.loses its indentation, which Indented code is rewritten as a fenced block on save #321 puts out of scope as a read-time difference no serializer change reaches. Typora keeps it, because its model holds the raw line rather than the parsed paragraph; that is a parser difference rather than a serializer one.corpus/commonmark/lists-and-blockquotes.mdloses its code-block differences entirely; what remains is one lazy continuation line gaining its block marker, which is A lazy continuation line gains its block marker on save #322.corpus/commonmark/code.mdstays converging on the classes above plus the escaped backtick runs on itsA backtick in a backtick info string does not open a fenceline. Saving inserts a blank line between blocks the author wrote adjacent #324 is already merged, so it is no longer what either file is waiting on.blockSeparatorMarkdown.tsnow reads a code fence under up to three spaces. A fence never carried its own indentation into that guard before, and without the widening the blank line before an indented fence could no longer be dropped, which shifted a recorded separator incorpus/commonmark/code.md.withCodeBlockSeparatoris replaced bywithCodeForm, so the code block carries the block separator alongside its own form the way every block Leafdown holds a form for already does.parseDOMandtoDOM, so a block copied out of one document and pasted into another is written with the default fence. That matches how a heading's authored form already behaves, and the fallback is the safe direction.