fix: keep the form a code block was authored with

CHANGELOG.md

@@ -45,6 +45,8 @@ Leafdown uses lightweight [Keep a Changelog](https://keepachangelog.com/en/1.1.0
 - Keep the address of a URL or email address written on its own when a `*`, `_`, or `~` follows it, so text such as `https://example.com*` keeps its link pointing where it did. The backslash the file writes to keep that marker literal was being read back as part of the address, which gained another backslash every time the document was opened and saved.
 - Keep a URL or email address written on its own bare when a run shaped like a character reference but naming nothing, such as `&notarealentity;`, follows it, so text such as `https://example.com&notarealentity;` is saved as it was written instead of gaining angle brackets. Markdown leaves such a run outside the link whether or not the name exists.
 - Keep a URL or email address written on its own bare when a literal `<` or `>` sits beside it, so text such as `\<test@example.com>` or `&lt;https://example.com&gt;` is saved as it was written, instead of putting angle brackets around it and saving `<<…>>`, which the next open reads as an angle-bracket URL between two literal brackets.
+- Keep a code block written in the form it was authored, so a block indented with four spaces stays indented instead of being rewritten as a fenced block on the first save, at the top level, inside a list item, and inside a quote. Fencing an indented block did not stop at the block either: content holding a fence of its own forced a longer fence around it and rewrote the lines that followed.
+- Keep the fence a code block was written with, so a `~~~` block stays a `~~~` block instead of becoming a ` ``` ` one, a fence written longer than it needs to be keeps its length, a fence indented up to three spaces keeps its indentation, and the spaces or tabs before an info string are kept. A `~~~` block may name a language containing a backtick where a ` ``` ` block may not, so rewriting the fence also rewrote that language as `&#x60;` and changed what other Markdown tools read. A block left unclosed at the end of the file stays unclosed, and so does one left unclosed at the end of a quote, a list item, or a footnote definition, because the block around it is what ends it; it is closed again as soon as something follows it inside that block. A block made in the editor is still written with ` ``` `.
 - Keep a horizontal rule written the way it was authored, so `---`, `_ _ _`, or any other accepted run stays as it is instead of being rewritten as `***` on the first save. A rule inserted from the editor is still written as `***`, and so is one whose own run would be read back as a heading underline or as part of its list item's bullet.
 - Keep the form a reference definition was written in, so `[field report]: <field-report.md>` keeps the angle brackets around its destination and a definition whose title was written on the line below it keeps that line, instead of both being rewritten onto one bare line on the first save. The spaces or tabs written after the colon and before the title are kept as well. A destination that cannot be read back without angle brackets still gets them, and a definition the editor creates is written on one line.
 - Keep a reference link, a reference image, and the definitions they point at, instead of rewriting every reference as an inline link carrying its own copy of the destination and deleting the definition block on the first save. A definition now appears in the document as the line it is written as, and a reference shows that reference source when the caret reaches it.

docs/decisions.md

@@ -150,6 +150,9 @@
 - A character reference is decoded by `micromark` before the mdast text node exists, so `©` and `©` are indistinguishable to everything downstream and a file written to stay ASCII does not stay ASCII. Leafdown records the authored form, decided in [issue #262](https://github.com/Azganoth/leafdown/issues/262), and writes it back in text and in link and image destinations alike. The run is recovered by walking each text node's value against the slice of the file it was built from, and carried on a mark whose stored source is verified against the text it covers before it is written, so an edit that invalidates it degrades to the character rather than to a stale reference. References written next to each other keep one mark each, decided in [issue #305](https://github.com/Azganoth/leafdown/issues/305), so breaking one converts only that one and leaves its neighbours preserved. ProseMirror merges neighbouring text nodes carrying an equal mark set, so a repeated reference still arrives as one node holding its characters repeated; only an equal mark merges, which makes that node whole repetitions of the one source it stores, and the verification counts them rather than reading the node as a reference the source does not spell. A preserved reference is inert for escaping: it opens no construct and closes none, and the escape passes read it as the characters it will be written as. That same verified source is what a caret reaching the reference projects, decided in [issue #298](https://github.com/Azganoth/leafdown/issues/298) on the rule [Offer the escape gesture only where the conversion exists](#offer-the-escape-gesture-only-where-the-conversion-exists) states, because breaking a valid reference commits the literal text it spells and the conversion therefore exists. This is the exception the byte-identity target in [issue #251](https://github.com/Azganoth/leafdown/issues/251) would otherwise have had to admit, and it is overridden rather than accepted, unlike the strikethrough run below, because a reference and the character it names are not interchangeable to an author who chose one.
 - The preset's single heading form is overridden. Its `heading` node carries only the level, which is all an ATX and a setext heading have in common, so both parse into the same node and are written back as an ATX heading with nothing closing it, rewriting every closed and every underlined heading in a file on its first save. Leafdown records the form on the node, decided in [issue #316](https://github.com/Azganoth/leafdown/issues/316), read from the slice of the file the node was built from: an ATX heading is one line, a setext heading ends on its underline, and only the second spans more than one, so the slice also names which form it holds. What is kept is the run closing an ATX heading, the spaces or tabs opening it, and the length of a setext underline; the underline's own character answers for the level rather than the file, so a heading moved between levels one and two is underlined by the character that level reads back as. `mdast-util-to-markdown` settles both forms from one option for the whole document and sizes each run from the content it just wrote, so the option carries the authored form for the length of the heading and the runs are put back on the handler's own output. A heading the editor creates is written as ATX with one space and nothing closing it, which is also what a recorded form gives way to where the lines it lands on would not be read back as the heading: a setext underline carries only levels one and two, and a setext heading written after a paragraph in a tight list item is joined to it by a single newline, which leaves its content read as more of that paragraph and its underline covering both. The blank line the serializer writes between two headings belongs to the blank-line class rather than to this one, so `corpus/commonmark/blocks.md` loses its heading-form differences without reaching byte identity.
 - The preset's single thematic break spelling is overridden. Its `hr` node carries no attributes, so `***`, `---`, `_ _ _`, and every other accepted run parse into the same node and are written back as `***`, rewriting every break in a file on its first save. Leafdown records the run on the node, decided in [issue #319](https://github.com/Azganoth/leafdown/issues/319), read from the slice of the file the node was built from, which is the whole of a break because it holds no children. Indentation stands outside that slice and the whitespace closing the line is trimmed off it, so what is kept is the characters and the spacing between them, tabs included. A break the editor creates carries `***`, which is also what a recorded run gives way to where the line it lands on would be read back as something other than a break. `mdast-util-to-markdown` joins a tight list item's children with a single newline, so a run of hyphens written after a paragraph there underlines it into a setext heading; and a run sharing its item's bullet character stands on the bullet's line, where the two read as one longer break with no list around them. The serializer already moves the bullet off the rule character it was configured with, but that character cannot answer for a run the node carries, so the run is what gives way rather than the bullet.
+- The preset's single code block form is overridden. Its `code` node carries a value and an info string, which is all an indented and a fenced block have in common, and `mdast-util-to-markdown` picks one form and one fence spelling for the whole document, so every indented block was rewritten as a backtick fence and every tilde fence rewritten as a backtick one. Leafdown records the form on the node, decided in [issue #321](https://github.com/Azganoth/leafdown/issues/321) for the choice between the two forms and [issue #320](https://github.com/Azganoth/leafdown/issues/320) for the way a fence is spelled, read from the slice of the file the node was built from: an indented block's slice opens on the line its indentation is written on and a fence's opens at the fence itself, so indented code can never stand on a fence run and the head of the slice names the form. What is kept for a fence is the character, the spacing before the info string, the indentation up to the three spaces CommonMark still reads a fence under, and whether the file closed it. The tilde is the spelling that carries content the backtick cannot: an info string may hold a backtick only when the fence is spelled with tildes, so rewriting the fence forced the info string to be written as ``` and changed what another tool reads. A block the editor creates is written as a backtick fence, which is also what a recorded form gives way to where the block can no longer be written in it: an indented block cannot carry an info string, open or close on a blank line, or hold nothing but whitespace, and an open fence runs to the end of the block that holds it, so a block that stops standing last there is closed. A fence is recorded open only where it stands last in its container, which is the only place one can be written open: recorded anywhere else it names a form the file can never hold, and the record flips on the save that closes it. The container is what ends it, so a fence the file leaves open at the end of a blockquote, a list item, or a footnote definition stays open and the blocks after that container stay outside the code, which is the reading `corpus/interactions.md` states under `An unclosed fence ends with its containing block`.
+- A fence's length is kept as the surplus over the shortest run that can hold its content rather than as the run the file spelled. A fence has to outrun anything inside it, so the length is a floor the content can raise at any time, and recording the number itself made the record shift whenever an edit — or a file whose own parse leaves a fence unclosed, as `corpus/commonmark/code.md` does — forced a wider run than the file was written with. The surplus survives that, because it is measured against the same floor on the way back in.
+- A fence's indentation is kept only at the document root, where the column the fence opens at is the indentation and nothing else. Inside a container that column also counts the prefix the container wrote, and mdast names neither separately. Deriving the prefix from the block's own lines was measured and rejected: it assumes the container writes the same prefix on the line the block opens on as on the lines under it, which a footnote definition does not — it writes its label on the first and indents the rest by four, so the difference read as indentation the file never wrote and was written into the code the block holds. A form axis that can reach the content is narrower than one that cannot, under the ordering in [Preserve the form a file was written in](#preserve-the-form-a-file-was-written-in), so a fence inside a container keeps no indentation of its own. That costs bytes rather than content, and it is the one axis of this class Typora 1.14.9 keeps and Leafdown does not.
 - The preset's outer table pipes are overridden. `mdast-util-gfm-table` calls `markdown-table` with the alignment, the padding, and the cell width it was configured with and never with `delimiterStart` or `delimiterEnd`, and exposes neither as a setting, so a table authored in GFM's pipe-optional form is written back with an outer pipe on both sides of every row. Leafdown records which outer pipes the rows carry, decided in [issue #349](https://github.com/Azganoth/leafdown/issues/349), read from the slice of the file each row was built from, and writes them from a `table` handler of its own. A table the editor creates carries both pipes, which is also what a recorded form gives way to where the rows it now holds would not be read back from the form. A blank cell at either end of a row leaves the written row opening or closing on a pipe of its own, which GFM strips before it splits the row, moving every cell after it one column; and a delimiter cell is as wide as its column, so a first column one character wide is written `-`, which opens a bullet list item where no pipe precedes it. Whether a table carries outer pipes is a property of the table rather than a layout computed across its cells, which is what separates it from the padding the consequence above normalizes: it survives an edit to any cell. The delimiter row is no node of its own, so the form is read off the rows that are, and a table whose rows disagree keeps the pipe rather than taking it off the rows that carry one.
 - The preset's strikethrough delimiter run is not preserved. Its strikethrough mark carries no marker attribute, unlike emphasis and strong, so a single-tilde run parses and serializes back as a double-tilde run. This is normalized on cost under [Preserve the form a file was written in](#preserve-the-form-a-file-was-written-in) rather than overridden as the autolink form was, because both runs mean the same thing to a GFM reader. Preserving the authored run would require carrying the marker on the mark.
 - The preset's strikethrough input rule is overridden. Its `(~{1,2})` backtracks to a one-tilde delimiter run when no two-tilde closing run exists yet, and its content group does not exclude the marker, so typing `~~text~~` created a mark over `~text` on the seventh keystroke and left a surplus tilde on each side that saved as an escaped character. Leafdown carries its own rule, decided in [issue #233](https://github.com/Azganoth/leafdown/issues/233), which excludes the marker from the content and anchors the match at the caret so a run stays literal text until the author closes it. This is the only input rule Leafdown owns; every other preset rule either anchors at the caret or excludes its own marker, and none of them can match a run this way.

src/features/editor/plugins/codeForm.ts

@@ -0,0 +1,56 @@
+import type { MarkdownNode } from "@milkdown/kit/transformer";
+import { $remark } from "@milkdown/kit/utils";
+
+import {
+  CODE_CLOSED_ATTRIBUTE_NAME,
+  CODE_FENCE_ATTRIBUTE_NAME,
+  CODE_FENCE_SURPLUS_ATTRIBUTE_NAME,
+  CODE_FENCED_ATTRIBUTE_NAME,
+  CODE_INDENT_ATTRIBUTE_NAME,
+  CODE_MARKDOWN_TYPE,
+  CODE_SEPARATOR_ATTRIBUTE_NAME,
+  findCodeForm,
+} from "../utils/codeMarkdown";
+
+// The parse records the value and the info string, and neither the two forms of code block nor the
+// fences that spell one differ in either, so the form survives only in the slice of the file the
+// node was built from. A node the parser gave no position keeps the defaults.
+const markAuthoredForm = (node: MarkdownNode, source: string, atRoot: boolean) => {
+  const children = node.children ?? [];
+
+  for (const child of children) {
+    const start = child.position?.start.offset;
+    const end = child.position?.end.offset;
+    const column = child.position?.start.column;
+
+    if (
+      child.type === CODE_MARKDOWN_TYPE &&
+      start !== undefined &&
+      end !== undefined &&
+      column !== undefined
+    ) {
+      const form = findCodeForm({
+        raw: source.slice(start, end),
+        value: (child.value as string | undefined) ?? "",
+        column,
+        atRoot,
+        endsDocument: child === children[children.length - 1],
+      });
+      const authored = child as Record<string, unknown>;
+
+      authored[CODE_FENCED_ATTRIBUTE_NAME] = form.fenced;
+      authored[CODE_FENCE_ATTRIBUTE_NAME] = form.fence;
+      authored[CODE_FENCE_SURPLUS_ATTRIBUTE_NAME] = form.fenceSurplus;
+      authored[CODE_SEPARATOR_ATTRIBUTE_NAME] = form.separator;
+      authored[CODE_INDENT_ATTRIBUTE_NAME] = form.indent;
+      authored[CODE_CLOSED_ATTRIBUTE_NAME] = form.closed;
+    }
+
+    markAuthoredForm(child, source, false);
+  }
+};
+
+export const createLeafdownCodeFormPlugin = () =>
+  $remark("leafdownCodeForm", () => () => (tree, file) => {
+    markAuthoredForm(tree as MarkdownNode, String(file), true);
+  });

src/features/editor/tests/corpusRoundTrip.test.ts

@@ -16,6 +16,7 @@ const byteIdenticalFiles = [
   "commonmark/html.md",
   "gfm/tagfilter.md",
   "isolated/end-of-file/incomplete-html-comment.md",
+  "isolated/end-of-file/unclosed-code-fence.md",
   "isolated/end-of-file/unclosed-directive.md",
   "isolated/end-of-file/unclosed-html-block.md",
 ];
@@ -33,7 +34,6 @@ const convergingFiles = [
   "gfm/strikethrough.md",
   "gfm/tables.md",
   "gfm/task-lists.md",
-  "isolated/end-of-file/unclosed-code-fence.md",
 ];
 
 const corpusFiles = [...byteIdenticalFiles, ...convergingFiles];