Skip to content

fix(csv-parse): count UTF-16LE line endings as code units - #542

Open
pauloflp wants to merge 1 commit into
adaltas:masterfrom
pauloflp:fix/utf16-line-counting-20261025
Open

pauloflp wants to merge 1 commit into
adaltas:masterfrom
pauloflp:fix/utf16-line-counting-20261025

Conversation

@pauloflp

Copy link
Copy Markdown

UTF-16LE characters can contain 0x0a or 0x0d without containing a newline. The parser currently counts those individual bytes as line breaks, so info.lines and error locations become incorrect, from_line starts too early, and to_line can discard valid records.

Minimal reproduction:

import { parse } from './packages/csv-parse/lib/sync.js';

parse(Buffer.from('a,b\n\u010a,c', 'utf16le'), {
  encoding: 'utf16le', to_line: 2,
});
// Before: [['a', 'b']]
// Expected and after: [['a', 'b'], ['Ċ', 'c']]

Count CR/LF only when they form a complete UTF-16LE code unit at an aligned byte offset. Determine the encoding after BOM detection, and use the cumulative byte offset so alignment remains correct across arbitrary stream chunks. Other encodings retain their existing line-counting path.

Add nine regression tests covering CR/LF bytes in either byte of a character, unaligned byte pairs, encoding aliases, genuine LF/CR/CRLF record endings, quoted newlines, comments, error locations, from_line, to_line, and one-byte writes with explicit encoding or BOM detection.

Validation on Node.js v24.22.1:

  • All nine new tests fail without the source fix.
  • npm run test --workspace=csv-parse: 638 passing, 2 pending; TypeScript passes.
  • Changed-file ESLint and git diff --check pass.
  • npm run build --workspace=csv-parse passes.
  • Additional local matrix: 2,596 checks pass, comprising all 1,018 non-CR/LF BMP code units containing a CR/LF byte with and without BOM, plus 560 two-chunk splits of representative fixtures.
  • Four sync API smoke checks pass against the generated CommonJS and browser ESM bundles, using each bundle's Buffer implementation.

This PR targets line counting. Record/field delimiter matching at unaligned offsets is a separate existing issue; the alignment-specific line-count regression uses | to isolate that behavior. AI-assisted implementation and local verification.

@wdavidw

wdavidw commented Oct 26, 2026

Copy link
Copy Markdown
Member

Is this not somehow related to #545 ?

@pauloflp

Copy link
Copy Markdown
Author

Is this not somehow related to #545 ?

Yes, they are related! Both address UTF-16LE byte alignment, but at different stages of parsing.

#545 fixes delimiter matching, preventing unaligned bytes from being interpreted as field or record separators.

#542 fixes the separate physical line counter, which can incorrectly count CR/LF bytes within ordinary UTF-16LE characters. This affects info.lines, from_line, to_line, and error locations.

The fixes are complementary, and the tests target different behaviors.

Happy to coordinate or adjust the implementation if needed. Thanks for pointing this out!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants