Skip to content

UTF-16LE delimiters match at unaligned byte offsets #543

Open

Description

@pauloflp

Describe the bug

csv-parse can recognize a record or field delimiter starting halfway through a UTF-16LE code unit. Valid characters are then split into empty records or fields, losing the original value.

Reproduced on upstream master 9fabcbe70ac6d3561db8c11fef3a567f7917cdc2 (package.json version 7.0.3), Node.js v24.22.1.

To Reproduce

Run from a repository checkout:

import { parse } from './packages/csv-parse/lib/sync.js';

const cases = [
  ['\u0a01\u0100', { record_delimiter: '\n' }],
  ['\u0d01\u0100', { record_delimiter: '\r' }],
  ['\u2c01\u0100', { delimiter: ',' }],
];

for (const [value, options] of cases) {
  const bytes = Buffer.from(value, 'utf16le');
  console.log({
    bytes: bytes.toString('hex'),
    expected: [[value]],
    actual: parse(bytes, { encoding: 'utf16le', ...options }),
  });
}

Actual outputs, in order:

// 010a0001: [[''], ['']]
// 010d0001: [[''], ['']]
// 012c0001: [['', '']]

Each input should remain one record containing the two original characters in one field. For the first example, the LF pattern 0a 00 occurs at byte offset 1 across 01 0a | 00 01; it is not a newline code unit. CR and comma exhibit the same behavior.

Additional context

All three failures also reproduce with one-byte stream writes. The same three values parse correctly as UTF-8 controls. Delimiters are explicitly configured; this does not depend on automatic delimiter discovery, BOM detection, or trimming.

Source inspection suggests __isRecordDelimiter() and __isDelimiter() accept these unaligned byte matches. This delimiter-recognition behavior also remains when only the separate line-counting change in #542 is applied.

AI-assisted reproduction and source inspection; all six failing executions and three UTF-8 controls were run locally.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions