Skip to content

fix(remote-helper): push silently drops or corrupts files containing non-ASCII text - #59

Open
am009 wants to merge 1 commit into
aloth:mainfrom
am009:patch-1
Open

am009 wants to merge 1 commit into
aloth:mainfrom
am009:patch-1

Conversation

@am009

@am009 am009 commented Oct 4, 2026

Copy link
Copy Markdown

Summary

git push through git-remote-overleaf silently drops files whose contents contain
non-ASCII characters, e.g. any UTF-8 LaTeX source with Chinese comments. In a 12-file
test tree the first push reported success, but only 6 files reached the project:
main.tex, ref.bib, build.sh, .gitignore and two section files never arrived, and
the file deleted in that commit was not deleted either. Content that does get through is
mangled, because the same code path re-encodes it as latin1.

Root cause

parseFastExport() reads the fast-export stream line by line and re-joins the lines of a
data <N> section (src/remote-helper.ts:189):

const lineBytes = Buffer.from(line + '\n', 'binary');   // latin1
dataBuffer = Buffer.concat([dataBuffer, lineBytes]);
if (dataBuffer.length >= pendingDataBytes) { ... }

data <N> counts UTF-8 bytes, but 'binary' (latin1) encodes every character as a
single byte. For a blob containing non-ASCII text the accumulated buffer is systematically
short, so the >= pendingDataBytes check fires late: the parser stays in the data state
and swallows the following M <mode> :<mark> <path> / blob / mark lines as blob
content. Files whose M line is consumed that way are never added to files and never
uploaded.

Deficits measured on the test tree (UTF-8 length vs. latin1 length):

file UTF-8 latin1 deficit
.gitignore 305 209 96
build.sh 1,792 1,462 330
main.tex 4,686 3,704 982
ref.bib 58,376 58,322 54

git pull / git fetch are not affected: the import path writes entry.getData()
buffers directly to stdout.

Fix

One line, so the byte count matches data <N> again:

-      const lineBytes = Buffer.from(line + '\n', 'binary');
+      const lineBytes = Buffer.from(line + '\n', 'utf8');

Verification

  • Reproduced on a 12-file project: first push uploaded 6 files, left the deleted file in
    place, and olcli diff still reported 6 missing files.
  • After the change, cleared .git/overleaf/marks, .git/overleaf/state.json and
    refs/overleaf/main and pushed again. The debug log (GIT_REMOTE_OVERLEAF_DEBUG=1)
    shows all 12 files uploaded and the deleted path removed.
  • Downloaded the project zip afterwards and compared every file byte for byte with the
    local working tree: 12/12 identical, including the files with Chinese text, and no
    remote-only files left. olcli diff reports no differences.

Repro

mkdir repro && cd repro && git init
printf '%% 中文注释\nhello\n' > main.tex
git add main.tex && git commit -m init
git remote add origin overleaf::https://www.overleaf.com/project/<project-id>
git push -u origin main
# main.tex is missing from the project (or arrives mangled)

Scope

This fixes text content (valid UTF-8), which is what Overleaf projects contain. Genuinely
binary blobs are still decoded by readline as UTF-8, so PNG/PDF payloads remain
unsupported on push; fixing that properly means reading data <N> as raw bytes instead of
going through readline. I left that out to keep this change minimal — happy to follow up
if you want it.

There is no automated test here: parseFastExport() is not exported, and the existing
tests cover the client/config/ignore layers. I can add a regression test if you are open
to exporting the function (or extracting the stream parser).

Change buffer encoding from binary to UTF-8 for data accumulation.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant