Lec2LaTeX Pipeline

Converts lecture videos (Bulgarian audio + handwritten whiteboard) into clean, compilable Bulgarian LaTeX notes. Instead of a frame-by-frame transcript it aligns spoken content with whiteboard states, reads the board with a vision-language model, verifies the math, and runs a self-correcting LaTeX compile loop.

The pipeline is local-first and dual-backend: every model-using stage can run on-device or in the cloud, on either an Apple-Silicon (Metal) machine or a CUDA/Linux box, and each stage is switched independently.

The repository has two layers. src/ is the pipeline that turns video into a first draft. lectures/ and scripts/ are the edited book that draft became — see The lecture book below.

probability-statistics-bg.pdf   THE BOOK — 172 pages, the deliverable
lectures/                       its LaTeX sources (and the 15 standalone PDFs)
scripts/                        build, table generation, reference check, OCR
src/                            the video→draft pipeline
run/                            pipeline artefacts per lecture — the evidence
refs/                           source PDFs the book was cross-checked against
docs/REMEDIATION.md             editing rules, defect log, decisions

The book PDF is not built in place: scripts/build_lectures.py compiles inside lectures/ and then moves the result to the repository root under its real name. scripts/check_refs.py does the same, so whichever you run last, the root PDF is the current one.

Architecture

Video
 ├─ Stage 1  Transcribe (device-aware ASR)              → audio/transcript.json
 ├─ Stage 2  Board detection (rectify optional)         → board_corners.json
 ├─ Stage 3  Board keyframes (dual-SSIM)                → board/*.png, board_states.json
 ├─ Stage 4  VLM whole-board OCR (→ text + LaTeX)       → ocr/*.json
 ├─ Stage 5  Temporal alignment (speech ↔ board state)  → ir/lecture_ir.json
 ├─ Stage 6  Math verification (SymPy + reasoning LLM)  → ir/lecture_ir.json (+report)
 └─ Stage 7  Generation + Tectonic self-correct loop    → latex/lecture.{tex,pdf}

Stage 4 replaces the former PaddleOCR + pix2tex + IoU-evolution stage (which was unreliable on handwriting): one VLM call reads the whole board into structured {text|equation} items. Stage 6 is new. Compilation uses Tectonic (a single cross-platform binary) with a Docker/pdflatex fallback.

The lecture book (lectures/)

The pipeline's raw output is a starting point, not a deliverable. lectures/ carries the edited result: a 172-page Bulgarian book covering 14 lectures plus one combinatorics упражнение. On one shared counter it runs 151 numbered statements — 19 theorems, 60 definitions, 39 propositions, 8 corollaries, 2 lemmas and 23 supplements, 31 of them marked landmark — alongside 47 proofs, 44 examples and 22 TikZ figures. Two appendices close it: a formula sheet (Приложение А) and statistical tables (Приложение Б).

What is source and what is generated

lectures/bodies/lecture_NN.tex   SOURCE OF TRUTH — hand-edited
lectures/bodies/frontmatter.tex  legend + notation table
lectures/bodies/formulas.tex     appendix А, the formula sheet — hand-written
lectures/bodies/tables.tex       appendix Б — GENERATED by scripts/gen_tables.py
lectures/preamble.tex            shared style, theorem tiers, semantic macros
lectures/lecture_NN.tex          GENERATED — standalone drivers
lectures/lectures_full.tex       GENERATED — the book driver
probability-statistics-bg.pdf    GENERATED — the compiled book, at the root

Never edit the generated files; scripts/build_lectures.py overwrites them on every run.

.venv/bin/python scripts/gen_tables.py            # only when table content changes
.venv/bin/python scripts/build_lectures.py        # drivers + full book
.venv/bin/python scripts/build_lectures.py --all  # + 15 standalone PDFs
.venv/bin/python scripts/check_refs.py            # run after ANY substantial edit

Visual hierarchy

Four tiers, so the book can be skimmed rather than only read:

Tier	Looks like	Means
Landmark	gold, ★, thick bar	keythm / keydefn / keylem — the results to memorise
Standard	blue (defn) / green (thm, prop, lem, cor)	ordinary numbered statements
Skippable	hairline left rule	example, proof
Supplement	dashed violet, †	material not from the lecture itself

Editorial conventions

• Statements are numbered environments, never hand-typed bold labels. All of defn / thm / prop / lem / cor / supp share one counter, so Твърдение 4.7 is unambiguous and \label / \ref work.
• Proofs are \begin{proof}, always outside the statement box, closed by an automatic flush-right □.
• Landmark starring is a signal and dilutes if overused: 31 of 151 statements are starred (21%). Keep it near that.
• Defined terms in \emph{}. Bulgarian quotation marks „ “. Decimal comma, braced in math ($0{,}95$) so TeX does not set it as punctuation.
• Independence is written \perp\!\!\!\perp; \ind is the indicator.
• Course and exam logistics stay out of the mathematical flow (see the note at the end of lecture 14).

Editing rules

The notes are a transcription of a real lecturer, so docs/REMEDIATION.md fixes five ground rules (R1–R5) that edits must respect. In short: don't "correct" the lecturer silently — footnote instead; don't remove content; never strengthen a claim; expansion must recover what was said, not invent; verify against run/lecture_NN/audio/transcript.json and the board OCR in run/lecture_NN/ocr/ before acting. That file also logs the defects found so far and the findings that were investigated and rejected.

The handwritten notes as a third source

refs/pesho-stat-1.pdf is a handwritten theory notebook — 37 phone photos of Bulgarian cursive on tinted ruled paper. scripts/ocr_pesho.py preprocesses each page (crop the desk away, divide out the tint and the ruling, stretch the contrast) and reads it with a VLM into the same items-JSON the video pipeline produces, landing in run/pesho/ocr/. The run is resumable, so a dropped connection costs only the page in flight.

.venv/bin/python scripts/ocr_pesho.py --prep-only   # inspect the preprocessing
.venv/bin/python scripts/ocr_pesho.py --pages 1-4   # sample
.venv/bin/python scripts/ocr_pesho.py               # all 37 pages

The point of it is cross-validation: transcript, board OCR and a hand-written third account agreeing on a convention is much stronger evidence than any one of them alone. The full diff against lectures 1–12 lives in the commit message of d1351a3; conventions and formulas agreed throughout, which is exactly the result worth having on record.

refs/ — what the book was checked against

Four PDFs, kept out of the root so it shows the deliverable rather than the working material. Names are as received; the table says what each one is.

File
pesho-stat-1.pdf	37 pp.	the handwritten notebook, OCR'd above
(нередактирано) Лекции 2020 СЕМ Андрей-1-1.pdf	115 pp.	unedited 2020 notes for the same course
Записки по Теория на Вероятностите - Упражнения-1 поток-1.pdf	98 pp.	exercises, Каменов & Стоенчев (2018)
probability1BG-2.pdf	58 pp.	course summary, ФМИ СУ

The exercises PDF is what the formula sheet (appendix А) was aimed at: the formulas in it are the ones those problems actually need.

check_refs.py

Catches two failure modes that produce no ?? and no build error, so a normal compile looks clean while the document is broken:

• a \label that never reaches the .aux — the statement still renders and still numbers correctly, only the write is lost;
• a \crosslecture pointing at a label that no longer exists, which expands to \hyperref and quietly prints unlinked text.

Whether a given \label survives depends on how its tcolorbox breaks across pages, so an unrelated edit elsewhere in a chapter can silently knock one out. Run the checker after any substantial edit, not just after adding labels. The fix is to move the \label a few words into the first sentence.

Setup

Common

• Python 3.11, FFmpeg, yt-dlp (optional, for pulling lecture videos)
• tectonic for compilation: brew install tectonic (macOS) / see tectonic docs (Linux)

python3.11 -m venv --system-site-packages .venv
.venv/bin/pip install --upgrade pip

Metal profile (Apple Silicon)

.venv/bin/pip install opencv-python-headless scikit-image sympy antlr4-python3-runtime \
                      mlx-whisper mlx-vlm
# local text LLM (verify/gen) via either:
#   * mlx_lm.server  (serves an MLX model at http://localhost:8080/v1), or
#   * Ollama         (http://localhost:11434/v1)

Local model store defaults to /Users/g8row/models (override with LEC2TEX_MODELS_DIR). A --*-model that names a directory there is used directly.

CUDA profile (Linux)

.venv/bin/pip install faster-whisper opencv-python-headless scikit-image sympy \
                      antlr4-python3-runtime
# ASR: faster-whisper (cuda) or NVIDIA NeMo Canary-1b-v2 (--asr-backend nemo-canary)
# LLM/VLM local: vLLM or llama.cpp exposing an OpenAI-compatible endpoint

Running

# Fully local on Metal (ASR local; OCR/verify/gen point at local servers):
.venv/bin/python src/main.py --video lecture.mp4 --device metal --mode local

# Local-first with cloud OCR (no local VLM installed):
.venv/bin/python src/main.py --video lecture.mp4 --device metal \
    --mode local --ocr-mode cloud --ocr-provider agy

# Everything in the cloud via Antigravity:
.venv/bin/python src/main.py --video lecture.mp4 --mode cloud --provider agy

Per-step backend switching

Each model stage has its own --<step>-mode {local,cloud}, --<step>-model, --<step>-base-url, and --<step>-provider, where <step> ∈ asr | ocr | verify | gen. --mode sets the default for all steps; --device {auto,metal,cuda,cpu} picks the local runtime. Example — local ASR + cloud OCR + local verify/gen against a local MLX server:

.venv/bin/python src/main.py --video lecture.mp4 --device metal --mode local \
    --ocr-mode cloud --ocr-provider agy \
    --verify-base-url http://localhost:11434/v1 --verify-model qwen3:4b \
    --gen-base-url http://localhost:8080/v1

Recommended local runtime per role (2026)

Stage	Metal (M1)	CUDA (Linux)
ASR	mlx-whisper large-v3	faster-whisper / NeMo Canary-1b-v2
Board VLM	MLX-VLM (qwen*-vl)	vLLM
Verify / Gen	mlx_lm.server or Ollama	vLLM / llama.cpp
Compile	Tectonic	Tectonic

Key flags

• --no-rectify — skip perspective correction; feed the full frame to the VLM (VLMs tolerate skew).
• --asr-backend {auto,mlx-whisper,faster-whisper,nemo-canary}
• --skip-{transcription,detection,tracking,ocr,alignment,verification,generation}
• --ssim-threshold / --stability-threshold — keyframe sensitivity.
• --align-window — speech↔board alignment window (s).
• --compiler {pdflatex,lualatex} — used by the Docker/local fallback (Tectonic ignores it).
• --main-font — Cyrillic-capable Unicode font for the preamble. Tectonic uses XeTeX, so notes compile via fontspec (\setmainfont), not T2A/inputenc/babel (Tectonic doesn't bundle the cm-super metrics that route needs). Default: Times New Roman on macOS, DejaVu Serif on Linux. Pass another installed font if you prefer.

Notes on RAM (Apple Silicon)

On a 32GB machine, avoid loading a >16GB model concurrently with other stages. Run stages sequentially (the orchestrator does), and prefer a small local reasoner (e.g. qwen3:4b via Ollama) for verification, or a cloud provider for OCR/generation, when memory is tight.