fix: warn when dataset samples get truncated past max_seq_length

[Anshul1336]

Summary

dataset.py was truncating any sample that went over max_seq_length without telling anyone. now it logs a warning after tokenizing, saying what percentage of samples got cut and by how much on average.

Scope

• What changed: trainer/dataset.py — added a _summarize_truncation helper and wired it into load_and_tokenize, so it measures each sample's real token length before truncating and logs a warning if anything exceeded max_seq_length.
• What did not change: the actual truncation behavior itself, still truncates the same way, this just adds visibility into it.
• Related issue: Add token budget warning when dataset exceeds context length #181

Product Impact

anyone fine-tuning on a dataset with long examples was losing content silently, no way to tell without manually checking token lengths themselves. now they get a heads up in the logs, e.g. "12.5% of samples (5/40) exceeded max_seq_length=512 and were truncated (avg 34.2 tokens lost per truncated sample)", so they know to either bump max_seq_length or trim their data.

Technical Notes

load_and_tokenize batches through .map(), so I collect lengths in a list declared outside the batch function (closure) and extend it on every batch call, then run the stats + log once after .map() finishes with the full dataset's lengths. required an extra truncation=False tokenizer call per batch just to get the untruncated length, since the real encode call already truncates before you can measure it.

Validation

• Unit tests added or updated
• Integration path tested
• Frontend manually verified
• Accessibility and responsive behavior reviewed
• Documentation updated

ran ruff format, ruff check, and mypy on the changed files, all clean. full trainer/dataset.py test file passes (7/7).

Screenshots Or Recordings

n/a, backend-only logging change.

Risk And Rollback

low risk, purely additive (a log line), doesn't change what gets fed to the trainer. if it turns out to be noisy or wrong, reverting this one file is enough.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 418a8dbb-f2b6-42c0-90af-1054f4600704

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Automated Review Checklist

Before approving this PR, confirm each item:

Code quality

• Logic is correct and edge cases are handled
• No debug logs, dead code, or commented-out blocks left in
• Naming is clear — no abbreviations that need a comment to decode

Tests

• New behaviour is covered by tests, or existing tests updated
• poetry run pytest passes locally

Frontend (if applicable)

• UI renders correctly across light/dark mode
• No layout regressions on narrow viewports

Infrastructure / config (if applicable)

• Secrets/env vars are not hardcoded
• Docker build still passes (docker build .)

Docs

• CHANGELOG or PR description explains the why, not just the what
• Public API changes are reflected in docs

---

Review, check off what applies, then submit your formal Approve or Request Changes.

[SahilKumar75]

Reviewed the truncation-warning implementation and the cache-safety follow-up. Full tests, lint, formatting, and GitHub CI pass.