fix(evaluation): Limit payload size for prompts

[AkhileshNegi]

Issue

Closes #PLEASE_TYPE_ISSUE_NUMBER

Summary

• Before: The prompt iteration sent the entire score-trace file in a single request, exceeding the 1M input-token limit of the provider.
• Now: The payload is capped by selecting only necessary fields and clamping long text, degrading repeats before rows to fit within the limit.
• Each trace is projected down to the fields needed for the brief.
• Repeats are decreased before dropping entire rows to maintain signal from questions.

Checklist

Before submitting a pull request, please ensure that you mark these task.

• If you've fixed a bug or added code that is tested and has test cases.
• Ran fastapi run --reload app/main.py — not exercised against a live run; covered by tests instead (see Notes).

Notes

Tests — app/tests/api/routes/test_improve_prompt.py, 43 passing. Four new cases: the worst-first cap on an oversized payload (with an unscoreable "N/A" row in the mix, which a naive sort raises on), the 5→4→3 ladder keeping all questions, row-dropping taking over at the floor, and _primary_score ranking unscoreable rows last. Each was mutation-checked — breaking the sort, the repeat limit, or the floor branch fails the corresponding test.

Why not the Files API — a document block with a file_id still expands into context and bills as input tokens, so it does not move this ceiling. The only surface that would is container_upload + the code-execution tool (file lands in the sandbox, the model queries it with Python), but prompt rewriting needs to read the failing answers rather than aggregate them, so they would come back into context anyway — with an unbounded agentic loop inside a Celery soft time limit. Not worth it here.

Known gap, unchanged by this PR — crud/evaluations/summary.py sends every trace whole and has the same latent ceiling (it carries its own ponytail: note saying so). Left alone deliberately: it needs a different projection, so sharing one would mean an abstraction with two diverging implementations. The wiki now points at both.

Original PR description

Issue

No issue filed — reported in-session:

prompt_generation_failed: Anthropic returned HTTP 400 — retry or contact Kaapi if persistent
anthropic_response: {"type":"error","error":{"type":"invalid_request_error",
"message":"prompt is too long: 1121367 tokens > 1000000 maximum"}}

Summary

Prompt iteration dumped the entire score-trace file into the drafting request
(json.dumps(traces), no bound). A large judged run — every question repeated
duplication_factor times, three metrics each, a paragraph of judge reasoning
per metric — overflowed the provider's 1M input-token ceiling, so Anthropic
returned a 400 and the job was marked FAILED.

Two commits, in the order they should be read:

1. Cap the payload. Each trace is projected down to the fields the brief
actually names (trace_id and, on v1, question_id and the judge comment
were shipped and referenced nowhere), long text fields are clamped, and rows are
filled to a character budget worst-scoring-first — the brief tells the model to
focus on the failing rows, so those are the ones a cap has to keep.

2. Degrade repeats before rows. Dropping rows loses questions the model never
sees at all; the 5th repeat of a question carries much less signal. The brief is
now measured with messages.count_tokens (same model and output schema as the
real call, since both are billed as input) and degraded until it fits:

• repeats first, one at a time — 5 → 4 → 3 — with every question still represented;
• at the floor of 3 repeats the degradation switches to dropping whole rows,
  worst-scoring-first, against a character budget rescaled from the measured count.

Which repeats survive is decided by file order, not score: repeats exist to show
the judge's spread on one question, so dropping the worst of them would erase the
signal the judge brief is told to read. A trace without a question_id is keyed
uniquely and never grouped, so a dataset uploaded without ids can't collapse into
one group and get cut to a handful of rows.

The brief states how many traces it is not seeing, and each degradation step logs
a line that distinguishes repeat-cutting from row-cutting.

Checklist

• If you've fixed a bug or added code that is tested and has test cases.
• Ran fastapi run --reload app/main.py — not exercised against a live run;
  covered by tests instead (see Notes).

Notes

Tests — app/tests/api/routes/test_improve_prompt.py, 43 passing. Four new
cases: the worst-first cap on an oversized payload (with an unscoreable "N/A"
row in the mix, which a naive sort raises on), the 5→4→3 ladder keeping all
questions, row-dropping taking over at the floor, and _primary_score ranking
unscoreable rows last. Each was mutation-checked — breaking the sort, the repeat
limit, or the floor branch fails the corresponding test.

Why not the Files API — a document block with a file_id still expands into
context and bills as input tokens, so it does not move this ceiling. The only
surface that would is container_upload + the code-execution tool (file lands in
the sandbox, the model queries it with Python), but prompt rewriting needs to read
the failing answers rather than aggregate them, so they would come back into
context anyway — with an unbounded agentic loop inside a Celery soft time limit.
Not worth it here.

Known gap, unchanged by this PR — crud/evaluations/summary.py sends every
trace whole and has the same latent ceiling (it carries its own ponytail: note
saying so). Left alone deliberately: it needs a different projection, so sharing
one would mean an abstraction with two diverging implementations. The wiki now
points at both.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)

• ready-for-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 918d78ea-ddfb-4abb-b6fe-c03b9a383597

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

OpenAPI changes   ⚪ No API surface changes

Note

This PR does not modify the API contract.

_{main ↔ 6a5a2366 · generated by oasdiff}

[AkhileshNegi]

Third commit added after a latency review of the ladder:

• Counting is skipped for a brief that cannot overflow. count_tokens is a network call from a worker with a 300s soft time limit (CELERY_TASK_SOFT_TIME_LIMIT), and the ladder can make up to four. A brief under 200K characters cannot reach the 900K-token budget even at a pessimistic three tokens per character, so it is not counted — which is where most runs land, leaving the common path at its previous single API call. Large runs still measure exactly.
• Covered the case the ladder cannot help with: v1 traces from datasets uploaded without question_id are keyed uniquely and never grouped as repeats, so the rescaled character budget is the only thing that can shrink an oversized brief there. Now tested.

[codecov]

Codecov Report

❌ Patch coverage is 99.11504% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...end/app/services/evaluations/prompt_improvement.py	98.21%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!