feat(assessment): Persist batch results and deliver them in callback metadata

[vprashrex]

Issue

Closes #1200

Summary

The BATCH assessment API could finish a paid-for run and leave the client with nothing: the callback inlined every row (413 from the receiver), carried no failure reason, and pointed at no durable copy of the results. Separately, the legacy cron was polling API-created runs and crashing on their differently-shaped state, so those runs never finalised and never fired a callback at all.

This makes the result durable and the callback self-describing, and clears the driver and provider bugs found alongside it.

Major changes

Storage layout restructured

• One prefix per assessment, <project.storage_path>/assessment/<assessment_id>/, holding submission.jsonl, per-stage results.jsonl and errors.jsonl; uploaded files move to assessment/submissions/<name>.<ext>.
• Duplicate submission names are rejected before the upload, instead of after it has already overwritten the existing file.

Batch results and errors are durable

• Every provider dump is recorded on assessment.result_files as its stage completes, so results outlive the Celery tick that produced them.
• New errors.jsonl per run, now including OpenAI's error file, which was previously read as a boolean only, so a run with 389 provider failures reported errors=0.

Callback carries the result

• metadata now holds a 24-hour presigned URL per result file and error holds the failure reason; both were hardcoded null.
• Result files are written before delivery is attempted, so durability never depends on the callback succeeding.

Submissions get their own table

• New assessment_submission table; evaluation_dataset multiplexed four surfaces behind a type column and its name uniqueness ignored that column, so an evaluation dataset name blocked an assessment one.
• BATCH input takes rows inline or by submission_doc_id, and those rows (3-6MB) now live in object storage rather than inline in Postgres.

Fixes

• Legacy cron no longer polls API-created runs, which it crashed on, so those runs never finalised and never fired a callback.
• Deterministic errors in the cron now fail the run instead of retrying forever, which is why the bug above went unnoticed for months.
• Gemini structured output no longer sends a duplicate ordering key, which was failing 100% of runs carrying a json_output_schema.
• Anthropic effort and thinking are mapped instead of silently dropped, so "high effort" is no longer a no-op plus a log warning.
• Batch polling saves each changed field instead of only on a status flip, which used to drop the provider's error-file id entirely.
• A duplicated stage tick is now a no-op rather than a second provider batch.
• RunExecution.pipeline admits both shapes actually stored in it, so the next mismatch is a type error rather than a runtime crash.

Migration

083 — new assessment_submission table, assessment.result_files / submission_input / submission_id (replacing dataset_id), batch_job.provider_error_file_id. Verified upgrade and downgrade on a throwaway database.

Notes

Not in this PR, deferred by decision: retrying a rejected callback, a per-row correlation id in the result, and a GET /assessments/{id} read path.

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)

• ready-for-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 44059a0c-e4c5-4f7d-8649-36a55dac0655

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

OpenAPI changes   🔴 12 breaking changes

Caution

Downstream consumers may need an update before merging.

Breaking changes  ·  12

	Method	Path	Change
🔴	GET	/api/v1/assessment/datasets	added #/components/schemas/AssessmentSubmissionPreview to the data/anyOf[subschema #1]/items/preview response property anyOf list for the response status 200
🔴	GET	/api/v1/assessment/datasets	removed the required property data/anyOf[subschema #1]/items/dataset_id from the response with the 200 status
🔴	GET	/api/v1/assessment/datasets	removed the required property data/anyOf[subschema #1]/items/dataset_name from the response with the 200 status
🔴	POST	/api/v1/assessment/datasets	added #/components/schemas/AssessmentSubmissionResponse to the data response property anyOf list for the response status 200
🔴	DELETE	/api/v1/assessment/datasets/{dataset_id}	for the path request parameter dataset_id, the type/format was changed from integer to string/uuid
🔴	GET	/api/v1/assessment/datasets/{dataset_id}	for the path request parameter dataset_id, the type/format was changed from integer to string/uuid
🔴	GET	/api/v1/assessment/datasets/{dataset_id}	added #/components/schemas/AssessmentSubmissionResponse to the data response property anyOf list for the response status 200
🔴	GET	/api/v1/assessment/runs	response property data/anyOf[subschema #1]/items/execution/anyOf[subschema #1: RunExecution]/pipeline list-of-types was widened by adding types array to media type application/json of response 200
🔴	POST	/api/v1/assessment/runs	added the new required request property submission_id
🔴	GET	/api/v1/assessment/runs/{run_id}	response property data/anyOf[subschema #1: AssessmentRunPublic]/execution/anyOf[subschema #1: RunExecution]/pipeline list-of-types was widened by adding types array to media type application/json of response 200
🔴	PATCH	/api/v1/assessment/runs/{run_id}/post-processing	response property data/anyOf[subschema #1: AssessmentRunPublic]/execution/anyOf[subschema #1: RunExecution]/pipeline list-of-types was widened by adding types array to media type application/json of response 200
🔴	POST	/api/v1/assessment/runs	removed the request property dataset_id

Full changelog  ·  47

	Method	Path	Change
🔴	GET	/api/v1/assessment/datasets	added #/components/schemas/AssessmentSubmissionPreview to the data/anyOf[subschema #1]/items/preview response property anyOf list for the response status 200
🔴	GET	/api/v1/assessment/datasets	removed the required property data/anyOf[subschema #1]/items/dataset_id from the response with the 200 status
🔴	GET	/api/v1/assessment/datasets	removed the required property data/anyOf[subschema #1]/items/dataset_name from the response with the 200 status
🔴	POST	/api/v1/assessment/datasets	added #/components/schemas/AssessmentSubmissionResponse to the data response property anyOf list for the response status 200
🔴	DELETE	/api/v1/assessment/datasets/{dataset_id}	for the path request parameter dataset_id, the type/format was changed from integer to string/uuid
🔴	GET	/api/v1/assessment/datasets/{dataset_id}	for the path request parameter dataset_id, the type/format was changed from integer to string/uuid
🔴	GET	/api/v1/assessment/datasets/{dataset_id}	added #/components/schemas/AssessmentSubmissionResponse to the data response property anyOf list for the response status 200
🔴	GET	/api/v1/assessment/runs	response property data/anyOf[subschema #1]/items/execution/anyOf[subschema #1: RunExecution]/pipeline list-of-types was widened by adding types array to media type application/json of response 200
🔴	POST	/api/v1/assessment/runs	added the new required request property submission_id
🔴	GET	/api/v1/assessment/runs/{run_id}	response property data/anyOf[subschema #1: AssessmentRunPublic]/execution/anyOf[subschema #1: RunExecution]/pipeline list-of-types was widened by adding types array to media type application/json of response 200
🔴	PATCH	/api/v1/assessment/runs/{run_id}/post-processing	response property data/anyOf[subschema #1: AssessmentRunPublic]/execution/anyOf[subschema #1: RunExecution]/pipeline list-of-types was widened by adding types array to media type application/json of response 200
🔴	POST	/api/v1/assessment/runs	removed the request property dataset_id
🟢	—	—	removed the schema APIResponse_AssessmentDatasetResponse_
🟢	—	—	removed the schema APIResponse_list_AssessmentDatasetResponse__
🟢	—	—	removed the schema AssessmentDatasetPreview
🟢	—	—	removed the schema AssessmentDatasetResponse
🟢	—	—	a breaking change was detected but the version is still 0.5.0
🟢	GET	/api/v1/assessment/datasets	removed the optional property data/anyOf[subschema #1]/items/file_extension from the response with the 200 status
🟢	GET	/api/v1/assessment/datasets	removed #/components/schemas/AssessmentDatasetPreview from the data/anyOf[subschema #1]/items/preview response property anyOf list for the response status 200
🟢	GET	/api/v1/assessment/datasets	added the required property data/anyOf[subschema #1]/items/name to the response with the 200 status
🟢	GET	/api/v1/assessment/datasets	added the required property data/anyOf[subschema #1]/items/submission_id to the response with the 200 status
🟢	POST	/api/v1/assessment/datasets	removed #/components/schemas/AssessmentDatasetResponse from the data response property anyOf list for the response status 200
🟢	GET	/api/v1/assessment/datasets/{dataset_id}	removed #/components/schemas/AssessmentDatasetResponse from the data response property anyOf list for the response status 200
🟢	POST	/api/v1/assessment/runs	added the optional property data/anyOf[subschema #1: AssessmentRunResponse]/submission_id to the response with the 200 status
🟢	POST	/api/v1/assessment/runs	added the optional property data/anyOf[subschema #1: AssessmentRunResponse]/submission_name to the response with the 200 status
🟢	POST	/api/v1/assessment/runs	removed the optional property data/anyOf[subschema #1: AssessmentRunResponse]/dataset_id from the response with the 200 status
🟢	POST	/api/v1/assessment/runs	removed the optional property data/anyOf[subschema #1: AssessmentRunResponse]/dataset_name from the response with the 200 status
🟢	POST	/api/v1/assessment/runs/{run_id}/resume	added the optional property data/anyOf[subschema #1: AssessmentRunResponse]/submission_id to the response with the 200 status
🟢	POST	/api/v1/assessment/runs/{run_id}/resume	added the optional property data/anyOf[subschema #1: AssessmentRunResponse]/submission_name to the response with the 200 status
🟢	POST	/api/v1/assessment/runs/{run_id}/resume	removed the optional property data/anyOf[subschema #1: AssessmentRunResponse]/dataset_id from the response with the 200 status
🟢	POST	/api/v1/assessment/runs/{run_id}/resume	removed the optional property data/anyOf[subschema #1: AssessmentRunResponse]/dataset_name from the response with the 200 status
🟢	POST	/api/v1/assessment/runs/{run_id}/retry	added the optional property data/anyOf[subschema #1: AssessmentRunResponse]/submission_id to the response with the 200 status
🟢	POST	/api/v1/assessment/runs/{run_id}/retry	added the optional property data/anyOf[subschema #1: AssessmentRunResponse]/submission_name to the response with the 200 status
🟢	POST	/api/v1/assessment/runs/{run_id}/retry	removed the optional property data/anyOf[subschema #1: AssessmentRunResponse]/dataset_id from the response with the 200 status
🟢	POST	/api/v1/assessment/runs/{run_id}/retry	removed the optional property data/anyOf[subschema #1: AssessmentRunResponse]/dataset_name from the response with the 200 status
🟢	POST	/api/v1/assessments	added the new optional request property input/anyOf[subschema #2: BatchInput]/submission_doc_id
🟢	POST	/api/v1/assessments	the request property input/anyOf[subschema #2: BatchInput]/data became optional
🟢	POST	/api/v1/assessments	request property input/anyOf[subschema #2: BatchInput]/data list-of-types was widened by adding types null to media type application/json
🟢	POST	/api/v1/assessments	the input/anyOf[subschema #2: BatchInput]/data request property's minItems was unset from 1
🟢	POST	/api/v1/configs	added the new optional request property config_blob/completion/anyOf[subschema #2: KaapiTextCompletionConfig]/params/thinking
🟢	POST	/api/v1/configs	added the new optional request property config_blob/completion/anyOf[subschema #2: KaapiTextCompletionConfig]/params/thinking_level
🟢	POST	/api/v1/llm/call	added the new optional request property config/blob/anyOf[subschema #1: ConfigBlob]/completion/anyOf[subschema #2: KaapiTextCompletionConfig]/params/thinking
🟢	POST	/api/v1/llm/call	added the new optional request property config/blob/anyOf[subschema #1: ConfigBlob]/completion/anyOf[subschema #2: KaapiTextCompletionConfig]/params/thinking_level
🟢	POST	/api/v1/llm/chain	added the new optional request property blocks/items/config/blob/anyOf[subschema #1: ConfigBlob]/completion/anyOf[subschema #2: KaapiTextCompletionConfig]/params/thinking
🟢	POST	/api/v1/llm/chain	added the new optional request property blocks/items/config/blob/anyOf[subschema #1: ConfigBlob]/completion/anyOf[subschema #2: KaapiTextCompletionConfig]/params/thinking_level
🟢	POST	/api/v1/llm/chain/sts	added the new optional request property rag/anyOf[subschema #1: RAGBlockSpec]/params/anyOf[subschema #1: TextLLMParams]/thinking
🟢	POST	/api/v1/llm/chain/sts	added the new optional request property rag/anyOf[subschema #1: RAGBlockSpec]/params/anyOf[subschema #1: TextLLMParams]/thinking_level

_{main ↔ 7eb4778d · generated by oasdiff}

[codecov]

Codecov Report

❌ Patch coverage is 96.17537% with 41 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
backend/app/crud/assessment/submission.py	84.61%	8 Missing ⚠️
backend/app/services/assessment/api/batch.py	78.12%	7 Missing ⚠️
backend/app/services/assessment/api/callbacks.py	28.57%	5 Missing ⚠️
...ackend/app/services/assessment/api/result_files.py	95.68%	5 Missing ⚠️
backend/app/crud/assessment/batch.py	72.72%	3 Missing ⚠️
backend/app/services/assessment/api/submission.py	88.46%	3 Missing ⚠️
backend/app/services/assessment/submission.py	85.71%	3 Missing ⚠️
backend/app/api/routes/assessment/datasets.py	88.23%	2 Missing ⚠️
backend/app/services/assessment/utils/export.py	77.77%	2 Missing ⚠️
backend/app/api/routes/assessment/assessments.py	50.00%	1 Missing ⚠️
... and 2 more

📢 Thoughts on this report? Let us know!