fix(#529): share /app/work/planner and /app/work/bioqueryai between workbench and celery-worker - #83
Merged
Conversation
…orkbench and celery-worker
Workflow Planner's and BioQueryAI's async (Celery) modes write their
status.json/.ask_index to /app/work/planner and /app/work/bioqueryai
respectively. Neither path was a shared volume between celery-worker
(where the task actually runs and writes the file) and workbench
(where the HTTP polling handler reads it) -- unlike every other
async-completion path bio_agent uses (runs, objects,
workflow_runner_exec, workflow_runner_runs, pathway_enrichment_runs),
which are all already shared.
Confirmed live in #529: the Celery task genuinely completes and
writes a real status.json inside celery-worker's filesystem, but
workbench's filesystem never has the file -- or even the parent
directory -- at all, so planner_ask_status/BioQueryAI polling reports
PENDING forever regardless of actual completion.
Mirrors the existing runs/objects mount exactly: same
${WORK_DIR}/<name>:/app/work/<name> bind-mount form, no extra options,
added to both the workbench and celery-worker services.
Closes #529.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3hyWgSR8wJ3YbRt8zdpqa
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #529. Workflow Planner's and BioQueryAI's async (Celery) modes were writing real, correct completion data (
status.json/.ask_index) insidecelery-worker's filesystem, butworkbench's HTTP polling handlers (planner_ask_status,bioquery_ask_status) read those same container-internal paths fromworkbench's own filesystem — where the paths never existed at all, because/app/work/plannerand/app/work/bioqueryaiwere never declared as shared volumes for either service. Every other async-completion pathbio_agentuses (runs,objects,workflow_runner_exec,workflow_runner_runs,pathway_enrichment_runs) already is shared; these two specifically were not.Fix
Adds the two missing bind mounts to both
workbenchandcelery-workerindocker-compose.yml, in the exact same form as the existingruns/objectsmounts (${WORK_DIR}/<name>:/app/work/<name>, no extra options):Scope check (re: #529's own follow-up scan)
Re-confirmed live against the current
docker-compose.ymlthat, with this change,workbenchandcelery-workernow mount an identical set of 7/app/work/*paths (runs,objects,workflow_runner_exec,workflow_runner_runs,pathway_enrichment_runs,planner,bioqueryai) — no other divergence between the two services' mount lists. This matches the issue's own follow-up comment, which already swept all 426 plugins and found this bug isolated tobio_agent's two paths (not a systemic pattern) — not re-running that full sweep here, just re-confirming the two services still match after this change.Testing (real, live, both done after rebuilding +
--force-recreateing bothworkbenchandcelery-worker; sha256 of the running image config matched a freshdocker compose buildof both, confirming no staleness)Workflow Planner — real submission (Django test client against the live view, real login session, no mocking) → real Celery completion → polled from
workbench:Full real multi-plan DAG result returned (
ask_id=planner_ffd7008d4a1b). Confirmed the underlyingstatus.jsonis now byte-identical when read from both containers (previously: present incelery-worker,No such file or directoryinworkbench).BioQueryAI — real submission → real Celery task (real
deepseek-r1:32bLLM calls via Ollama, ~4m42s) → polled fromworkbench:Task finished after my poll loop's timeout; re-polled afterward and
workbenchcorrectly reportsstate: COMPLETED(ask_id=ask_1789100190756). The query itself came backstatus: FAILED/ "Time budget exceeded" — that's my test's own overly tight 2-minutebq_time_budget_minutes, not a symptom of this bug; the point being tested (workbench can see real state instead of PENDING-forever) is proven either way. No sign of #530's SIGSEGV during this run.Historical stuck-PENDING data (pre-fix)
Checked: not recoverable.
/app/work/plannerand/app/work/bioqueryaiwere never a named volume or bind mount pre-fix, so anything a Celery task wrote there before this fix lived only incelery-worker's ephemeral container-writable layer. That container has since been recreated (both by the in-progress work this PR finishes, and again by this PR's own--force-recreateverification step) — no stopped container, docker-committed image, or named volume preserves the old layer (docker ps -a/docker images/docker volume lsall checked, nothing found). Any run that completed incelery-workerbefore this fix and was never separately re-run is gone for good; there is no data-recovery step to run here.🤖 Generated with Claude Code
https://claude.ai/code/session_01V3hyWgSR8wJ3YbRt8zdpqa