Repository navigation
Fix/111 align with dissmodel 0.6 origin/main - #114
Merged
profsergiocosta merged 7 commits intoOct 21, 2026
Merged
Conversation
Running the worker against dissmodel 0.6.6 showed that the run itself
works but its provenance is lost:
- a job that finishes in under 1 ms failed with ZeroDivisionError in the
profiling report, after the output had already been written;
- no output_path was set, so executors wrote to the worker's working
directory inside the container; every record now gets
<DISSMODEL_OUTPUT_ROOT>/<experiment_id>/output<suffix>
(default s3://dissmodel-outputs/experiments, suffix from
model.output_suffix, default .tif);
- with a bare file name as output_path, the profiling report and the
record JSON were written "inside" the output file and dropped with a
warning; they now go next to the output;
- a job cancelled through the API was still run;
- the registry cached specs with lru_cache, and nothing cleared it when
the configs-sync sidecar pulled; specs are now read from disk;
- MINIO_SECURE=false turned TLS on (bool("false") is True).
Also:
- reproductions record the dissmodel version installed now (not the
original one) and link to the original in metrics['reproduces']; when
a reproduction ends, the worker compares the outputs: file checksum
and, for GeoTIFF, SHA-256 of the decoded bands
(metrics['reproduction']), so equal data written with different
compression is reported as such;
- each run records Python, the executor class and the versions of
dissmodel, disscube, disslucc, numpy, rasterio and the executor's
package (metrics['environment']); an unpinned model.package is
flagged in the logs;
- CONFIGS_PATH is read from the environment (it was hard-coded);
- dead code removed (two unused sync_configs variants, the Intake
catalog loader, _resolve_executor, the duplicate pip install in
job_runner).
Tests in services/worker/tests (20) run the real subprocess path with a
temporary registry and a local output root; no Redis, MinIO or Docker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GWfF4Gxu6K5Pa4J5Stw9L8
…cord
- POST /experiments/{id}/reproduce builds the new record with
worker.runner.build_reproduction: the original spec, input and
parameters, but the dissmodel version installed now (it used to copy
the original code_version, so the record claimed a version that did
not run); the worker compares the outputs when the run ends.
- GET /experiments/{id} returns the full ExperimentRecord (spec,
artifacts, metrics, logs). The documentation pointed to it for
citation, but it did not exist.
- DELETE /job/{id} also removes the id from the queues.
- /publish includes period, artifacts and metrics (environment and
reproduction result).
- /admin/sync reports the commit new jobs will record; the checkout is
read-only in the API container and is updated by the sidecar. The
commented-out scheduler is gone.
- MinIO client and bucket creation come from worker.storage (one place
for MINIO_SECURE); REDIS_PASSWORD is honoured.
Tests in services/api/tests (9) use FastAPI's TestClient and an
in-memory Redis.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GWfF4Gxu6K5Pa4J5Stw9L8
- dissmodel==0.6.6 in the API, worker and Jupyter images (was dissmodel>=0.5.0, so a rebuild could silently change the code that runs); disslucc==0.5.0 in the worker and Jupyter, disscube==0.6.0 in Jupyter. - requirements-test.txt (pytest, fakeredis, httpx<0.28 for the TestClient of fastapi<0.110). - CI: the API and worker jobs run their test suites with an in-memory Redis instead of skipping when no tests/ folder existed; mypy covers runner, job_runner and compare; ruff also checks scripts/. - Removed entrypoint_install.sh (API and worker), which installed dissmodel from the update-platform branch of LambdaGeo/dissmodel and were not used by the Dockerfiles, and the ../dissmodel volume mounts; removed dependabot.yml.disabled (dependabot.yml is active). - DISSMODEL_OUTPUT_ROOT in docker-compose.yml; tests/ in .dockerignore. - CHANGELOG: Unreleased section for this branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GWfF4Gxu6K5Pa4J5Stw9L8
- README and docs index: status note — the platform is optional and
experimental; reproducing DisSModel, DisSCube and DisSLUCC results
does not depend on it.
- Remove claims of Dask workers, Pangeo integration and Intake catalogs
(none is implemented; the worker consumes a Redis queue); Kubernetes
and Intake are marked as planned; the Pangeo Helm chart instructions
are gone.
- input_format values match JobRequest (tiff, vector, auto; optional).
- API reference: GET /experiments/{id}, DELETE /job/{id}, the
reproduction comparison (metrics['reproduction']) and what
/admin/sync does now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GWfF4Gxu6K5Pa4J5Stw9L8
Running examples/dissmodel-configs/lucc_continuous.toml from DisSLUCC 0.5.0 (LuccME Lab 1, csAC) through the worker failed twice: - executor_module = "disslucc.executors" registered nothing: the package re-exports its executors lazily (PEP 562), so no executor class is created just by importing it. The job runner now also imports the package's direct submodules. - record.parameters held only [model.parameters]. DisSModel's CLI (_load_toml) and ModelExecutor.run() expect the [model] spec fields too (land_use_types, [[model.potential]], [model.static]...), with [model.parameters] winning on overlap; the request's parameters now win over both. Registration keys (name, class, executor_module, package, dissmodel, description, output_suffix) stay out. Also: model.dissmodel, the version range a spec declares, is checked against the installed dissmodel and a mismatch is logged. With both fixes, Lab 1 runs through the worker in about 4 s and the GeoTIFF has the same SHA-256 as the one written by `python -m disslucc.executors.continuous run` with the same TOML (d987b6c2…66f9f32); a reproduction reports file and decoded values identical. New tests check the parameters against dissmodel.executor.cli._load_toml on a spec with the DisSLUCC layout and use a lazily re-exporting package; packaging added to the requirements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GWfF4Gxu6K5Pa4J5Stw9L8
Found while rewriting the documentation and from the Windows installation report in docs/reports/: - .env.example: its placeholders for MINIO_ENDPOINT and REDIS_HOST (e.g. minio.local.xxxxxx:9000) were substituted into the Compose files and replaced the working defaults, so `cp .env.example .env` broke the stack. It now lists only the variables that are read: the two secrets (MINIO_ROOT_PASSWORD, API_KEYS) as "change-me" and every other one with the default the stack uses, plus a note not to set MINIO_SECURE (dissmodel 0.6.6 reads "false" as true). - MinIO health check: it called curl, which the image does not have, so the container was always "unhealthy"; it now runs `mc ready local`. - .gitattributes keeps *.sh with LF: CRLF checkouts on Windows made the entrypoints fail with "no such file or directory". - scripts/backup.sh archived ./data/* and .env, but MinIO and Redis keep their data in named volumes; it now copies the volumes (development or production stack) and .env to backups/<date>/. - scripts/health-check.sh checked MinIO on 9000; it is published on 19000. - Removed DISSMODEL_CLOUD, WORKER_QUEUE and WORKER_CONCURRENCY from the Compose files; nothing reads them. - POST /submit_job_inline answers 400 for a spec that does not parse (was a 500 from the generic handler); test added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GWfF4Gxu6K5Pa4J5Stw9L8
The documentation described a platform that was planned rather than the one in the repository: an executor hierarchy and a dissmodel-executors repository that do not exist, a ModelExecutor interface with the wrong signatures, Intake catalogs, scratch directories and a read-only root filesystem, APScheduler and lru_cache, examples with models that are not registered and without the API key, and the wrong MinIO port. It was split between English and Portuguese and between docs/ and docs/platform/. Now, in English and in one flat docs/ folder: - README: what the platform is, its status (optional, experimental), the executors in the worker image, a quick start and a first job. - getting-started: install, put the inputs in place, run LuccME Lab 1 with DisSLUCC, find the results, reproduce, register the model. - architecture: services and the path of a job, with the code files. - experiment-record: every field and who sets it, a real record from the Lab 1 run, reproduction (file vs decoded values) and citation. - registry: spec format, registration keys, the three-layer parameter merge, synchronisation, inline specs. - executors: the dissmodel 0.6.6 contract with an example checked with ExecutorTestHarness, available executors, three ways to install one, testing, variable mapping (and that applying it is up to the executor). - data, api-reference (every route, checked against main.py), security (including what is not protected: JupyterLab without a token, Redis published in development, no sandbox for model code), deployment (production stack, variables, backups, troubleshooting). - The Windows installation report moved to docs/reports/, with the status of each issue it found. examples/ has the Lab 1 spec (from DisSLUCC 0.5.0, without `package`) and submit_lab1.py, which submits, waits, reproduces and prints the comparison using only the standard library. Run against the API and a worker with a real Redis: completed, output SHA-256 d987b6c2…66f9f32 (the same as DisSLUCC's own CLI), file and values identical on reproduction. `mkdocs build --strict` passes and every in-page anchor resolves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GWfF4Gxu6K5Pa4J5Stw9L8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.