ACCESS-MOPPy is a CMORisation tool designed to post-process ACCESS model output and produce CMIP-compliant datasets.
- Python API for integration into notebooks and scripts
- Batch processing system for HPC environments with PBS
- Real-time monitoring with web-based dashboard
- Flexible CMORisation of individual variables
- Dask-enabled for scalable parallel processing
- Cross-platform compatibility (not limited to NCI Gadi)
- CMIP6 and CMIP7 FastTrack support
ACCESS-MOPPy requires Python >= 3.11.
The conda/analysis3-latest environment maintained by ACCESS-NRI already
includes access_moppy and its dependencies, so no pip install is needed:
module use /g/data/xp65/public/modules
module load conda/analysis3-latestAll command-line tools (moppy-cmorise, moppy-tui, moppy-qc, …) are
available immediately after loading the module. You'll need membership of
the xp65 NCI project for the module itself, plus whichever projects hold
the model output and CV/table data you're processing. Pin a dated release
(e.g. conda/analysis3-26.04) instead of -latest if you need a
reproducible environment for a production run.
pip install access_moppyThe controlled vocabularies under src/access_moppy/vocabularies/ are pulled
in as git submodules. If you install from a local clone, initialise them
first, otherwise the CMOR tables/CVs will be missing and imports will fail
with an error like No module named 'access_moppy.vocabularies.CMIP6_CVs':
git clone --recurse-submodules https://github.com/ACCESS-NRI/ACCESS-MOPPy.git
cd ACCESS-MOPPy
pip install .If you already have a clone without the submodules populated, run:
git submodule update --init --recursive
pip install .import glob
from access_moppy import ACCESS_ESM_CMORiser
# Select input files
files = glob.glob("/path/to/model/output/*mon.nc")
# Create CMORiser instance
cmoriser = ACCESS_ESM_CMORiser(
input_data=files,
compound_name="Amon.pr", # table.variable format
experiment_id="historical",
source_id="ACCESS-ESM1-5",
variant_label="r1i1p1f1",
grid_label="gn",
activity_id="CMIP",
output_path="/path/to/output"
)
# Run CMORisation
cmoriser.run()
cmoriser.write()For large-scale processing on HPC systems:
- Create a configuration file (
batch_config.yml):
variables:
- Amon.pr
- Omon.tos
- Amon.ts
experiment_id: piControl
source_id: ACCESS-ESM1-5
variant_label: r1i1p1f1
grid_label: gn
input_folder: "/g/data/project/model/output"
output_folder: "/scratch/project/cmor_output"
file_patterns:
Amon.pr: "output[0-4][0-9][0-9]/atmosphere/netCDF/*mon.nc"
Omon.tos: "output[0-4][0-9][0-9]/ocean/*temp*.nc"
Amon.ts: "output[0-4][0-9][0-9]/atmosphere/netCDF/*mon.nc"
# PBS configuration
queue: normal
cpus_per_node: 16
mem: 32GB
walltime: "02:00:00"
scheduler_options: "#PBS -P your_project"
storage: "gdata/project+scratch/project"
worker_init: |
module load conda
conda activate your_environment- Submit batch job:
moppy-cmorise batch_config.yml- Monitor progress at http://localhost:8501
The batch processing system provides:
- Parallel execution: Each variable processed as a separate PBS job
- Real-time monitoring: Web dashboard showing job status and progress
- Automatic tracking: SQLite database maintains job history and status
- Error handling: Failed jobs can be easily identified and resubmitted
- Resource optimization: Configurable CPU, memory, and storage requirements
- Environment management: Automatic setup of conda/module environments
- Streamlit Dashboard: Real-time web interface at http://localhost:8501
- Command line: Use standard PBS commands (
qstat,qdel) - Database: SQLite tracking at
{output_folder}/cmor_tasks.db - Log files: Individual stdout/stderr for each job
work_directory/
├── batch_config.yml # Your configuration
├── cmor_job_scripts/ # Generated PBS scripts and logs
│ ├── cmor_Amon_pr.sh # PBS script
│ ├── cmor_Amon_pr.py # Python processing script
│ ├── cmor_Amon_pr.out # Job output
│ └── cmor_Amon_pr.err # Job errors
└── output_folder/
├── cmor_tasks.db # Progress tracking
└── [CMORised files] # Final output
- Getting Started:
docs/source/getting_started.rst - Example Configuration:
src/access_moppy/examples/batch_config.yml - API Reference: [Coming soon]
Integration and end-to-end tests require an external test-data tree set via
the ACCESS_MOPPY_DATA_ROOT environment variable.
- Covered tests: full CMOR integration and end-to-end real-file tests
- No fallback: test-data fixtures in
tests/data/are not used by these tests - Requirement:
ACCESS_MOPPY_DATA_ROOTmust point to a valid dataset root containingoutput*/atmosphere/netCDF,output*/ocean, andoutput*/ice
Example:
export ACCESS_MOPPY_DATA_ROOT=/path/to/CMIP7_Test_data/esm-historical
pixi run -e dev python -m pytest tests/integration/test_full_cmorisation.py
pixi run -e dev python -m pytest tests/integration/test_cmip7_baseline_cmorisation.py
pixi run -e dev python -m pytest tests/e2e/test_end_to_end.pyCMIP7 baseline test note:
tests/integration/test_cmip7_baseline_cmorisation.pyruns one case per CMIP7 baseline variable listed insrc/access_moppy/examples/batch_config_esm1-6_cmip7_baseline.yml- By default, this suite checks end-to-end CMORisation success (run/write/output)
- To additionally enforce WCRP compliance-checker validation for this suite,
set
ACCESS_MOPPY_BASELINE_VALIDATE_WCRP=1
Example with strict WCRP validation enabled:
export ACCESS_MOPPY_DATA_ROOT=/path/to/CMIP7_Test_data/esm-historical
export ACCESS_MOPPY_BASELINE_VALIDATE_WCRP=1
pixi run -e dev python -m pytest tests/integration/test_cmip7_baseline_cmorisation.py --validation-tool=wcrp- Stable project status: ACCESS-MOPPy is suitable for supported CMORisation workflows and ongoing production-oriented use.
- Ocean variables: Ocean variables are supported, including dedicated ocean CMORisers and resource guidance for large 3D variables.
- Variable mapping: Mapping coverage continues to be reviewed and improved for CMIP6/CMIP7 compliance.
- Issues: Submit via GitHub Issues
- Questions: Contact ACCESS-NRI support
- Contributions: Welcome via Pull Requests
ACCESS-MOPPy is licensed under the Apache-2.0 License.