DAReflow is a prototype orchestration platform built to support the DARe Integrated Modelling Platform (IMP). It uses Apache Airflow to orchestrate interconnected modelling workflows, datasets, and data transformations.
DAReflow is a prototype orchestration platform developed to support the DARe Integrated Modelling Platform (IMP). It uses Apache Airflow to orchestrate interconnected modelling workflows, datasets, and data transformations.
An IMP combines multiple weather, hydrodynamic, heat, and agent-based transport models to investigate the impacts of extreme weather events on transportation networks. These models are typically developed independently and can differ substantially in their technologies, interfaces, computational requirements, and input/output formats. As a result, integrating them into a single workflow can require bespoke data transformations and manual configuration.
DAReflow addresses these challenges by providing a common orchestration layer around the models and their associated data. Airflow DAGs define the sequence of processing steps, while Docker containers provide a consistent environment for individual model applications.
Each model or processing step can be represented as a node within an Airflow DAG. Nodes can execute model applications, transform data, or perform data transfer operations. Model versions are containerised using Docker, allowing specific versions to be stored and subsequently retrieved when a workflow is executed.
Model outputs are stored as objects in MinIO, using its S3-compatible API, while PostgreSQL stores associated metadata. This separates large filesystem-based model outputs from the metadata required to manage and trace them.
Git provides version control for the orchestration system and its associated components, supporting reproducibility and historical tracking of modelling configurations.
| Component | Purpose |
|---|---|
| Apache Airflow | Workflow orchestration and execution |
| Docker | Containerisation of model applications |
| MinIO | Object storage for model data and outputs |
| PostgreSQL | Metadata and relational data storage |
| Git | Version control and workflow history |
DAReflow is a proof-of-concept rather than a complete implementation of the DARe IMP. It demonstrates how orchestration, containerisation, object storage, metadata management, and version control can be combined to support complex integrated modelling workflows.
Run using Docker and follow the example DAGs included to structure your own.
Initial build:
docker compose build
docker compose up airflow-init
docker compose up -d
Update to requirements:
docker compose down
docker compose build
docker compose up airflow-init
docker compose up -d
Reload airflow and dependencies while keeping persistent data (e.g. an update to environment variables):
docker compose down
docker compose up airflow-init
docker compose up -d
DAReflow represents individual modelling workflows as Apache Airflow DAGs. Each DAG defines the sequence of operations required to configure, execute, and manage a model, including input staging, model execution, output handling, and metadata recording.
The repository contains example DAGs demonstrating how individual models can be integrated into the DAReflow orchestration framework.
| DAG | Description |
|---|---|
| MATSim_DAG.py | Executes a MATSim transport simulation |
| PT2matsim_DAG.py | Executes PT2-MATSim public transport processing |
| CityCAT_DAG.py | Executes a CityCAT hydrodynamic simulation |
| UDM_DAG.py | Executes Urban Development Model |
| floodEvent_DAG.py | Executes DARe floodEvent module code |
| floodEvent_MATSim_DAG.py | Executes DARe floodEvent and MATSim DAGs in sequence |
The MATSim DAG demonstrates the typical workflow used by DAReflow:
- Record the start of the run in PostgreSQL.
- Create temporary input and output directories.
- Retrieve input data from MinIO.
- Configure the MATSim scenario.
- Select and prepare the required Docker image.
- Execute MATSim using the selected container.
- Periodically upload generated outputs to MinIO.
- Upload the final outputs to MinIO.
- Record metadata for the completed run.
- Determine the final DAG run state.
Model parameters are presented to the user when a DAG is triggered and can also be populated dynamically when DAGs are executed sequentially.
MATSim Docker images are selected using name:tag. Images not already available on the Airflow host are retrieved from MinIO and loaded into Docker, allowing users to upload and deploy model images independently of the orchestration host.
Because model runs can be long-running, output data is periodically copied to MinIO while the model is executing. This preserves intermediate results in the event of a failure. Post-processing and run-state tasks use ALL_DONE trigger rules so that execution status and available outputs can still be recorded when an upstream task fails.
Run metadata, including configuration, Docker image, Git commit, task states, and output location, is stored in PostgreSQL to support both operational monitoring and model provenance.