A lightweight agentic reinforcement learning framework for everyone
English · 简体中文
- 10/2026 📣📣📣 HarborRL released
HarborRL is a lightweight framework for training tool-using agents on verifiable, sandboxed tasks. It makes model serving, trajectory generation, tool execution, verifier rewards, and policy updates into a single reproducible pipeline.
HarborRL supports flexible combinations of Models × Algorithms × Tasks × Harnesses (MATH):
- Models: RL training using Qwen and GLM as base models.
- Algorithms: GRPO and DAPO for RL training.
- Tasks: RL training on 200+ Harbor-format tasks in isolated workers, including Terminal-Bench, SWE-Bench, Deep-SWE, and more.
- Harnesses: RL training with 40+ harness frameworks, including Claude Code, Codex, LangGraph, SWE-agent, and more.
Confirm that the node exposes at least the configured number of GPUs:
nvidia-smiExpected result: nvidia-smi lists the expected GPUs and their memory state. The example YAML defaults to eight GPUs, split into four actor GPUs and four rollout GPUs.
On a node with Git, Docker, and network access, install the pinned open-source stack:
scripts/bootstrap_backends.sh /path/to/backends
. /path/to/backends/harborrl-backend.envExpected result: the script checks out Slime commit 3778dbf6d1a533ab478ecf5ddaa11449a47752b2, Megatron-LM commit 1dcf0dafa884ad52ffb243625717a3471643e087, pulls lmsysorg/sglang:v0.5.15.post1-cu129, and writes harborrl-backend.env. Sourcing that file sets SLIME_DIR, MEGATRON_DIR, and SGLANG_IMAGE in the current shell.
For an offline GPU node, prepare the same repositories and image on a compatible machine, transfer them through shared storage, and export the same three environment variables. Doctor fails if the paths or versions do not match.
Copy examples/native/train_qwen_native.yaml to an untracked private path. Replace the generic worker destinations, model and reference checkpoint paths, gateway origin, tokenizer/template digests, GPU layout, and output location. Also make tasks.catalog and harness.profile point to the intended catalog and profile; paths are resolved relative to the private YAML.
Expected result: no tracked example file is modified, and the private YAML contains only values that are valid for your machines. Do not commit credentials, proxy settings, private task paths, or cluster-specific values.
Validate the normalized private plan before preparing workers:
python -m harborrl.cli train --config /path/to/private.yaml --dry-runExpected result: the JSON contains your private values and resolved paths, training.backend_contract remains slime-v0.3.2-native-v1, and the command still exits without creating a run directory or starting backends.
Each worker needs Linux, Docker Engine, passwordless SSH from the trainer, Python 3.12 or newer, and Harbor 0.23.0. The worker must be able to pull or locally build every task image in the catalog and reach the advertised Messages gateway. It does not need a model-provider API key.
Install the matching HarborRL runner and Harbor runtime on each worker. A direct probe looks like:
ssh WORKER 'cd /path/to/Harbor-RL && /opt/harbor/bin/python -m harborrl.rollout.harbor_job.runner --probe'Expected result: the worker prints JSON containing harbor_version: 0.23.0, a Python version of 3.12 or newer, available Claude option fields, and callable Harbor trial interfaces. The command exits with status 0 without creating a trial.
Run workers under unprivileged accounts. Keep API keys, Claude settings, proxy credentials, and private task data off workers; the rollout passes only the sanitized runtime values required for a trial.
harborrl doctor --config /path/to/private.yamlExpected result: doctor prints a JSON object containing checks and scope. Every check has "ok": true, the process exits with status 0, and no training run starts.
Doctor checks the private YAML, catalog digests, model files, worker SSH probes, pinned backend paths and commits, required imports, hooks, GPU budget, and SGLang image tag. It does not start the model service or generate tokens, so serving protocol behavior, token IDs, and logprobs are verified by the subsequent live semantic check or actual training rather than by doctor.
harborrl train --config /path/to/private.yamlExpected result: doctor runs again, then HarborRL creates a fresh directory under output.root/training/ named with a timestamp and run ID. The run records its launch plan, source manifest, materialized task catalog, prompt rows, trajectories, policy history, metrics, and checkpoints according to the configured save interval. A successful run exits with status 0.
If any preflight check fails, training does not start and failed checks are printed to stderr as JSON. If the backend exits unsuccessfully, the child status is propagated and the run directory retains the evidence available at failure for inspection.
Use Python 3.10 or newer. The CPU development install does not require CUDA, Docker, model weights, Slime, Megatron-LM, or SGLang.
Create and enter a virtual environment:
python -m venv .venv
. .venv/bin/activateExpected result: .venv/ is created and the shell prompt shows (.venv). Neither command starts a service or modifies the examples.
Install the editable package with development tools:
python -m pip install --upgrade pip
python -m pip install -e .[dev]Expected result: pip installs HarborRL with pytest and ruff, but does not install the heavyweight training stack. The command completes without compiling CUDA kernels.
Check the command-line entry point:
harborrl --helpExpected result: argparse prints the training CLI usage and exits with status 0. The public commands are train and doctor; both require --config.
Training is configured with one YAML file that has a fixed section and field set. Relative paths resolve against the YAML file. Repeated --set section.field=value overrides are parsed as YAML values and validated again.
| Section | Purpose |
|---|---|
execution |
Selects the fixed harbor_job backend |
tasks |
Selects a nonempty immutable task catalog |
harness |
Selects the Claude CLI, model, and timeout profile |
harbor |
Configures Harbor version, interpreter, and SSH workers |
gateway |
Configures the Messages listener, origin, serving digests, and raw-logprob audit |
model |
Configures actor/reference checkpoints and the Slime model preset |
training |
Configures rollout count, learning rate, save cadence, and backend identifier |
sampling |
Configures group size, batch groups, retries, context, and generation limits |
deployment |
Configures GPU count and actor/rollout parallelism |
output |
Configures the run-tree root |
A native task is a content-addressed directory:
task/
├── task.toml
├── instruction.md
├── environment/
│ └── Dockerfile
└── tests/
└── test.sh
The catalog identity is:
{
"id": "task",
"revision": "source-revision",
"path": "relative/or/absolute-task",
"task_digest": "64-hex-sha256",
"reward_profile": {
"key": "reward",
"scale": 1.0,
"offset": 0.0,
"raw_range": [0, 1]
}
}The verifier writes a shared-environment terminal result and reward.json or reward.txt. HarborRL resolves both through the reward profile, requires the selected values to agree, and records source bytes and SHA-256 receipts. Boolean and nonfinite rewards are rejected.
The machine-readable Hub inventory is configs/harbor_hub/manifests.yaml. A dataset is marked supported only after its upstream revision, license, task layout, resource requirements, digests, worker-class reward contrast, and a complete native training batch have been recorded.
HarborRL is released under the MIT license.
If you use HarborRL for your research, please cite our technical report:
We thank Slime and Harbor for developing the foundational infrastructure for training and task execution.