Robots need experience with unexpected situations, including what to do after an error. During navigation and manipulation, a grasp can miss, a gripper can open unexpectedly, or the robot can drift away from its intended path. Small deviations can accumulate across a long task. Successful demonstrations show how a task should unfold, but by themselves do not provide examples of how to recover when execution leaves that path.
Our goal is to build controlled counterfactual datasets for error recognition and recovery. Starting from a successful demonstration, we introduce deliberate, configurable disturbances and compare alternative continuations of the same task:
- Successful source: what happens without the injected disturbance?
- Perturbed control: after the disturbance, what happens if the robot continues the original remaining actions without feedback?
- Recovery: from the same verified first perturbed state, what happens if the robot observes the current state and changes its actions?
These paired trajectories are intended to support research on detecting deviations, choosing corrective actions, replanning, and completing navigation and manipulation tasks despite interruptions. Users can vary when disturbances occur, their type, strength, duration and repetition, so the dataset includes more than one scripted failure case. Failed recoveries are retained to expose the limits of the current controller.
Here, “counterfactual” means alternative executions constructed under controlled simulator conditions. A qualified correction requires a successful source, a verified deviation, a failed perturbed control and a successful recovery. The comparison does not establish a general causal effect or a learned model's robustness: training benefits and transfer to unseen errors or real robots still need separate evaluation. Pairing protocol and evidence requirements.
Five new task experts now have native successful sources. YCB grasping: 4/5 new seeds; RoboCasa additions remain experimental. Measured results and videos · Launch examples.
Success-source gaps and repair priorities
Per-task baselines and calibrated perturbations · Calibration CLI and acceptance rules
Latest per-task calibration: 56 task classes measured; 28 passed the recorded small-sample held-out gate (25 MetaWorld, 3 ManiSkill). Another 23 MetaWorld classes require a fuller scene-identity check, 3 classes miss the baseline target, and 2 have no qualifying perturbation. This is not completion of the 546-class inventory.
Planner tools, task coverage and execution success are separate. See success rates and planner diagnostics for the native tools already used, the optional MPlib planner portfolio and matched-run measurement.
Previous inventory snapshot (before calibration): 1,322 saved trajectory records; 248 qualified correction groups in the published experiments. The raw count excludes exact file copies and includes earlier versions, diagnostics and failures. Definitions and per-environment counts. Native RoboCasa now combines navigation, grasping, carrying and placement, with three successful recovery schedules in one fixed scene. Videos and launch commands.
RoboTwin video library: 50 task classes and 62 playable clips. Sources, corrections and failed recoveries · Per-benchmark perturbation types, units and recovery methods · Parameter reference.
New: native ManiSkill / experimental RoboTwin action streaming, task-level coverage, and 144 real animations embedded below. Coverage and launch commands.
Configurable multi-error schedules. Choose source step / fraction / milestone, type, strength, duration, repetition and gap. Interactive configuration & examples · 10 launch recipes and Python API. All 50 MetaWorld task classes now have at least one verified correction sample across the documented experiments: the first sweep gave 69/100 qualified pairs across 48 tasks; a separate tool-task follow-up gave 18/40 and covered the remaining two. This is task-class coverage, not universal recovery or an MT50 benchmark score.
Collect successful demonstrations, induce execution errors, and verify recovery in real simulator dynamics.
Live dashboard · 中文 · Correction protocol · Collection results · Contributing
Robot Stack is a research collection toolkit for paired error-and-recovery data. It separates simulator adapters, task policies, native success checks and dataset bookkeeping. It retains failed attempts, counts every injected/recovery action, and checks the same first perturbed state. Later scheduled interventions match commands and nominal ticks; branch states may diverge after feedback starts.
Current controllers use privileged simulator state or upstream scripted experts. These are collection experiments, not learned-policy benchmark scores. New tasks still need a capable policy and an appropriate success oracle.
Native MetaWorld action replay: source / perturbed control / recovery. Rendered at a viewing frame rate; not physical-time playback.
RoboTwin now has successful sources for all 50 task classes. Initial fixed-seed sweep: 37/50; separate follow-up: 11/26; final bounded bootstrapping: 5/13, retaining all failures. Qualified recovery now covers dual-arm stacking and cup placement, not all 50. Evidence.
Seed 1100: 1,984 / 2,224 / 3,722 real physics steps. Source succeeds, perturbed control fails, replanning succeeds. All three independent replays have zero state error. Six-attempt report, including failures.
source | perturbed | recovery |
Native PandaOmron, PickPlaceCounterToSink, layout/style 1, apple, seed 1100. Three schedules share one scene: source succeeds, perturbed control fails, feedback recovery succeeds. All nine trajectories independently replay with zero state error. Videos and reproduction · All outcomes, including the earlier failed recovery.
| 移动中错误转向 | ||
|---|---|---|
正常执行 · 成功 | 错误后原动作 · 失败 | 当前状态恢复 · 成功 |
| 持物时机械臂偏移 | ||
正常执行 · 成功 | 错误后原动作 · 失败 | 当前状态恢复 · 成功 |
| 抬起后故意张爪 | ||
正常执行 · 成功 | 错误后原动作 · 失败 | 当前状态恢复 · 成功 |
Every animation below is recorded from native simulation. Labels distinguish source demonstrations, verified recovery and MimicGen augmentation. Viewing speed is not simulation time. Search all tasks and coverage.
MetaWorld · repeated errors Recovery / 3 interventions | MetaWorld · tool use Recovery / stick pulling |
AI2-THOR · six destinations Recovery / 3 navigation errors | RoboCasa · kitchen navigation Recovery / mobile base |
ManiSkill · cube stacking Successful source demonstration | RoboTwin · dual-arm stacking Successful source demonstration |
MimicGen · generated success Augmented demonstration | MimicGen · retained failure Failure / not a recovery sample |
Seed 101: source succeeds in 3,935 physics steps; perturbed control fails after 4,175; recovery succeeds in 7,748. All three independently replay with zero state error. The four-attempt experiment yielded three qualified pairs; the failure is retained. Replay evidence.
| Source · success | Perturbed · failure | Recovery · success |
|---|---|---|
 |  |  |
Each row shows native trajectories from the same seed and demonstration: successful source, failed perturbed control, and successful replanned recovery. All 21 independently replayed with zero recorded state error. Replay evidence.
| Source · success | Perturbed · failure | Recovery · success |
|---|---|---|
| LiftPegUpright-v1 · seed 810 | ||
 |  |  |
| PegInsertionSide-v1 · seed 810 | ||
 |  |  |
| PickCube-v1 · seed 810 | ||
 |  |  |
| PlaceSphere-v1 · seed 800 | ||
 |  |  |
| PullCube-v1 · seed 811 | ||
 |  |  |
| PushCube-v1 · seed 811 | ||
 |  |  |
| StackCube-v1 · seed 810 | ||
 |  |  |
Expand 37 successful RoboTwin source-task animations
These are the 37 successful source tasks from the unfiltered 50-task sweep at seed 900, recorded during native collection. Source demonstrations here are not counted as correction pairs. The other 13 failures/errors remain in the full report. All 50 attempts.
adjust_bottle 82 recorded frames · source | beat_block_hammer 65 recorded frames · source | blocks_ranking_rgb 254 recorded frames · source |
blocks_ranking_size 264 recorded frames · source | click_alarmclock 52 recorded frames · source | click_bell 44 recorded frames · source |
grab_roller 52 recorded frames · source | handover_block 169 recorded frames · source | handover_mic 124 recorded frames · source |
hanging_mug 196 recorded frames · source | lift_pot 61 recorded frames · source | move_can_pot 83 recorded frames · source |
move_playingcard_away 63 recorded frames · source | move_stapler_pad 80 recorded frames · source | pick_dual_bottles 70 recorded frames · source |
place_a2b_right 81 recorded frames · source | place_burger_fries 132 recorded frames · source | place_cans_plasticbox 159 recorded frames · source |
place_container_plate 89 recorded frames · source | place_dual_shoes 128 recorded frames · source | place_fan 87 recorded frames · source |
place_mouse_pad 86 recorded frames · source | place_object_basket 139 recorded frames · source | place_object_scale 80 recorded frames · source |
place_object_stand 77 recorded frames · source | place_phone_stand 71 recorded frames · source | press_stapler 65 recorded frames · source |
put_bottles_dustbin 364 recorded frames · source | rotate_qrcode 85 recorded frames · source | shake_bottle 136 recorded frames · source |
shake_bottle_horizontally 154 recorded frames · source | stack_blocks_three 254 recorded frames · source | stack_blocks_two 173 recorded frames · source |
stack_bowls_three 270 recorded frames · source | stack_bowls_two 183 recorded frames · source | stamp_seal 77 recorded frames · source |
turn_switch 53 recorded frames · source |
Expand 13 additional source-task successes from follow-up collection
From separately reported fixed-seed follow-ups and bounded success-target collection. These source successes are not correction pairs or part of the initial sweep rate. Each animation links to its experiment report.
dump_bin_bigbin seed 902 · 190 frames · source | move_pillbottle_pad seed 901 · 85 frames · source | open_laptop seed 901 · 84 frames · source |
open_microwave seed 903 · 293 frames · source | pick_diverse_bottles seed 905 · 68 frames · source | place_a2b_left seed 901 · 82 frames · source |
place_bread_basket seed 901 · 128 frames · source | place_bread_skillet seed 903 · 96 frames · source | place_can_basket seed 903 · 136 frames · source |
place_empty_cup seed 902 · 102 frames · source | place_shoe seed 901 · 102 frames · source | scan_object seed 902 · 92 frames · source |
put_object_cabinet seed 909 · 149 frames · source |
One qualified recovery example per task class, chosen from the documented experiments. Each shown recovery was independently replayed again with zero recorded state error. Click an animation for its replay evidence. Source/control/recovery triplets also passed 150 independent replays. These are example-level results, not a guarantee for all seeds, perturbations or MT50 evaluation settings.
assembly-v3 seed 600 · 219 actions | basketball-v3 seed 600 · 174 actions | bin-picking-v3 seed 600 · 220 actions |
box-close-v3 seed 600 · 158 actions | button-press-topdown-v3 seed 600 · 104 actions | button-press-topdown-wall-v3 seed 600 · 94 actions |
button-press-v3 seed 600 · 97 actions | button-press-wall-v3 seed 600 · 110 actions | coffee-button-v3 seed 600 · 79 actions |
coffee-pull-v3 seed 600 · 103 actions | coffee-push-v3 seed 600 · 91 actions | dial-turn-v3 seed 600 · 117 actions |
disassemble-v3 seed 600 · 203 actions | door-close-v3 seed 600 · 73 actions | door-lock-v3 seed 600 · 115 actions |
door-open-v3 seed 600 · 121 actions | door-unlock-v3 seed 600 · 88 actions | drawer-close-v3 seed 600 · 108 actions |
drawer-open-v3 seed 600 · 149 actions | faucet-close-v3 seed 600 · 95 actions | faucet-open-v3 seed 600 · 92 actions |
hammer-v3 seed 600 · 103 actions | hand-insert-v3 seed 600 · 95 actions | handle-press-side-v3 seed 600 · 80 actions |
handle-press-v3 seed 600 · 79 actions | handle-pull-side-v3 seed 600 · 115 actions | handle-pull-v3 seed 600 · 156 actions |
lever-pull-v3 seed 600 · 105 actions | peg-insert-side-v3 seed 600 · 125 actions | peg-unplug-side-v3 seed 600 · 151 actions |
pick-out-of-hole-v3 seed 600 · 178 actions | pick-place-v3 seed 600 · 88 actions | pick-place-wall-v3 seed 600 · 183 actions |
plate-slide-back-side-v3 seed 600 · 95 actions | plate-slide-back-v3 seed 600 · 81 actions | plate-slide-side-v3 seed 600 · 213 actions |
plate-slide-v3 seed 600 · 88 actions | push-back-v3 seed 600 · 116 actions | push-v3 seed 600 · 96 actions |
push-wall-v3 seed 600 · 151 actions | reach-v3 seed 600 · 65 actions | reach-wall-v3 seed 600 · 68 actions |
shelf-place-v3 seed 600 · 150 actions | soccer-v3 seed 600 · 143 actions | stick-pull-v3 seed 621 · 186 actions |
stick-push-v3 seed 622 · 105 actions | sweep-into-v3 seed 600 · 114 actions | sweep-v3 seed 600 · 144 actions |
window-close-v3 seed 600 · 114 actions | window-open-v3 seed 600 · 122 actions |
A successful endpoint alone does not prove correction. Each eligible sample contains:
flowchart LR
A[Successful source] --> B[Replay a verified action prefix]
B --> C[Execute perturbation actions]
C --> D[Original remaining actions]
C --> E[Feedback recovery]
D --> F[Control fails]
E --> G[Native task succeeds]
F --> H[Qualified correction pair]
G --> H
Both branches reconstruct the source prefix through actual actions. Recovery never resets the environment, teleports the robot, attaches the object, or installs a successful simulator state. The protocol measures recovery from a specified injected error; it does not claim superiority over another closed-loop policy.
| Integration | Native collection evidence | Perturbation / recovery |
|---|---|---|
| MetaWorld full task-class sweep | All 50 registered classes have qualified samples across first sweep + tool follow-up | First sweep 69/100; separate follow-up 18/40. One triplet per class: 150/150 native replays pass, state error 0. Full conditions |
| MetaWorld 3.1.1 | 8 selected tasks, 80/80 successful demonstrations | Generic three-branch protocol: 100/100 qualified pairs across push and pick-place, 50 native task instances each |
| robosuite Lift | 20/20 clean, 0/20 grasp-offset control, 20/20 recovery | Earlier bounded retry experiment; distinct from the new source-prefix branching protocol |
| ManiSkill 3.0.1 | Current 12-expert sweep: 19/24 successful sources across 10 classes; four setup errors retained | 8/24 qualified pairs; combined with the first trial, 7 classes have qualified examples. Conditions |
| RoboTwin | All 50 native experts attempted: 37/50 successful sources at seed 900 | stack_blocks_two: 3/4 qualified pairs, 12 files audited; arbitrary-task recovery remains experimental. Conditions |
| RoboCasa 1.0.1 | NavigateKitchen, layout/style 1, seeds 220–222 | 3/3 qualified navigation correction pairs; seed 220 independently replayed with zero state error |
| AI2-THOR 5.0.0 | Native FloorPlan1 PointNav, seeds 220–222 | 3/3 qualified navigation correction pairs; independent recovery replay has zero pose error |
| RoboDojo | Shared official assets and Isaac Sim 5.1 / IsaacLab environment located | Native stack_bowls reset + 10 physics steps verified; task policy success remains unverified |
| MimicGen | Official generator, two Lift sources → ten new trajectories | 7/10 successful; all ten independently replayed; augmented demos, not correction pairs |
The 100 MetaWorld pairs are a fixed-configuration experiment: task indices 0–49, reset seeds 300–349, a 15-step Cartesian error burst, and a 500-action cap. All failures and non-qualifying attempts remain in the dataset. This is not complete MT10/MT50 evaluation. See data semantics and earlier benchmark conditions.
Caching task definitions reduced measured collection time from 155 to 43 seconds (push) and 150 to 45 seconds (pick-place), including reconstruction and saving. All 300 trajectory files had identical states, actions, phases and success labels before/after the optimization. These concurrent shared-server measurements are not isolated throughput benchmarks. Audit and timing evidence. The repeated timing baseline and earlier 10-pair pilot are excluded from the 100-pair total.
Use a separate Python 3.10 environment for each simulator. The core itself needs only NumPy and h5py.
git clone https://github.com/pm1255/robot_stack.git
cd robot_stack
python -m venv .venv
source .venv/bin/activate
pip install -e '.[metaworld]'
MUJOCO_GL=egl OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 \
robot-stack-collect --backend metaworld --task pick-place-v3 \
--output outputs/pick-place --episodes 50 --seed-start 300 \
--max-steps 500 --perturb-steps 15
python -m robot_stack.audit outputs/pick-place --output outputs/pick-place-audit.json
robot-stack-replay outputs/pick-place/ep_0300/recovery.hdf5 \
--output outputs/replay.json --video outputs/recovery.mp4Replay video needs EGL and FFmpeg. Each new run requires a new output directory. MetaWorld uses different native task indices by default; setting task_index explicitly in --adapter-options intentionally holds that task instance fixed.
outputs/pick-place/
run_config.json
summary.json
ep_0220/
source.hdf5 # successful source
perturbed.hdf5 # original suffix after injected actions
recovery.hdf5 # feedback from the same error state
*_episode_result.json # native outcome, provenance, SHA256
correction.json # eligibility and paired-state checks
The HDF5 files contain T actions, T post-action success labels and T+1 states. Phase labels and half-open action ranges identify the perturbation and recovery. Use split_group, not episode filenames, when splitting training and validation; identical initial states are conservatively grouped even if reset seeds differ.
The new adapter API is in robot_stack/core.py. A policy chooses actions; an adapter executes them, returns numeric state and evaluates the task. The core handles branching, provenance, budgets and paired labels. Implement an adapter and call collect_triplet(...) to use a different policy or task.
- RoboCasa: the current controller targets native
NavigateKitchenusing the native base Jacobian, bounded friction compensation and PandaOmron base velocities and the environment's position/orientation success check. It requires matching kitchen assets. It is not yet a general collision-aware mobile manipulation planner. - AI2-THOR: the current adapter performs oracle PointNav in a native
FloorPlanscene usingGetReachablePositions, grid planning, physical moves and rotations. This is not official ObjectNav evaluation. It records agent pose in a static scene, not a full Unity snapshot. - RoboDojo: the bridge accepts a caller-created official
EvalEnv, a real policy callback, a state reader and a perturbation generator.success=Truealone is insufficient: native completion and actual execution are also required. See integration setup.
The original Isaac/A2D pipeline remains in isaac_collector/; its planner/execution checks have been improved, but its task completion is still unverified. Implementation notes document that boundary.
A small native correction triplet is included for inspection. Validate its files without a simulator: python -m robot_stack.audit examples/correction --output /tmp/robot-stack-example-audit.json.
python scripts/build_collection_report.py --input outputs/pick-place --output docs/report.json
python -m http.server 8000 --directory docs
python -m unittest discover -s tests -vThe static dashboard shows real records and local videos, and can be deployed with the GitHub Pages workflow. CI runs lightweight contract tests; native simulator checks are separate and must not be inferred from a green CI badge.
MimicGen now has a native prepare → generate → replay path for Panda Lift. Preparation verifies source dynamics and derives grasp/lift subtask annotations; generation retains failures and checks final stable success. Reproduction commands and measured limits. Extending it to another task still requires its object frames, action conversion and subtask boundaries.
Next milestones are collision-aware navigation, more task policies, controlled natural-failure recovery and MimicGen integration for additional tasks. A simulator name or HDF5 extension is not a compatibility guarantee.
We build on robosuite, MetaWorld, ManiSkill, RoboTwin, RoboCasa, RoboDojo, AI2-THOR, and MimicGen. Their code, assets and policies retain their upstream terms and attribution.