Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation

This repository provides simulation training and real-robot deployment for PRISM, based on HoloSoma.

Code branches

Branch	Use
main	Simulation: teacher training, rollout collection and student distillation
sim2real	Real-robot deployment with FastFoundationStereo

For real-robot deployment, switch an existing clone to sim2real and follow that branch's README:

git fetch origin
git switch sim2real
git submodule update --init --recursive

Installation

Linux, Python 3.11 and a compatible NVIDIA GPU are required. See system requirements and environment details.

git clone --branch main https://github.com/amazon-far/PRISM.git
cd PRISM
python3.11 -m venv .venv
source .venv/bin/activate
bash install.sh
wandb login

Data

Download and prepare the Hugging Face dataset under data/:

bash download_data.sh

Includes object meshes and prepares the student training shards. Dataset details.

Both training scripts use all visible GPUs on this machine. Set CUDA_VISIBLE_DEVICES=0 for one GPU or CUDA_VISIBLE_DEVICES=0,1 for a subset. Use --envs-per-gpu to lower memory use. Training details.

Teacher Training

Train a privileged motion-tracking policy with PPO. Teacher-training data is still under review; supply a prepared teacher bank.

bash train_teacher.sh --motion-bank /path/to/teacher_bank --entity YOUR_WANDB_ENTITY

Rollout

Collect teacher trajectories and contact sidecars into outputs/rollout/.

bash rollout.sh --motion-bank /path/to/teacher_bank

Student Distillation

Distill the teacher into a depth policy using the prepared data under data/.

bash train_student.sh --entity YOUR_WANDB_ENTITY

Citation

If you use PRISM in your research, please cite:

@article{wang2026counterfactual,
  title={Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation},
  author={Wang, Zihan and Wu, Zhen and Abbeel, Pieter and Duan, Rocky and Malik, Jitendra and Sferrazza, Carmelo and Liu, C. Karen and Shi, Guanya and Kanazawa, Angjoo},
  journal={arXiv preprint arXiv:2609.38172},
  year={2026}
}

License and security

See LICENSE, NOTICE and THIRD_PARTY_LICENSES.

See CONTRIBUTING for security reporting.