Steins Gate competition: final Best submission and verification record.
DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic States
Bozhou Zhang, Nan Song, Li Zhang
School of Data Science, Fudan University
NeurIPS 2024
DeMo++: Motion Decoupling for Autonomous Driving
Bozhou Zhang, Nan Song, Xiatian Zhu, Li Zhang
arXiv preprint arXiv:2507.17342
Accurate motion forecasting for traffic agents is crucial for ensuring the safety and efficiency of autonomous driving systems in dynamically changing environments. Mainstream methods adopt a one-query-one-trajectory paradigm, where each query corresponds to a unique trajectory for predicting multi-modal trajectories. While straightforward and effective, the absence of detailed representation of future trajectories may yield suboptimal outcomes, given that the agent states dynamically evolve over time. To address this problem, we introduce DeMo, a framework that decouples multi-modal trajectory queries into two types: mode queries capturing distinct directional intentions and state queries tracking the agent's dynamic states over time. By leveraging this format, we separately optimize the multi-modality and dynamic evolutionary properties of trajectories. Subsequently, the mode and state queries are integrated to obtain a comprehensive and detailed representation of the trajectories. To achieve these operations, we additionally introduce combined Attention and Mamba techniques for global information aggregation and state sequence modeling, leveraging their respective strengths. Extensive experiments on both the Argoverse 2 and nuScenes benchmarks demonstrate that our DeMo achieves state-of-the-art performance in motion forecasting.
7 Dec, 2024: Checkout our work RealMotion [NeurIPS 2024 Spotlight] 🥰, which proposes a novel motion forecasting framework for continuous driving.
conda create -n DeMo python=3.10
conda activate DeMo
# Install dependency packpages
pip install torch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 --index-url https://download.pytorch.org/whl/cu118
pip install -r ./requirements.txt
pip install av2==0.2.1
# Some packages may be useful
pip install tensorboard
pip install torch-scatter -f https://data.pyg.org/whl/torch-2.1.1+cu118.html
pip install protobuf==3.20.3
pip install numpy==1.26.3
pip install transformers==4.30
# Install Mamba, we follow the settings outlined in [VideoMamba](https://github.com/OpenGVLab/VideoMamba).
cd third_party/causal-conv1d
pip install . --no-build-isolation
cd third_party/mamba
pip install . --no-build-isolation
data_root
├── train
│ ├── 0000b0f9-99f9-4a1f-a231-5be9e4c523f7
│ ├── 0000b6ab-e100-4f6b-aee8-b520b57c0530
│ ├── ...
├── val
│ ├── 00010486-9a07-48ae-b493-cf4545855937
│ ├── 00062a32-8d6d-4449-9948-6fedac67bfcd
│ ├── ...
├── test
│ ├── 0000b329-f890-4c2b-93f2-7e2413d4ca5b
│ ├── 0008c251-e9b0-4708-b762-b15cb6effc27
│ ├── ...
python preprocess.py --data_root=/path/to/data_root -p
└── data
└── DeMo_processed
├── train
├── val
└── test
# Train
python train.py
# Val, remember to change the checkpoint to your own in eval.py
python eval.py
# Test for submission
python eval.py gpus=1 test=true
# Visualization
python visual.py
@inproceedings{zhang2024demo,
title={DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic States},
author={Zhang, Bozhou and Song, Nan and Zhang, Li},
booktitle={NeurIPS},
year={2024},
}