Motion Before Action: Diffusing Object Motion as Manipulation Condition

Authors: Yue Su*, Xinyu Zhan*, Hongjie Fang, Yong-Lu Li, Cewu Lu, Lixin Yang†

🛫 Getting Started

🌟 About Sim

You can view the whole eval results in the sim bench at

💻 Installation

We use RISE as our real robot baseline, Please following the installation guide to install the mba conda environments and the dependencies, as well as the real robot environments. Also, remember to adjust the constant parameters in dataset/constants.py and utils/constants.py according to your own environment.

📷 Calibration

Please calibrate the camera(s) with the robot before data collection and evaluation to ensure correct spatial transformations between camera(s) and the robot. Please refer to calibration guide for more details.

🛢️ Data Collection

You can view the sampled data (cut task) from this link, which contains record info, mocap records and task data (one trajectory for instance). We follow the data managemnet pattern as RH20T.

Task_name
`-- train/
    |-- [episode identifier 1]
    |   |-- metadata.json              # metadata
    |   |-- timestamp.txt              # calib timestamp  
    |   |-- cam_[serial_number 1]/    
    |   |   |-- color                  # RGB
    |   |   |   |-- [timestamp 1].png
    |   |   |   |-- [timestamp 2].png
    |   |   |   |-- ...
    |   |   |   `-- [timestamp T].png
    |   |   |-- depth                  # depth
    |   |   |   |-- [timestamp 1].png
    |   |   |   |-- [timestamp 2].png
    |   |   |   |-- ...
    |   |   |   `-- [timestamp T].png
    |   |   |-- tcp                    # tcp
    |   |   |   |-- [timestamp 1].npy
    |   |   |   |-- [timestamp 2].npy
    |   |   |   |-- ...
    |   |   |   `-- [timestamp T].npy
    |   |   `-- gripper_command        # gripper command
    |   |       |-- [timestamp 1].npy
    |   |       |-- [timestamp 2].npy
    |   |       |-- ...
    |   |       `-- [timestamp T].npy
    |   `-- cam_[serial_number 2]/     # similar camera structure
    `-- [episode identifier 2]         # similar episode structure

🧑🏻‍💻 Training

The training scripts are saved in script.

conda activate mba
bash script/command_train__place__mba.sh # Add MBA module
bash script/command_train__place.sh # For baseline

🤖 Evaluation

Please follow the deployment guide to modify the evaluation script.

Modify the arguments in command_eval.sh, then

conda activate mba
bash command_eval.sh

✍️ Citation

@ARTICLE{MBA,
  author={Su, Yue and Zhan, Xinyu and Fang, Hongjie and Li, Yong-Lu and Lu, Cewu and Yang, Lixin},
  journal={IEEE Robotics and Automation Letters}, 
  title={Motion Before Action: Diffusing Object Motion as Manipulation Condition}, 
  year={2025},
  volume={10},
  number={7},
  pages={7428-7435},
 doi={10.1109/LRA.2025.3577424}}

📃 License

MBA is licensed under CC BY-NC-SA 4.0