Code of the paper "Exploring Vision Transformers for 3D Human Motion-Language Models with Motion Patches" (CVPR 2024).
- Python 3.11
- PyTorch 2.0.1+
Using Poetry (Recommended)
poetry install
conda create -n MoPa python=3.11
conda activate MoPa
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia
pip install -r requirements.txt
Download HumanML3D Dataset and KIT-ML Dataset from the repository of HumanML3D. Unzip and locate them in the data
folder.
The whole directory should be look like this:
MotionPatches
│ README.md
│ requirements.txt
| ...
|
└───conf
└───scripts
└───...
│
└───data
└───HumanML3D
| └───new_joint_vecs
| └───new_joints
| └───...
│
└───KIT-ML
└───new_joint_vecs
└───new_joints
└───...
Then calculate the mean and variance of each dataset by:
python scripts/cal_mean_var.py
Download pre-trained model from huggingface and put them in checkpoints/pretrained/
.
Evaluate the model with HumanML3D via retrieval:
python scripts/test.py dataset=HumanML3D exp_name=pretrained
Evaluate the model with KIT-ML via retrieval:
python scripts/test.py dataset=KIT-ML exp_name=pretrained
Using scripts/test_batch.py
can get the result of small batches with 32 samples.
Train the model with HumanML3D:
python scripts/train.py dataset=HumanML3D
Train the model with KIT-ML:
python scripts/train.py dataset=KIT-ML
Evaluate the model with HumanML3D via retrieval:
python scripts/test.py dataset=HumanML3D
Evaluate the model with KIT-ML via retrieval:
python scripts/test.py dataset=KIT-ML
Using scripts/test_batch.py
can get the result of small batches with 32 samples.
@InProceedings{yu2024exploring,
title={Exploring Vision Transformers for 3D Human Motion-Language Models with Motion Patches},
author={Yu, Qing and Tanaka, Mikihiro and Fujiwara, Kent},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2024}
}
Additionally, this repository contains third-party software. Refer NOTICE.txt for more details and follow the terms and conditions of their use.