Nikos Athanasiou · Mathis Petrovich · Michael J. Black · Gül Varol
Check our upcoming YouTube video for a quick overview and our paper for more details.
This implementation:
- Instruction on how to prepare the datasets used in the experiments.
- The training code:
- For both baselines
- For TEACH method
- A simple interacting demo that given some prompts with texts and durations returns back:
- a
npyfile containing the vertices of the body generated by TEACH. - a video that demonstrates the result.
- a
To be uploaded:
- Instructions about the baselines and how to run them.
- Instructions for sampling and evaluating with the code all of the models.
- The rendering code for the blender renderings used in the paper.
TEACH has been implemented and tested on Ubuntu 20.04 with python >= 3.9.
Clone the repo:
git clone https://github.com/athn-nik/teach.gitAfter it do this to install DistillBERT:
cd deps/
git lfs install
git clone https://huggingface.co/distilbert-base-uncased
cd ..Install the requirements using virtualenv :
# pip
source scripts/install.shYou can do something equivalent with conda as well.
We have prepared a nice demo code to run TEACH on arbitrary videos.
First, you need download the required data(i.e our trained model from our website).
The path/to/experiment directory should look like:
experiment
│
└───.hydra
│ | config.yaml
| | overrides.yaml
| | hydra.yaml
|
└───checkpoints
│ last.ckpt
Then, running the demo is as simple as:
python interact_teach.py folder=/path/to/experiment output=/path/to/yourfname texts='[text prompt1, text prompt2, text prompt3, <more prompts comma divided>]' durs='[dur1, dur2, dur3, ...]'
Download the data from AMASS website. Then, run this command to extract the amass sequences that are annotated in babel:
python scripts/process_amass.py --input-path /path/to/data --output-path path/of/choice/default_is_/babel/babel-smplh-30fps-male --use-betas --gender maleDownload the data from TEACH website, after signing in. The data TEACH was trained was a processed version of BABEL. Hence, we provide them directly to your via our website, where you will also find more relevant details. Finally, download the male SMPLH male body model from the SMPLX website. Specifically the AMASS version of the SMPLH model. Then, follow the instructions here to extract the smplh model in pickle format.
The run this script and change your paths accordingly inside it extract the different babel splits from amass:
python scripts/amass_splits_babel.pyThen create a directory named data and put the babel data and the processed amass data in.
You should end up with a data folder with the structure like this:
data
|-- amass
| `-- your-processed-amass-data
|
|-- babel
| `-- babel-teach
| `...
| `-- babel-smplh-30fps-male
| `...
|
|-- smpl_models
| `-- smplh
| `--SMPLH_MALE.pkl
Be careful not to push any data! Then you should softlink inside this repo. To softlink your data, do:
ln -s /path/to/data
To start training after activating your environment. Do:
python train.py experiment=baseline logger=noneExplore configs/train.yaml to change some basic things like where you want
your output stored, which data you want to choose if you want to do a small
experiment on a subset of the data etc.
[TODO]: More on this coming soon.
Here are some commands if you want to sample from the validaiton set and evaluate on the metrics reported in the paper:
python sample_seq.py folder=/path/to/experiment align=full slerp_ws=8In general the folder is: folder_our/<project>/<dataname_config>/<experimet>/<run_id>
This folder should contain a checkpoints directory with a last.ckpt file inside and a .hydra directory from which the configuration
will be pulled and the relevant checkpoint. This folder is created during training in the output directory and is provided in our website
for the experiments in the paper.
align=trans: chooses if translation will be aligned or if the global orientation also(align=full)slerp_ws: decides on whether slerp is done or not(=null) and what is the size of its window.
Then for the evaluation you should do:
python eval.py folder=/path/to/experiment align=true slerp=truethe two extra parameters decide the samples on which the evaluation will be performed.
-
Without alignment column:
python compute_td.py folder=/path/to/experiment align_full_bodies=false align_only_trans=true -
With alignment column:
python compute_td.py folder=/path/to/experiment align_full_bodies=true align_only_trans=false
[TODO]: More on this coming soon.
@inproceedings{TEACH:3DV:2022,
title={TEACH: Temporal Action Compositions for 3D Humans},
author={Athanasiou, Nikos and Petrovich, Mathis and Black, Michael J. and Varol, G\"{u}l },
booktitle = {International Conference on 3D Vision (3DV)},
month = {September},
year = {2022}
}This code is available for non-commercial scientific research purposes as defined in the LICENSE file. By downloading and using this code you agree to the terms in the LICENSE. Third-party datasets and software are subject to their respective licenses.
We thank Benjamin Pellkofer for his IT support.
Many part of this code were based on the official implementation of TEMOS. Here are some great resources we benefit:
- SMPL models and layer is from SMPL-X model.
This code repository was implemented mainly by Nikos Athanasiou with the help of Mathis Petrovich.
Give a ⭐ if you like.
For commercial licensing (and all related questions for business applications), please contact [email protected].
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent