Official PyTorch implementation of the paper "TEMOS: Generating diverse human motions from textual descriptions", ECCV 2022 (Oral).
Please visit our webpage for more details.
If you find this code useful in your research, please cite:
@inproceedings{petrovich22temos,
title = {{TEMOS}: Generating diverse human motions from textual descriptions},
author = {Petrovich, Mathis and Black, Michael J. and Varol, G{\"u}l},
booktitle = {European Conference on Computer Vision ({ECCV})},
year = {2022}
}
You can also put a star ⭐, if the code is useful to you.
Click to expand
Instructions
conda create python=3.9 --name temos
conda activate temos
Install PyTorch 1.10 inside the conda environment, and install the following packages:
pip install pytorch_lightning --upgrade
pip install torchmetrics==0.7
pip install hydra-core --upgrade
pip install hydra_colorlog --upgrade
pip install shortuuid
pip install rich
pip install pandas
pip install transformers
pip install psutil
pip install einopsThe code was tested on Python 3.9.7 and PyTorch 1.10.0.
Instructions
Be sure to read and follow their license agreements, and cite accordingly.
Use the code from Ghosh et al. to download and prepare the kit dataset (extraction of xyz joints coodinates data from axis-angle Master Motor Map). Move or copy all the files which ends with "_meta.json", "_annotations.json" and "_fke.csv" inside the datasets/kit folder.
"
These motions are process by the Master Motor Map (MMM) framework. To be able to generate motions with SMPL body model, please look at the next section.
Be sure to read and follow their license agreements, and cite accordingly.
Create this folder:
mkdir datasets/AMASS/Go to the AMASS website, register and go to the Download tab. Then download the "SMPL+H G" files corresponding to the datasets [KIT, CMU, EKUT] into the datasets/AMASS directory and uncompress the archives:
cd datasets/AMASS/
tar xfv CMU.tar.bz2
tar xfv KIT.tar.bz2
tar xfv EKUT.tar.bz2
cd ../../Instructions
cd deps/
git lfs install
git clone https://huggingface.co/distilbert-base-uncased
cd ..Instructions
This is only useful if you want to use generate 3D human meshes like in the teaser. In this case, you also need a subset of the AMASS dataset (see instructions below).
Go to the MANO website, register and go to the Download tab.
- Click on "Models & Code" to download
mano_v1_2.zipand place it in the folderdeps/smplh/. - Click on "Extended SMPL+H model" to download
smplh.tar.xzand place it in the folderdeps/smplh/.
The next step is to extract the archives, merge the hands from mano_v1_2 into the Extended SMPL+H models, and remove any chumpy dependency.
All of this can be done using with the following commands. (I forked both scripts from this repo SMPLX repo, updated them to Python 3, merged them, and made it compatible with .npz files).
pip install scipy chumpy
bash prepare/smplh.shThis will create SMPLH_FEMALE.npz, SMPLH_MALE.npz, SMPLH_NEUTRAL.npz inside the deps/smplh folder.
Instructions
Make sure to have gdown installed
pip install --user gdownThen, please run this command line:
bash prepare/download_pretrained_models.shInside the pretrained models folder, you will find one for each type of data (see Section datasets below for more information).
pretrained_models
├── kit-amass-rot
│ └── 1cp6dwpa
├── kit-amass-xyz
│ └── 5xp9647f
└── kit-mmm-xyz
└── 3l49g7hv
Click to expand
The command to launch a training experiment is the folowing:
python train.py [OPTIONS]The parsing is done by using the powerful Hydra library. You can override anything in the configuration by passing arguments like foo=value or foo.bar=value.
Each training will create a unique output directory (referred to as FOLDER below), where logs, configuations and checkpoints are stored.
By default it is defined as outputs/${data.dataname}/${experiment}/${run_id} with data.dataname the name of the dataset (see examples below), experiment=baseline and run_id a 8 unique random alpha-numeric identifier for the run (everything can be overridden if needed).
This folder is printed during logging, it should look like outputs/kit-mmm-xyz/baseline/3gn7h7v6/.
data=kit-mmm-xyz: KIT-ML motions processed by the MMM framework (as in the original data) loaded as xyz joint coordinates (after axis-angle transformation → xyz) (by default)data=kit-amass-rot: KIT-ML motions loaded as SMPL rotations and translations, from AMASS (processed with MoSh++)data=kit-amass-xyz: KIT-ML motions loaded as xyz joint coordinates, from AMASS (processed with MoSh++) after passing through a SMPL layer and regressing the correct joints.
trainer=gpu: training with CUDA, on an automatically selected GPU (default)trainer=cpu: training on the CPU (not recommended)
Click to expand
To get results comparable to previous work, we use the same splits as in Language2Pose and Ghosh et al.. To be explicit, and not rely on random seeds, you can find the list of id-files in datasets/kit-splits/ (train/val/test).
When sampling Ghosh et al.'s motions with their code, I noticed that their dataloader is missing some sequences (see the discussion here).
In order to compare all the methods with the same test set, we use the 520 sequences produced by Ghosh et al. code for the test set (instead of the 587 sequences). This split is refered as gtest (for "Ghosh test"). It is used per default in the sampling/evaluation/rendering code. You can change this set by specifying split=SPLIT in each command line.
You can also find in datasets/kit-splits/, the split used for the human-study (human-study) and the split used for the visuals of the paper (visu).
The command line to sample one motion per sequence is the following:
python sample.py folder=FOLDER [OPTIONS]This command will create the folder FOLDER/samples/SPLIT and save the motions in the npy format.
mean=false: Take the mean value for the latent vector, instead of sampling (default is false)number_of_samples=X: GenerateXmotions (by default it generates only one)fact=X: Multiplies sigma byXduring sampling (1.0 by default, diversity can be increased whenfact>1)
If your model has been trained with data=kit-amass-rot, it produces SMPL rotations and translations. In this case, you can specify the type of data you want to save after passing through the SMPL layer.
jointstype=mmm: Generate xyz joints compatible with the MMM bodies (by default). This gives skeletons comparable todata=kit-mmm-xyz(needed for evaluation).jointstype=vertices: Generate human body meshes (needed for rendering).
Click to expand
To evaluate TEMOS on the metrics defined in the paper, you must generate motions first (see above), and then run:
python evaluate.py folder=FOLDER [OPTIONS]This will compute and store the metrics in the file FOLDER/samples/metrics_SPLIT in a yaml format.
Same parameters as in sample.py, it will choose the right directories for you. In the case of evaluating with number_of_samples>1, the script will compute two metrics metrics_gtest_multi_avg (the average of single metrics) and metrics_gtest_multi_best (chosing the best output for each motion). Please check the paper for more details.
Currently, evaluation is only implemented on skeletons with MMM format. You must therefore use jointstype=mmm during sampling.
Please use this command line to download the motions generated from previous work:
bash prepare/download_previous_works.shThen, to evaluate a method, you can do for example:
python evaluate.py folder=previous_work/ghoshor change "ghosh" with "jl2p" or "lin".
To give an overview on how to extract their motions:
- Generate motions with their code (it is still in the rifke feature space)
- Save them in xyz format (I "hack" their render script, to save them in xyz npy format instead of rendering)
- Load them into the evaluation code, as shown above.
Click to expand
To get the visuals of the paper, I use Blender 2.93. The setup is not trivial (installation + running), I do my best to explain the process but don't hesitate to tell me if you have a problem.
The goal is to be able to install blender so that it can be used with python scripts (so we can use ``import bpy''). There seem to be many different ways to do this, I will explain the one I use and understand (feel free to use other methods or suggest an easier way). The installation of Blender will be done as a standalone package. To use my scripts, we will run blender in the background, and the python executable in blender will run the script.
In any case, after the installation, please do step 5/6. to install the dependencies in the python environment.
- Please follow the instructions to install blender 2.93 on your operating system. Please install exactly this version.
- Locate the blender executable if it is not in your path. For the following commands, please replace
blenderwith the path to your executable (or create a symbolic link or use an alias).- On Linux, it could be in
/usr/bin/blenderor/snap/bin/blender(already in your path). - On macOS, it could be in
/Applications/Blender.app/Contents/MacOS/Blender(not in your path)
- On Linux, it could be in
- Check that the correct version is installed:
blender --background --versionshould return "Blender 2.93.X".blender --background --python-expr "import sys; print('\nThe version of python is '+sys.version.split(' ')[0])"should return "3.9.X".
- Locate the python installation used by blender the following line. I will refer to this path as
/path/to/blender/python.
blender --background --python-expr "import sys; import os; print('\nThe path to the installation of python of blender can be:'); print('\n'.join(['- '+x.replace('/lib/python', '/bin/python') for x in sys.path if 'python' in (file:=os.path.split(x)[-1]) and not file.endswith('.zip')]))"- Install pip
/path/to/blender/python -m ensurepip --upgrade- Install these packages in the python environnement of blender:
/path/to/blender/python -m pip install --user numpy
/path/to/blender/python -m pip install --user matplotlib
/path/to/blender/python -m pip install --user hydra-core --upgrade
/path/to/blender/python -m pip install --user hydra_colorlog --upgrade
/path/to/blender/python -m pip install --user moviepy
/path/to/blender/python -m pip install --user shortuuidNow that blender is installed, if we want to run the script script.py with the blender API (the bpy module), we can use:
blender --background --python script.pyIf you need to add additional arguments, this will probably fail (as blender will interpret the arguments). Please use the double dash -- to tell blender to ignore the rest of the command.
I then only parse the last part of the command (check temos/launch/blender.py if you are interested).
To render only one motion, please use this command line:
blender --background --python render.py -- npy=PATH_TO_DATA.npy [OPTIONS]Please use this command line to render all the npy inside a specific folder.
blender --background --python render.py -- folder=FOLDER_WITH_NPYS [OPTIONS]Don't forget to generate the data with the option jointstype=vertices before.
The renderer will automatically detect whether the motion is a sequence of joints or meshes.
downsample=true: Render only 1 frame every 8 frames, to speed up rendering (by default)canonicalize=true: Make sure the first pose is oriented canonically (by translating and rotating the entire sequence) (by default)mode=XXX: Choose the rendering mode (default ismode=sequence)video: Render all the frames and generate a video (as in the supplementary video)sequence: Render a single frame, withnum=8bodies (sampled equally, as in the figures of the paper)frame: Render a single frame, at a specific point in time (exact_frame=0.5, generates the frame at about 50% of the video)
quality=false: Render to a higher resolution and denoise the output (default to false to speed up))
This code is distributed under an MIT LICENSE.
Note that our code depends on other libraries, including SMPL, SMPL-X, PyTorch3D, Hugging Face, Hydra, and uses datasets which each have their own respective licenses that must also be followed.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent