saving the latest model (epoch 3, total_steps 375825)
saved net: ./checkpoints/experiment_1/net_epoch_3_id_G.pth
saved net: ./checkpoints/experiment_1/net_epoch_3_id_D.pth
[27/08/2018 18:08:54] (T, epoch: 3, it: 121/7516, t/smpl: 0.017s) g_fake:0.000 g_cond:0.000 g_mskd_fake:-0.242 g_mskd_cond:3.193 g_cyc:1.215 g_rgb:0.000 g_rgb_un:0.000 g_rgb_s:0.000 g_m1:0.067 g_m2:0.067 g_m1_s:0.134 g_m2_s:0.129 g_idt:0.000 d_real:-0.856 d_cond:2.661 d_fake:0.491 d_gp:0.107
Traceback (most recent call last):
File "train.py", line 141, in
Train()
File "train.py", line 27, in init
self._train()
File "train.py", line 40, in _train
self._train_epoch(i_epoch)
File "train.py", line 59, in _train_epoch
for i_train_batch, train_batch in enumerate(self._dataset_train):
File "/appl/opt/python/mlpython-2.7.10/site-packages/torch/utils/data/dataloader.py", line 210, in next
return self._process_next_batch(batch)
File "/appl/opt/python/mlpython-2.7.10/site-packages/torch/utils/data/dataloader.py", line 230, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
RuntimeError: Traceback (most recent call last):
File "/appl/opt/python/mlpython-2.7.10/site-packages/torch/utils/data/dataloader.py", line 42, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/appl/opt/python/mlpython-2.7.10/site-packages/torch/utils/data/dataloader.py", line 116, in default_collate
return {key: default_collate([d[key] for d in batch]) for key in batch[0]}
File "/appl/opt/python/mlpython-2.7.10/site-packages/torch/utils/data/dataloader.py", line 116, in
return {key: default_collate([d[key] for d in batch]) for key in batch[0]}
File "/appl/opt/python/mlpython-2.7.10/site-packages/torch/utils/data/dataloader.py", line 105, in default_collate
return torch.stack([torch.from_numpy(b) for b in batch], 0)
RuntimeError: the given numpy array has zero-sized dimensions. Zero-sized dimensions are not supported in PyTorch

Hello, I get an error like this. Can you give me some suggestions? Maybe the pytorch version is not right?

Hello, great approach!

It was not too clear for me how do you handle the labels of the dataset. I do have access to the EmotionNet dataset, yet with discrete labels. Just to clarify what you state in the footnote page 3 ("this dataset was re-annotated...."), are you using different labels than those provided by the dataset? Is this re-annotation publicly available?

I have several other questions that have nothing to do with the code, but with the analysis and experiment setup, so I will reach to you by email. I would appreciate your feedback.

Thanks :D

I have trained the model but I can see that desired condition in test.py file is expression = np.random.uniform(0, 1, opt.cond_nc)

I am running it as python test.py --input_path ds/imgcelebaprocessed/000009.jpg --name experiment_128v2

How can I modify expression of generated output? Have you used different test file which is not shared?

How to train and test on the RaFD dataset?

Also async is a reserved word in Python 3.7+... pytorch/pytorch#4999

flake8 testing of https://github.com/albertpumarola/GANimation on Python 3.7.0

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./networks/networks.py:20:38: E999 SyntaxError: invalid syntax
        print "Network %s was created" % network_name
                                     ^
./models/ganimation.py:106:84: E999 SyntaxError: invalid syntax
            self._input_real_img = self._input_real_img.cuda(self._gpu_ids[0], async=True)
                                                                                   ^
./models/models.py:91:40: E999 SyntaxError: invalid syntax
            print 'loaded optimizer: %s' % load_path
                                       ^
./data/dataset_aus.py:37:63: E999 SyntaxError: invalid syntax
                print 'error reading image %s, skipping sample' % sample_id
                                                              ^
4     E999 SyntaxError: invalid syntax
4

------------ Options -------------
aus_file: aus_openface.pkl
batch_size: 4
checkpoints_dir: ./checkpoints
cond_nc: 17
data_dir: None
dataset_mode: aus
do_saturate_mask: False
gpu_ids: [0]
image_size: 128
images_folder: imgs
input_path: test
is_train: False
load_epoch: 78
model: ganimation
n_threads_test: 1
name: experiment_1
output_dir: ./output
serial_batches: False
test_ids_file: test_ids.csv
train_ids_file: train_ids.csv
-------------- End ----------------
./checkpoints/experiment_1
Network generator_wasserstein_gan was created
Network discriminator_wasserstein_gan was created
loaded net: ./checkpoints/experiment_1/net_epoch_78_id_G.pth
Model GANimation was created
Traceback (most recent call last):
File "test.py", line 74, in
main()
File "test.py", line 69, in main
morph.morph_file(image_path, expression)
File "test.py", line 28, in morph_file
morphed_img = self._img_morph(img, expresion)
File "test.py", line 33, in _img_morph
bbs = face_recognition.face_locations(img)
File "/usr/local/lib/python2.7/dist-packages/face_recognition/api.py", line 118, in face_locations
return [_trim_css_to_bounds(_rect_to_css(face), img.shape) for face in _raw_face_locations(img, number_of_times_to_upsample, model)]
File "/usr/local/lib/python2.7/dist-packages/face_recognition/api.py", line 102, in _raw_face_locations
return face_detector(img, number_of_times_to_upsample)
TypeError: call(): incompatible function arguments. The following argument types are supported:
1. (self: dlib.fhog_object_detector, image: array, upsample_num_times: int=0L) -> dlib.rectangles

Invoked with: <dlib.fhog_object_detector object at 0x7fb1aa88e960>, None, 1
when i use the command !python test.py --input_path test/myimage.jpg ,it worked ,but the result is very bad

We use CK+ dataset, but when I use test dataset image to test ,the result doesn’t expect,The following is the process of my train. Can you give me some suggestions?

1、Train GANimation

2、The GPU working status

3、The test result

4、Train loss

5、The train image result

Thank you for your novel work! I would like to try this on my laptop. It would be very nice if you make your trained model(.pth) available. Thanks in advance.

Just like i said above , i'd like to know how to overcome or release this problem, thanks~

Hi, everyone!

I came across this error when running the train process for several epochs. I know that it is a PyTorch error, however I want to know is there some problems with my dataset that raise this error, and whether this error can be resolved by this project.

I tried to track the problem, this was caused by "src->nDimention <= 0" in PyTorch, but I don't know what was wrong with my data.

Any reply will be appreciated!

I use a single GTX 1080TI which is the same as the author,but it raise cuda out of memeory. why?
(dl) root@63c12776c653:/data/GANimataion# bash launch/run_train.sh
------------ Options -------------
D_adam_b1: 0.5
D_adam_b2: 0.999
G_adam_b1: 0.5
G_adam_b2: 0.999
aus_file: aus_openface.pkl
batch_size: 25
checkpoints_dir: ./checkpoints
cond_nc: 17
data_dir: /input
dataset_mode: aus
display_freq_s: 300
do_saturate_mask: False
gpu_ids: [0]
image_size: 128
images_folder: imgs
is_train: True
lambda_D_cond: 4000
lambda_D_gp: 10
lambda_D_prob: 1
lambda_cyc: 10
lambda_mask: 0.1
lambda_mask_smooth: 1e-05
load_epoch: 0
lr_D: 0.0001
lr_G: 0.0001
model: ganimation
n_threads_test: 1
n_threads_train: 1
name: experiment_ck
nepochs_decay: 10
nepochs_no_decay: 20
num_iters_validate: 1
poses_g_sigma: 0.06
print_freq_s: 60
save_latest_freq_s: 3600
serial_batches: False
test_ids_file: test_ids.csv
train_G_every_n_iterations: 5
train_ids_file: train_ids.csv
-------------- End ----------------
./checkpoints/experiment_ck
Dataset AusDataset was created
Dataset AusDataset was created
#train images = 8791
#test images = 4
Network generator_wasserstein_gan was created
Network discriminator_wasserstein_gan was created
Model GANimation was created
Traceback (most recent call last):
File "train.py", line 141, in
Train()
File "train.py", line 27, in init
self._train()
File "train.py", line 40, in _train
self._train_epoch(i_epoch)
File "train.py", line 69, in _train_epoch
self._model.optimize_parameters(keep_data_for_visuals=do_visuals, train_generator=train_generator)
File "/data/GANimataion/models/ganimation.py", line 207, in optimize_parameters
loss_D, fake_imgs_masked = self._forward_D()
File "/data/GANimataion/models/ganimation.py", line 273, in _forward_D
fake_imgs, fake_img_mask = self._G.forward(self._real_img, self._desired_cond)
File "/data/GANimataion/networks/generator_wasserstein_gan.py", line 53, in forward
features = self.main(x)
File "/usr/local/miniconda3/envs/dl/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/usr/local/miniconda3/envs/dl/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/usr/local/miniconda3/envs/dl/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/data/GANimataion/networks/generator_wasserstein_gan.py", line 68, in forward
return x + self.main(x)
File "/usr/local/miniconda3/envs/dl/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/usr/local/miniconda3/envs/dl/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/usr/local/miniconda3/envs/dl/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/usr/local/miniconda3/envs/dl/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
self.padding, self.dilation, self.groups)
RuntimeError: CUDA error: out of memory
(dl) root@63c12776c653:/data/GANimataion# bash launch/run_train.sh
------------ Options -------------
D_adam_b1: 0.5
D_adam_b2: 0.999
G_adam_b1: 0.5
G_adam_b2: 0.999
aus_file: aus_openface.pkl
batch_size: 25
checkpoints_dir: ./checkpoints
cond_nc: 17
data_dir: /input
dataset_mode: aus
display_freq_s: 300
do_saturate_mask: False
gpu_ids: [0]
image_size: 128
images_folder: imgs
is_train: True
lambda_D_cond: 4000
lambda_D_gp: 10
lambda_D_prob: 1
lambda_cyc: 10
lambda_mask: 0.1
lambda_mask_smooth: 1e-05
load_epoch: 0
lr_D: 0.0001
lr_G: 0.0001
model: ganimation
n_threads_test: 1
n_threads_train: 1
name: experiment_ck
nepochs_decay: 10
nepochs_no_decay: 20
num_iters_validate: 1
poses_g_sigma: 0.06
print_freq_s: 60
save_latest_freq_s: 3600
serial_batches: False
test_ids_file: test_ids.csv
train_G_every_n_iterations: 5
train_ids_file: train_ids.csv
-------------- End ----------------
./checkpoints/experiment_ck
Dataset AusDataset was created
Dataset AusDataset was created
#train images = 8791
#test images = 4
Network generator_wasserstein_gan was created
Network discriminator_wasserstein_gan was created
Model GANimation was created
Traceback (most recent call last):
File "train.py", line 141, in
Train()
File "train.py", line 27, in init
self._train()
File "train.py", line 40, in _train
self._train_epoch(i_epoch)
File "train.py", line 69, in _train_epoch
self._model.optimize_parameters(keep_data_for_visuals=do_visuals, train_generator=train_generator)
File "/data/GANimataion/models/ganimation.py", line 207, in optimize_parameters
loss_D, fake_imgs_masked = self._forward_D()
File "/data/GANimataion/models/ganimation.py", line 273, in _forward_D
fake_imgs, fake_img_mask = self._G.forward(self._real_img, self._desired_cond)
File "/data/GANimataion/networks/generator_wasserstein_gan.py", line 53, in forward
features = self.main(x)
File "/usr/local/miniconda3/envs/dl/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/usr/local/miniconda3/envs/dl/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/usr/local/miniconda3/envs/dl/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/data/GANimataion/networks/generator_wasserstein_gan.py", line 68, in forward
return x + self.main(x)
File "/usr/local/miniconda3/envs/dl/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/usr/local/miniconda3/envs/dl/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/usr/local/miniconda3/envs/dl/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/usr/local/miniconda3/envs/dl/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
self.padding, self.dilation, self.groups)
RuntimeError: CUDA error: out of memory

Hello,

I am trying to recreate the experiment. But when I run the pipeline, I see a lot of skipped samples. I traced the error and found that it is because the image loader has the 3 channel checks. So it seems there are a lot of grayscale images in the data. So I was wondering if you have your train/test split filenames available somewhere. It would be nice to be able to use those and download those images, to make sure that I get proper 3 channel images and have the training run properly.

I've trained a model and experiencing problems to path an image through it. It says:

------------ Options -------------
aus_file: aus_openface.pkl
batch_size: 4
checkpoints_dir: ./checkpoints
cond_nc: 17
data_dir: None
dataset_mode: aus
do_saturate_mask: False
gpu_ids: [0]
image_size: 128
images_folder: imgs
input_path: ./sample_dataset/imgs/N_0000001507_00202.jpg
is_train: False
load_epoch: 30
model: ganimation
n_threads_test: 1
name: experiment_1
output_dir: ./output
serial_batches: False
test_ids_file: test_ids.csv
train_ids_file: train_ids.csv
-------------- End ----------------
./checkpoints/experiment_1
Network generator_wasserstein_gan was created
Network discriminator_wasserstein_gan was created
Traceback (most recent call last):
File "test.py", line 74, in
main()
File "test.py", line 65, in main
morph = MorphFacesInTheWild(opt)
File "test.py", line 19, in init
self._model = ModelsFactory.get_by_name(self._opt.model, self._opt)
File "/home/WX466836/docker_mount/GANimation/models/models.py", line 15, in get_by_name
model = GANimation(*args, **kwargs)
File "/home/WX466836/docker_mount/GANimation/models/ganimation.py", line 26, in init
self.load()
File "/home/WX466836/docker_mount/GANimation/models/ganimation.py", line 376, in load
self._load_network(self._G, 'G', load_epoch)
File "/home/WX466836/docker_mount/GANimation/models/models.py", line 105, in _load_network
network.load_state_dict(torch.load(load_path))
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 719, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for Generator:
Missing key(s) in state_dict: "main.0.weight", "main.1.bias", "main.1.weight", "main.3.weight", "main.4.bias", "main.4.weight", "main.6.weight", "main.7.bias", "main.7.weight", "main.9.main.0.weight", "main.9.main.1.bias", "main.9.main.1.weight", "main.9.main.3.weight", "main.9.main.4.bias", "main.9.main.4.weight", "main.10.main.0.weight", "main.10.main.1.bias", "main.10.main.1.weight", "main.10.main.3.weight", "main.10.main.4.bias", "main.10.main.4.weight", "main.11.main.0.weight", "main.11.main.1.bias", "main.11.main.1.weight", "main.11.main.3.weight", "main.11.main.4.bias", "main.11.main.4.weight", "main.12.main.0.weight", "main.12.main.1.bias", "main.12.main.1.weight", "main.12.main.3.weight", "main.12.main.4.bias", "main.12.main.4.weight", "main.13.main.0.weight", "main.13.main.1.bias", "main.13.main.1.weight", "main.13.main.3.weight", "main.13.main.4.bias", "main.13.main.4.weight", "main.14.main.0.weight", "main.14.main.1.bias", "main.14.main.1.weight", "main.14.main.3.weight", "main.14.main.4.bias", "main.14.main.4.weight", "main.15.weight", "main.16.bias", "main.16.weight", "main.18.weight", "main.19.bias", "main.19.weight", "img_reg.0.weight", "attetion_reg.0.weight".
Unexpected key(s) in state_dict: "module.main.0.weight", "module.main.1.weight", "module.main.1.bias", "module.main.3.weight", "module.main.4.weight", "module.main.4.bias", "module.main.6.weight", "module.main.7.weight", "module.main.7.bias", "module.main.9.main.0.weight", "module.main.9.main.1.weight", "module.main.9.main.1.bias", "module.main.9.main.3.weight", "module.main.9.main.4.weight", "module.main.9.main.4.bias", "module.main.10.main.0.weight", "module.main.10.main.1.weight", "module.main.10.main.1.bias", "module.main.10.main.3.weight", "module.main.10.main.4.weight", "module.main.10.main.4.bias", "module.main.11.main.0.weight", "module.main.11.main.1.weight", "module.main.11.main.1.bias", "module.main.11.main.3.weight", "module.main.11.main.4.weight", "module.main.11.main.4.bias", "module.main.12.main.0.weight", "module.main.12.main.1.weight", "module.main.12.main.1.bias", "module.main.12.main.3.weight", "module.main.12.main.4.weight", "module.main.12.main.4.bias", "module.main.13.main.0.weight", "module.main.13.main.1.weight", "module.main.13.main.1.bias", "module.main.13.main.3.weight", "module.main.13.main.4.weight", "module.main.13.main.4.bias", "module.main.14.main.0.weight", "module.main.14.main.1.weight", "module.main.14.main.1.bias", "module.main.14.main.3.weight", "module.main.14.main.4.weight", "module.main.14.main.4.bias", "module.main.15.weight", "module.main.16.weight", "module.main.16.bias", "module.main.18.weight", "module.main.19.weight", "module.main.19.bias", "module.img_reg.0.weight", "module.attetion_reg.0.weight".

Do you have any advice to make it work?

Thank you for sharing your code. But I meet an error when I run test.py. The error is like this:
load_path), 'Weights file not found. Have you trained a model!? We are not providing one' % load_path
And I just see that the folder have no .pth files. Would you please share me a trained model. Thank you!

could you shared your image dataset?

dear friend:
thank you for sharing your code,however when i "import torchvision.transforms as transforms" and "from utils import face_utils",there was a "Segmentation fault (core dumped)".when i run only one of them it 's fine.i seems that there is a conflict .Would you mind take some suggest.

Hi, my computer don't have gpu, so is cuda must be installed ?

hello,when I run the code( train.py ) with your sample_dataset,I run this order
"python train.py
--data_dir sample_dataset/
--name experiment_1
--batch_size 2 \ "

I get this error message:

Traceback (most recent call last):
File "/home/xsky/projects/GANimation-master/train.py", line 142, in
Train()
File "/home/xsky/projects/GANimation-master/train.py", line 14, in init
data_loader_test = CustomDatasetDataLoader(self._opt, is_for_train=False)
File "/home/xsky/projects/GANimation-master/data/custom_dataset_data_loader.py", line 10, in init
self._create_dataset()
File "/home/xsky/projects/GANimation-master/data/custom_dataset_data_loader.py", line 13, in _create_dataset
self._dataset = DatasetFactory.get_by_name(self._opt.dataset_mode, self._opt, self._is_for_train)
File "/home/xsky/projects/GANimation-master/data/dataset.py", line 16, in get_by_name
dataset = AusDataset(opt, is_for_train)
File "/home/xsky/projects/GANimation-master/data/dataset_aus.py", line 17, in init
self._read_dataset_paths()
File "/home/xsky/projects/GANimation-master/data/dataset_aus.py", line 68, in _read_dataset_paths
self._ids = self._read_ids(use_ids_filepath)
File "/home/xsky/projects/GANimation-master/data/dataset_aus.py", line 96, in _read_ids
return [id[:-4] for id in ids]
TypeError: iteration over a 0-d array

I don't know where the problem is，so, would you please answer it?Thank you very much!

Hi, thank you very much for sharing your training and testing codes.
Could you please also share the training data?
I have applied for the training data on the website but still get no reply.
Thank you very much.

Hi,
I run OpenFace on the imgs under sample_dataset folder, and found the AU results are different from the AU labels in aus_openface.pkl. For example, for N_0000000356_00190.jpg, the labels provided by you are:
[2.86, 2.27, 1.45, 1.1 , 0. , 0.65, 0.05, 0. , 0.75, 1.65, 0.6 , 0. , 1.86, 0. , 0.62, 0.25, 0. ],
while the detected results from OpenFace are:
[0.48, 0. , 0. , 1.47, 0.23, 0.06, 0. , 0.72, 0.48, 1.36, 0.3 , 0. , 1.76, 0. , 0.44, 0.78, 0. ].

I used the command: ./bin/FaceLandmarkImg -fdir GANimation/sample_dataset/imgs -out_dir GANimation/sample_dataset/features -aus

Is it correct?
Thanks!

Hello!
Is there a way to launch it using several gpus?
How long does it take using 1 gpu to reach approximately your performance?
Thank you in advance

I have an error when I launch training, it says:

Network generator_wasserstein_gan was created
Network discriminator_wasserstein_gan was created
Model GANimation was created
Traceback (most recent call last):
File "train.py", line 141, in
Train()
File "train.py", line 27, in init
self._train()
File "train.py", line 40, in _train
self._train_epoch(i_epoch)
File "train.py", line 59, in _train_epoch
for i_train_batch, train_batch in enumerate(self._dataset_train):
File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py", line 336, in next
return self._process_next_batch(batch)
File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py", line 357, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
RuntimeError: Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py", line 106, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py", line 184, in default_collate
return {key: default_collate([d[key] for d in batch]) for key in batch[0]}
File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py", line 184, in
return {key: default_collate([d[key] for d in batch]) for key in batch[0]}
File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py", line 164, in default_collate
return torch.stack(batch, 0, out=out)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 558 and 357 in dimension 2 at /pytorch/aten/src/TH/generic/THTensorMath.cpp:3616

These two numbers (558 and 357) are different with each launching. Do you have any ideas how to fix it?

/usr/bin/python3.6 /home/bugtan/code/GANimation/train.py --data_dir /home/bugtan/code/GANimation/ck_dataset --name experiment_ck --batch_size 2
------------ Options -------------
D_adam_b1: 0.5
D_adam_b2: 0.999
G_adam_b1: 0.5
G_adam_b2: 0.999
aus_file: aus_openface.pkl
batch_size: 2
checkpoints_dir: ./checkpoints
cond_nc: 17
data_dir: /home/bugtan/code/GANimation/ck_dataset
dataset_mode: aus
display_freq_s: 300
do_saturate_mask: False
gpu_ids: [0]
image_size: 128
images_folder: imgs
is_train: True
lambda_D_cond: 4000
lambda_D_gp: 10
lambda_D_prob: 1
lambda_cyc: 10
lambda_mask: 0.1
lambda_mask_smooth: 1e-05
load_epoch: 0
lr_D: 0.0001
lr_G: 0.0001
model: ganimation
n_threads_test: 1
n_threads_train: 4
name: experiment_ck
nepochs_decay: 10
nepochs_no_decay: 20
num_iters_validate: 1
poses_g_sigma: 0.06
print_freq_s: 60
save_latest_freq_s: 3600
serial_batches: False
test_ids_file: test_ids.csv
train_G_every_n_iterations: 5
train_ids_file: train_ids.csv
-------------- End ----------------
./checkpoints/experiment_ck
Dataset AusDataset was created
Dataset AusDataset was created
#train images = 8791
#test images = 4
Network generator_wasserstein_gan was created
Network discriminator_wasserstein_gan was created
Model GANimation was created
Exception ignored in: <bound method _DataLoaderIter.del of <torch.utils.data.dataloader._DataLoaderIter object at 0x7f8e651aea90>>
Traceback (most recent call last):
File "/home/bugtan/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 399, in del
self._shutdown_workers()
File "/home/bugtan/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 378, in _shutdown_workers
self.worker_result_queue.get()
File "/usr/lib/python3.6/multiprocessing/queues.py", line 337, in get
return _ForkingPickler.loads(res)
File "/home/bugtan/.local/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 151, in rebuild_storage_fd
fd = df.detach()
File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
return reduction.recv_handle(conn)
File "/usr/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
return recvfds(s, 1)[0]
File "/usr/lib/python3.6/multiprocessing/reduction.py", line 153, in recvfds
msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size))
ConnectionResetError: [Errno 104] Connection reset by peer
Traceback (most recent call last):
File "/home/bugtan/code/GANimation/train.py", line 141, in
Train()
File "/home/bugtan/code/GANimation/train.py", line 27, in init
self._train()
File "/home/bugtan/code/GANimation/train.py", line 40, in _train
self._train_epoch(i_epoch)
File "/home/bugtan/code/GANimation/train.py", line 69, in _train_epoch
self._model.optimize_parameters(keep_data_for_visuals=do_visuals, train_generator=train_generator)
File "/home/bugtan/code/GANimation/models/ganimation.py", line 207, in optimize_parameters
loss_D, fake_imgs_masked = self._forward_D()
File "/home/bugtan/code/GANimation/models/ganimation.py", line 280, in _forward_D
self._loss_d_cond = self._criterion_D_cond(d_real_img_cond, self._real_cond) / self._B * self._opt.lambda_D_cond
File "/home/bugtan/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/bugtan/.local/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 421, in forward
return F.mse_loss(input, target, reduction=self.reduction)
File "/home/bugtan/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 1716, in mse_loss
return _pointwise_loss(lambda a, b: (a - b) ** 2, torch._C._nn.mse_loss, input, target, reduction)
File "/home/bugtan/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 1674, in _pointwise_loss
return lambd_optimized(input, target, reduction)
RuntimeError: input and target shapes do not match: input [2 x 17 x 6 x 9], target [2 x 17] at /pytorch/aten/src/THCUNN/generic/MSECriterion.cu:12

Anyone can help me solve this question?

Congratulations! You have done a great job.
But I have a question about the Attention-based Generator. This paper said that the generator outputs two masks, a color mask C and attention mask A. And you gave the following formula:

So, I'm wondering how did you come up with this formula? I mean, how do you explain this formula?
Looking forward to your apply.

Hi albert,

I found the identity loss in your paper but could not see the lambda parameter and the loss settings in your code. Am I missing some key notes?

Thanks in advance.

Hi @albertpumarola ,

Thank you for sharing your great work!

I am recently working on migrating the idea of your work to other tasks related to face attribute editing. However, I am struggling with searching for hyper-parameters to obtain good results as there are too many to tune. Would you kindly share some experiences on tuning your model (e.g. which terms is the most important one? what is the key point?).

To get some knowledge of reasonable losses, I will appreciate it if you could share some log information of training your final model. I just would like to know the scales and relationships between those loss terms.

Thank you in advance!

@albertpumarola
Hi, I make my own database and want to train the model.
I use this to train:python train.py --data_dir /media/xirui/XIRUI/database1 --name try3 --batch_size 25
But during it , I found this problem:

------------ Options -------------
D_adam_b1: 0.5
D_adam_b2: 0.999
G_adam_b1: 0.5
G_adam_b2: 0.999
aus_file: aus_openface.pkl
batch_size: 25
checkpoints_dir: ./checkpoints
cond_nc: 17
data_dir: /media/xirui/XIRUI/database1
dataset_mode: aus
display_freq_s: 300
do_saturate_mask: False
gpu_ids: [0]
image_size: 128
images_folder: imgs
is_train: True
lambda_D_cond: 4000
lambda_D_gp: 10
lambda_D_prob: 1
lambda_cyc: 10
lambda_mask: 0.1
lambda_mask_smooth: 1e-05
load_epoch: 0
lr_D: 0.0001
lr_G: 0.0001
model: ganimation
n_threads_test: 1
n_threads_train: 4
name: try3
nepochs_decay: 10
nepochs_no_decay: 20
num_iters_validate: 1
poses_g_sigma: 0.06
print_freq_s: 60
save_latest_freq_s: 3600
serial_batches: False
test_ids_file: test_ids.csv
train_G_every_n_iterations: 5
train_ids_file: train_ids.csv
-------------- End ----------------
./checkpoints/try3
Dataset AusDataset was created
Dataset AusDataset was created
#train images = 478
#test images = 479
Network generator_wasserstein_gan was created
Network discriminator_wasserstein_gan was created
Model GANimation was created
/home/xirui/XIr/GANimation-master/models/ganimation.py:320: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('g_mskd_fake', self._loss_g_masked_fake.data[0]),
/home/xirui/XIr/GANimation-master/models/ganimation.py:321: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('g_mskd_cond', self._loss_g_masked_cond.data[0]),
/home/xirui/XIr/GANimation-master/models/ganimation.py:322: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('g_cyc', self._loss_g_cyc.data[0]),
/home/xirui/XIr/GANimation-master/models/ganimation.py:326: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('g_m1', self._loss_g_mask_1.data[0]),
/home/xirui/XIr/GANimation-master/models/ganimation.py:327: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('g_m2', self._loss_g_mask_2.data[0]),
/home/xirui/XIr/GANimation-master/models/ganimation.py:328: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('g_m1_s', self._loss_g_mask_1_smooth.data[0]),
/home/xirui/XIr/GANimation-master/models/ganimation.py:329: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('g_m2_s', self._loss_g_mask_2_smooth.data[0]),
/home/xirui/XIr/GANimation-master/models/ganimation.py:331: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('d_real', self._loss_d_real.data[0]),
/home/xirui/XIr/GANimation-master/models/ganimation.py:332: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('d_cond', self._loss_d_cond.data[0]),
/home/xirui/XIr/GANimation-master/models/ganimation.py:333: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('d_fake', self._loss_d_fake.data[0]),
/home/xirui/XIr/GANimation-master/models/ganimation.py:334: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('d_gp', self._loss_d_gp.data[0])])
[30/09/2018 17:14:19] (Tv, epoch: 1, it: 0/19, t/smpl: 0.039s) g_fake:0.000 g_cond:0.000 g_mskd_fake:0.025 g_mskd_cond:1.769 g_cyc:4.888 g_rgb:0.000 g_rgb_un:0.000 g_rgb_s:0.000 g_m1:0.046 g_m2:0.046 g_m1_s:0.742 g_m2_s:0.753 g_idt:0.000 d_real:0.013 d_cond:1.705 d_fake:-0.009 d_gp:7.552
Traceback (most recent call last):
File "train.py", line 141, in
Train()
File "train.py", line 27, in init
self._train()
File "train.py", line 40, in _train
self._train_epoch(i_epoch)
File "train.py", line 82, in _train_epoch
self._display_visualizer_train(self._total_steps)
File "train.py", line 98, in _display_visualizer_train
self._tb_visualizer.display_current_results(self._model.get_current_visuals(), total_steps, is_train=True)
File "/home/xirui/XIr/GANimation-master/utils/tb_visualizer.py", line 27, in display_current_results
self._writer.add_image(sum_name, image_numpy, it)
File "/usr/local/lib/python2.7/dist-packages/tensorboardX/writer.py", line 412, in add_image
self.file_writer.add_summary(image(tag, img_tensor), global_step, walltime)
File "/usr/local/lib/python2.7/dist-packages/tensorboardX/summary.py", line 205, in image
image = make_image(tensor, rescale=rescale)
File "/usr/local/lib/python2.7/dist-packages/tensorboardX/summary.py", line 243, in make_image
image = Image.fromarray(tensor)
File "/usr/local/lib/python2.7/dist-packages/PIL/Image.py", line 2463, in fromarray
raise TypeError("Cannot handle this data type")
TypeError: Cannot handle this data type

I've checked the pic in my database and they're both 3 channels and 128*128. And I've tried some database with different number and I found that when the number of pic is more than 48, this problem would occur, while the process of training was right when the number of pic is less than 48

Is there someone who can help me? Thanks!!

It seems to me that later pytorch will have some problem when training like poping runtime error "sizes must be non-negative". I was using the latest one which is 0.4.1 and trying to debug on 0.3.1 now. Just want to be sure.

Use environment:
Ubuntu16.04
cuda8.0 + cudnn6.0
tensorflow-gpu1.4.1 torch0.4.1

Problem：
I use CK+ dataset to train the Ganimation.But I kept getting the error :"RuntimeError:cuda error : out of memory", and when I changed the batch_size to be larger than the training set, the error stopped.

Did anyone make the same mistake ?

Any idea why this error occurs? I am using 224x224 celeba aligned database.

Traceback (most recent call last):
  File "train.py", line 141, in <module>
    Train()
  File "train.py", line 27, in __init__
    self._train()
  File "train.py", line 40, in _train
    self._train_epoch(i_epoch)
  File "train.py", line 69, in _train_epoch
    self._model.optimize_parameters(keep_data_for_visuals=do_visuals, train_generator=train_generator)
  File "/home/aidas/GANimation/models/ganimation.py", line 208, in optimize_parameters
    loss_D, fake_imgs_masked = self._forward_D()
  File "/home/aidas/GANimation/models/ganimation.py", line 281, in _forward_D
    self._loss_d_cond = self._criterion_D_cond(d_real_img_cond, self._real_cond) / self._B * self._opt.lambda_D_cond
  File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/modules/loss.py", line 421, in forward
    return F.mse_loss(input, target, reduction=self.reduction)
  File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/functional.py", line 1716, in mse_loss
    return _pointwise_loss(lambda a, b: (a - b) ** 2, torch._C._nn.mse_loss, input, target, reduction)
  File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/functional.py", line 1674, in _pointwise_loss
    return lambd_optimized(input, target, reduction)
RuntimeError: input and target shapes do not match: input [25 x 17 x 2 x 2], target [25 x 17] at /pytorch/aten/src/THCUNN/generic/MSECriterion.cu:12

Hi,

Thanks for sharing the codes!
I have a small question:
why do you divide the AU intensity value by 5 instead of using the original value?
return self._conds[id]/5.0

Thanks!

python3 train.py --data_dir sample_dataset --name experiment_1 --batch_size 25
------------ Options -------------
D_adam_b1: 0.5
D_adam_b2: 0.999
G_adam_b1: 0.5
G_adam_b2: 0.999
aus_file: aus_openface.pkl
batch_size: 25
checkpoints_dir: ./checkpoints
cond_nc: 17
data_dir: sample_dataset
dataset_mode: aus
display_freq_s: 300
do_saturate_mask: False
gpu_ids: [0]
image_size: 128
images_folder: imgs
is_train: True
lambda_D_cond: 4000
lambda_D_gp: 10
lambda_D_prob: 1
lambda_cyc: 10
lambda_mask: 0.1
lambda_mask_smooth: 1e-05
load_epoch: 0
lr_D: 0.0001
lr_G: 0.0001
model: ganimation
n_threads_test: 1
n_threads_train: 4
name: experiment_1
nepochs_decay: 10
nepochs_no_decay: 20
num_iters_validate: 1
poses_g_sigma: 0.06
print_freq_s: 60
save_latest_freq_s: 3600
serial_batches: False
test_ids_file: test_ids.csv
train_G_every_n_iterations: 5
train_ids_file: train_ids.csv
-------------- End ----------------
./checkpoints/experiment_1
Traceback (most recent call last):
File "train.py", line 141, in
Train()
File "train.py", line 13, in init
data_loader_train = CustomDatasetDataLoader(self._opt, is_for_train=True)
File "/root/GANimation/data/custom_dataset_data_loader.py", line 10, in init
self._create_dataset()
File "/root/GANimation/data/custom_dataset_data_loader.py", line 13, in _create_dataset
self._dataset = DatasetFactory.get_by_name(self._opt.dataset_mode, self._opt, self._is_for_train)
File "/root/GANimation/data/dataset.py", line 16, in get_by_name
dataset = AusDataset(opt, is_for_train)
File "/root/GANimation/data/dataset_aus.py", line 17, in init
self._read_dataset_paths()
File "/root/GANimation/data/dataset_aus.py", line 66, in _read_dataset_paths
self._conds = self._read_conds(conds_filepath)
File "/root/GANimation/data/dataset_aus.py", line 93, in _read_conds
return pickle.load(f)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1 in position 0: ordinal not in range(128)

First，I use the below command line to generation AU
E:\MagicProject\code\OpenFace_2.0.4_win_x64>FeatureExtraction.exe -f E:\Dataset_processing\20180821_GANimation_CK+\imgs\S005_001_00000001.png -out_dir E:\out

I generate CSV file of AU is from 679 column to 696，like below

Finally, I use this to train GANimation. But my train result is not expectation. I suspect that I generation is false.

Today ,I use FeatureExtraction.exe to generate AU ,but I choice different parameter，the command line and generation CSV of Au in below

1. generation AU from column 6 to column 25
   E:\MagicProject\code\OpenFace_2.0.4_win_x64>FeatureExtraction.exe -f E:\Dataset_processing\20180821_GANimation_CK+\imgs\S005_001_00000001.png -aus -out_dir E:\

E:\MagicProject\code\OpenFace_2.0.4_win_x64>FeatureExtraction.exe -f E:\Dataset_processing\20180821_GANimation_CK+\imgs\S005_001_00000001.png -aus -au_static -out_dir E:\out

But ,today I use 1 and 2 openface command line to generate AU ,the result is different.
(I used the same picture for the AU generation test)

Can you tell me how to use the right openface command parameter to generate CSV of AU.

1. Attention loss goes directly to zero after a few iterations, so the mask is zero, then the generator ignores the real image here https://github.com/albertpumarola/GANimation/blob/master/models/ganimation.py#L228. Regularizing the mask prevents the saturation to 1, but what about going to zero?

2. What is the purpose of https://github.com/albertpumarola/GANimation/blob/master/models/ganimation.py#L404-L405? As far as I can see saturate=True is never called.

Got below error. Dlibs api seems incompatiable...
~/GANimation$ python test.py --input_path sample_dataset/imgs/
------------ Options -------------
aus_file: aus_openface.pkl
batch_size: 4
checkpoints_dir: ./checkpoints
cond_nc: 17
data_dir: None
dataset_mode: aus
do_saturate_mask: False
gpu_ids: [0]
image_size: 128
images_folder: imgs
input_path: sample_dataset/imgs/
is_train: False
load_epoch: 0
model: ganimation
n_threads_test: 1
name: experiment_1
output_dir: ./output
serial_batches: False
test_ids_file: test_ids.csv
train_ids_file: train_ids.csv
-------------- End ----------------
./checkpoints/experiment_1
Network generator_wasserstein_gan was created
Network discriminator_wasserstein_gan was created
NOT!! loaded net: ./checkpoints/experiment_1/net_epoch_0_id_G.pth
Model GANimation was created
Traceback (most recent call last):
File "test.py", line 74, in
main()
File "test.py", line 69, in main
morph.morph_file(image_path, expression)
File "test.py", line 28, in morph_file
morphed_img = self._img_morph(img, expresion)
File "test.py", line 33, in _img_morph
bbs = face_recognition.face_locations(img)
File "/home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/face_recognition/api.py", line 118, in face_locations
return [_trim_css_to_bounds(_rect_to_css(face), img.shape) for face in _raw_face_locations(img, number_of_times_to_upsample, model)]
File "/home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/face_recognition/api.py", line 102, in _raw_face_locations
return face_detector(img, number_of_times_to_upsample)
TypeError: call(): incompatible function arguments. The following argument types are supported:
1. (self: dlib.fhog_object_detector, image: array, upsample_num_times: int=0L) -> dlib.rectangles

Invoked with: <dlib.fhog_object_detector object at 0x7f52b0efba40>, None, 1

python train.py --data_dir mydata --name experiment_1 --batch_size 25
------------ Options -------------
D_adam_b1: 0.5
D_adam_b2: 0.999
G_adam_b1: 0.5
G_adam_b2: 0.999
aus_file: aus_openface.pkl
batch_size: 25
checkpoints_dir: ./checkpoints
cond_nc: 17
data_dir: mydata
dataset_mode: aus
display_freq_s: 300
do_saturate_mask: False
gpu_ids: [0]
image_size: 128
images_folder: imgs
is_train: True
lambda_D_cond: 4000
lambda_D_gp: 10
lambda_D_prob: 1
lambda_cyc: 10
lambda_mask: 0.1
lambda_mask_smooth: 1e-05
load_epoch: 0
lr_D: 0.0001
lr_G: 0.0001
model: ganimation
n_threads_test: 1
n_threads_train: 4
name: experiment_1
nepochs_decay: 10
nepochs_no_decay: 20
num_iters_validate: 1
poses_g_sigma: 0.06
print_freq_s: 60
save_latest_freq_s: 3600
serial_batches: False
test_ids_file: test_ids.csv
train_G_every_n_iterations: 5
train_ids_file: train_ids.csv
-------------- End ----------------
./checkpoints/experiment_1
Dataset AusDataset was created
Dataset AusDataset was created
#train images = 91
#test images = 91
Network generator_wasserstein_gan was created
Network discriminator_wasserstein_gan was created
Model GANimation was created
Traceback (most recent call last):
File "train.py", line 141, in
Train()
File "train.py", line 27, in init
self._train()
File "train.py", line 40, in _train
self._train_epoch(i_epoch)
File "train.py", line 69, in _train_epoch
self._model.optimize_parameters(keep_data_for_visuals=do_visuals, train_generator=train_generator)
File "/data/GANimation/models/ganimation.py", line 207, in optimize_parameters
loss_D, fake_imgs_masked = self._forward_D()
File "/data/GANimation/models/ganimation.py", line 278, in _forward_D
d_real_img_prob, d_real_img_cond = self._D.forward(self._real_img)
File "/data/GANimation/networks/discriminator_wasserstein_gan.py", line 29, in forward
out_aux = self.conv2(h)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 477, in call
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/util.py", line 274, in _run_finalizers
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/conv.py", line 301, in forward
finalizer()
File "/usr/lib/python2.7/multiprocessing/util.py", line 207, in call
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python2.7/shutil.py", line 252, in rmtree
self.padding, self.dilation, self.groups)
RuntimeError: sizes must be non-negative

@albertpumarola Thank you for your contribution on this work !
When I run the test like this:
python test.py --input_path ./sample_dataset/imgs/N_0000001507_00202.jpg
It came :

And there is an image in output/ like this :

Is the model not public available?
Thank you! Looking forward to your reply.

Command "/usr/local/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-r3m4ignc/dlib/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-k1sm7d3d-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-r3m4ignc/dlib/

stops at 73% and show the error

Hi Albert, thanks for sharing your code.

In the paper, the fake image is generated by using the equation
.

While in the code, I find that the fake image is generated in a different way, which is
fake_imgs_masked = fake_img_mask * self._real_img + (1 - fake_img_mask) * fake_imgs,
refers to https://github.com/albertpumarola/GANimation/blob/master/models/ganimation.py#L228 and https://github.com/albertpumarola/GANimation/blob/master/models/ganimation.py#L275, so it should be

Would you please kindly clarify this? Thanks in advance.

I am creating a virtual environment in conda, running the program under python3.5.5
Then my matplotlib is 2.1.1, installed using conda install. When I run test.py, matplotlib often reports an error.
Error is as follows

File "/home/zp/software/miniconda3/envs/meng35/lib/python3.5/site-packages/matplotlib/collections.py", line 26, in
     From . import (_path, artist, cbook, cm, colors as mcolors, docstring,
ImportError: cannot import name _path

I want to ask which version of this code should be in python and the version of matplotlib

Hi,

You mentioned L2 loss for Eq 2, but there is L1 in the code (https://github.com/albertpumarola/GANimation/blob/master/models/ganimation.py#L244-L245). Is this ok?

@albertpumarola
Hi, I have a question about a way to incorporate the AU conditions into an input.

Your paper says that desired AU conditions (expression) are originally a N-length vector which has a normalized activation value between 0 and 1 respectively, and are concatenated to input as additional channels of input by expanding them into the same size as that of input image.
I think this expansion is for the input to be compatible with the first convolutional layer.
I wonder what is the most reasonable way to construct an input with non-image-like (scalar or vector) conditions.

A possible alternative way to do it is concatenating AU conditions as a vector with image unrolled into a vector and replace the first convolutional layer with a fully-connected layer.
How do you think of it?

wish to have the model included.and does it work if i don't install cuda

./checkpoints/experiment_1
Dataset AusDataset was created
Dataset AusDataset was created
#train images = 200000
#test images = 2055
Network generator_wasserstein_gan was created
Network discriminator_wasserstein_gan was created
Model GANimation was created
Traceback (most recent call last):
File "train.py", line 141, in
Train()
File "train.py", line 27, in init
self._train()
File "train.py", line 40, in _train
self._train_epoch(i_epoch)
File "train.py", line 69, in _train_epoch
self._model.optimize_parameters(keep_data_for_visuals=do_visuals, train_generator=train_generator)
File "/home/ubuntu/GANimation/models/ganimation.py", line 207, in optimize_parameters
loss_D, fake_imgs_masked = self._forward_D()
File "/home/ubuntu/GANimation/models/ganimation.py", line 275, in _forward_D
fake_imgs_masked = fake_img_mask * self._real_img + (1 - fake_img_mask) * fake_imgs
RuntimeError: The size of tensor a (176) must match the size of tensor b (178) at non-singleton dimension 3

I checked the shapes for more detail:
(Pdb) fake_img_mask.shape
torch.Size([25, 1, 216, 176])
(Pdb) fake_imgs.shape
torch.Size([25, 3, 216, 176])
(Pdb) self._real_img.shape
torch.Size([25, 3, 218, 178])

python train.py --data_dir mydata2 --name experiment_1 --batch_size 25

------------ Options -------------
D_adam_b1: 0.5
D_adam_b2: 0.999
G_adam_b1: 0.5
G_adam_b2: 0.999
aus_file: aus_openface.pkl
batch_size: 25
checkpoints_dir: ./checkpoints
cond_nc: 17
data_dir: mydata2
dataset_mode: aus
display_freq_s: 300
do_saturate_mask: False
gpu_ids: [0]
image_size: 128
images_folder: imgs
is_train: True
lambda_D_cond: 4000
lambda_D_gp: 10
lambda_D_prob: 1
lambda_cyc: 10
lambda_mask: 0.1
lambda_mask_smooth: 1e-05
load_epoch: 0
lr_D: 0.0001
lr_G: 0.0001
model: ganimation
n_threads_test: 1
n_threads_train: 4
name: experiment_1
nepochs_decay: 10
nepochs_no_decay: 20
num_iters_validate: 1
poses_g_sigma: 0.06
print_freq_s: 60
save_latest_freq_s: 3600
serial_batches: False
test_ids_file: test_ids.csv
train_G_every_n_iterations: 5
train_ids_file: train_ids.csv
-------------- End ----------------
./checkpoints/experiment_1
Dataset AusDataset was created
Dataset AusDataset was created
#train images = 202056
#test images = 202056
Network generator_wasserstein_gan was created
Network discriminator_wasserstein_gan was created
Model GANimation was created
/data/GANimation/models/ganimation.py:320: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('g_mskd_fake', self._loss_g_masked_fake.data[0]),
/data/GANimation/models/ganimation.py:321: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('g_mskd_cond', self._loss_g_masked_cond.data[0]),
/data/GANimation/models/ganimation.py:322: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('g_cyc', self._loss_g_cyc.data[0]),
/data/GANimation/models/ganimation.py:326: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('g_m1', self._loss_g_mask_1.data[0]),
/data/GANimation/models/ganimation.py:327: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('g_m2', self._loss_g_mask_2.data[0]),
/data/GANimation/models/ganimation.py:328: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('g_m1_s', self._loss_g_mask_1_smooth.data[0]),
/data/GANimation/models/ganimation.py:329: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('g_m2_s', self._loss_g_mask_2_smooth.data[0]),
/data/GANimation/models/ganimation.py:331: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('d_real', self._loss_d_real.data[0]),
/data/GANimation/models/ganimation.py:332: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('d_cond', self._loss_d_cond.data[0]),
/data/GANimation/models/ganimation.py:333: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('d_fake', self._loss_d_fake.data[0]),
/data/GANimation/models/ganimation.py:334: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
('d_gp', self._loss_d_gp.data[0])])
[18/10/2018 18:24:40] (Tv, epoch: 1, it: 0/8082, t/smpl: 0.049s) g_fake:0.000 g_cond:0.000 g_mskd_fake:-0.028 g_mskd_cond:15517.005 g_cyc:3.964 g_rgb:0.000 g_rgb_un:0.000 g_rgb_s:0.000 g_m1:0.057 g_m2:0.057 g_m1_s:0.764 g_m2_s:0.755 g_idt:0.000 d_real:-0.026 d_cond:16090.376 d_fake:0.013 d_gp:7.328
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/util.py", line 274, in _run_finalizers
finalizer()
File "/usr/lib/python2.7/multiprocessing/util.py", line 207, in call
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/util.py", line 274, in _run_finalizers
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python2.7/shutil.py", line 252, in rmtree
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/util.py", line 274, in _run_finalizers
finalizer()
File "/usr/lib/python2.7/multiprocessing/util.py", line 207, in call
onerror(os.remove, fullname, sys.exc_info())
Traceback (most recent call last):
File "/usr/lib/python2.7/shutil.py", line 250, in rmtree
File "/usr/lib/python2.7/multiprocessing/util.py", line 274, in _run_finalizers
os.remove(fullname)
res = self._callback(*self._args, **self._kwargs)
OSError: [Errno 16] Device or resource busy: '/tmp/pymp-XanIEN/.nfs00000000087e0d9300004faf'
File "/usr/lib/python2.7/shutil.py", line 252, in rmtree
finalizer()
File "/usr/lib/python2.7/multiprocessing/util.py", line 207, in call
finalizer()
res = self._callback(*self._args, **self._kwargs)
onerror(os.remove, fullname, sys.exc_info())
File "/usr/lib/python2.7/multiprocessing/util.py", line 207, in call
File "/usr/lib/python2.7/shutil.py", line 252, in rmtree
File "/usr/lib/python2.7/shutil.py", line 250, in rmtree
res = self._callback(*self._args, **self.kwargs)
os.remove(fullname)
File "/usr/lib/python2.7/shutil.py", line 252, in rmtree
OSError: [Errno 16] Device or resource busy: '/tmp/pymp-w9YT_u/.nfs00000000087e0d8f00004fb1'
onerror(os.remove, fullname, sys.exc_info())
File "/usr/lib/python2.7/shutil.py", line 250, in rmtree
os.remove(fullname)
OSError: [Errno 16] Device or resource busy: '/tmp/pymp-VtcW1b/.nfs00000000087e0d9100004fb2'
onerror(os.remove, fullname, sys.exc_info())
File "/usr/lib/python2.7/shutil.py", line 250, in rmtree
os.remove(fullname)
OSError: [Errno 16] Device or resource busy: '/tmp/pymp-DjMpR/.nfs00000000087e0d9500004fb0'
Traceback (most recent call last):
File "train.py", line 141, in
Train()
File "train.py", line 27, in init
self._train()
File "train.py", line 40, in _train
self._train_epoch(i_epoch)
File "train.py", line 82, in _train_epoch
self._display_visualizer_train(self._total_steps)
File "train.py", line 98, in _display_visualizer_train
self._tb_visualizer.display_current_results(self._model.get_current_visuals(), total_steps, is_train=True)
File "/data/GANimation/models/ganimation.py", line 347, in get_current_visuals
visuals['1_input_img'] = plot_utils.plot_au(self._vis_real_img, self._vis_real_cond, title=title_input_img)
File "/data/GANimation/utils/plots.py", line 14, in plot_au
fig = plt.figure()
File "/usr/local/lib/python2.7/dist-packages/matplotlib/pyplot.py", line 533, in figure
**kwargs)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/backend_bases.py", line 161, in new_figure_manager
return cls.new_figure_manager_given_figure(num, fig)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/backends/_backend_tk.py", line 1046, in new_figure_manager_given_figure
window = Tk.Tk(className="matplotlib")
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 1818, in init
self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use)
_tkinter.TclError: no display name and no $DISPLAY environment variable

is that something wrong with my data?

I have two 1080Ti graphics cards, How to call two graphics cards at the same time

i pulled the openface with docker.
and test with the command build/bin/FaceLandmarkImg -f sample1.jpg
build/bin/FaceLandmarkImg -f sample2.jpg etc.
and it produced a directory name processed with several files.
sample1_aligned sample1_of_details.txt sample1.csv sample1.hog sample1.jpg(with annotation)
sample2_aligned sample2_of_details.txt sample2.csv sample2.hog sample2.jpg(with annotation) and so on.
and then i want to generate the aus_openface.pkl with the following command
python data/prepare_au_annotations.py
and the command need some arguments.so instead i us p
ython data/prepare_au_annotations.py --input_aus_filesdir processed(as produced before with several files) --output_path outpkl
and there is a file named aus.pkl which is not as you described aus_openface.pkl.
in order to continue,i just rename it to aus_openface.pkl.
and below i just make a directory named mydata
inside the mydata is imgs folder and aus_openface.pkl,train_ids.csv,test_ids.csv
within the imgs folder,i don't know to put the original sample*.jpg or put the image within sample1_aligned folder .and in sample*_aligned folder is different image but the same name face_det_000000.bmp.
i am quite confused.
i want to know that which images should i put in the imgs folder,and should i just change the pkl name?are there any thing i do wrong in the process?
forgive my poor expression,thank you for your patient guidance

Matplotlib 3.0+ does not support Python 2.x, 3.0, 3.1, 3.2, 3.3, or 3.4.
Beginning with Matplotlib 3.0, Python 3.5 and above is required.

This may be due to an out of date pip.

Make sure you have pip >= 9.0.1.

so i upgrade python to python3.6 in order to install Matplotlib 3.0
however,i start to train with bash launch/run_train.sh
launch/run_train.sh: line 3: python: command not found
how can i solve this