taotaonice / faceshifter Goto Github PK

I am trying to run it on colab. I am saving file in stead of cv2.imshow since colab does not support that.
But it results totally in black image

我复现网络的时候，发现dst图像放到identity中 大小不适配，问了作者，作者是用arcface重新训练的，我现在也正在重新训练，查找资料看到你的工作，觉得你写的挺好的，想跟你沟通一下，你这边会接着做第二阶段的工作吗？我们加个扣扣？1529838335

Hi, taotonice
Nice work! paper author said he use CelebHQ\FFHQ\VGGface dataset, and did not tell about some details about dataset. what dataset did you use? and since you have did some experiments, could you give some suggestions or experience about trainning? thank you very much

Hi, taotaonice.
Thanks for the model you provided.

I ran demo, it worked well.
but when I tried train HEARnet and ran HEARnet demo, I could not find demo.

in your code (train_HEAR.py), It require 3 datasets input(hearnet_data, ego_hands_png, shapenet_png).
but I don't know what is meaning of that specifically. In your paper, these datasets don't need hand_data or shapenet_data. but it require that in code.

could you let me know what dataset you used?

and if you are ok, could you provide pretrained model what you trained?
([email protected])

I found where dataset is to your paper.
but when I ran train_HEAR.py, It cannot work well.
so I printed everywhere and i found that data didn't put into AugmentedOcclusions.

dataset = AugmentedOcclusions('../data/face_data', #'../data/hearnet_data',  
                              ['../data/ego_hands_png/ego_hands','../data/ego_hands_png/GTEA_hand2k/Images'],
                              shape_list, same_prob=0.5)

I also try ptint in AugmentedOcclusions class(in util/Dataset.py).
but when I ran code, I didn't appear in screen.
could you teach me a solution??

Hi, thanks for your great work！I have questions about the training.
1、Given a certain dataset, how to choose the pairs of source/target imgs in the training? Do you randomly pick two images as source and target imgs respectively in each iter? Or do you create the fixed list for pairs of source/target imgs before the training and train based on this list?
2、Besides，do the genders of each pair of source/target images need to be same? Or any othe limitation for the choice of pairs?
Thanks!

Hello
The way you choose a different image_path for Xt suppose each data_path has roughly the same number of images. If that's not the case, images from the smaller data_paths will appear more often as target. I would rather pick a random number using the full length of the dataset (and even ensuring it's different from item) and find which data_path it belongs to, as in the beginning of the function.

Could you please provide the pretrained model of Arcface using in your project? Thanks.

great job.
Can you send me the pre-trained model to
[email protected]

Can You please send me the trained models or at least a usage guide on how to train them.

I train AEI net and the error is "expected scalar type Half but found Float"
I found this error is caused by L_id. After I delete L_id from lossG the error becomes "CUDA error: an illegal memory access was encountered"
with amp.scale_loss(lossG, opt_G) as scaled_loss: scaled_loss.backward()

Should you discard gradient calculation like in line 97 when computing arcface embeddings ?

Can you write a readme document to show how to perform this repo ?
I will appreciate it.

Can you please explain why you are loading D and G weights on the cpu with a model on gpu ?
Thanks in advance

Has anyone downloaded the FFHQ,VGGFace and celeba-HQ dataset? It is too slow to download from the beginning, and many data links are invalid。Thanks a lot!

你好，你复现出效果来了吗？我也在复现这篇文章，可以交流一下，方便加个QQ吗

Hi, if you are reproducing this project, we can communicate. My email is [email protected]. I also have many problems.

Could you please send me a pretained model? My email is [email protected]. Thanks!

Hi taotao, did you (or have you heard anyone) reproduced the performance claimed by the original paper?

image->opencv->PIL，then the PIL image will be in BGR format, so the model will use this as input. This may influence the arcface feature.

I am confused about line 105 in train_AEI.py which is same_person = diff_person. Could you please explain it ?

The authors of the original paper have issued a new version in late June, and I noticed they corrected the training strategy of AEI and now Xs Xt are supposed to be different (and not identical anymore) 80% of the time
So same,_prob=0.8 should be changed to 0.2 on line 56 of train_AEI.py
I'm going to try this new value and I'll let you know

Why are the L_rec and L_att loss functions implemented different than that of the paper?

in the paper L_att, and L_rec are the L2 distance but you do squaring followed by meaning
Shouldn't they be for L2 distance: torch.norm((X - Y),2) instead of torch.mean(torch.pow(X - Y, 2))?
The difference in the loss between them is large.

Hello

Can you explain why you did not use the mean for L_adv for calculating G loss but use it for loss_true and loss_fake when calculating D loss ?

Thanks in advance

@taotaonice please could you send me the pretrained model:

[email protected]

I can host it to and link it here as many people request this

IMHO this is due to MTCNN weights trained with RGB images whereas OpenCV use BGR: changing channels order from BGR to RGB before calling detector.align in inference_demo.py or online_preview.py or before calling mtcnn.align_multi in preprocess_images.py (and always restoring BGR after since AEI is trained with BGR images), give me far better face detection

Can you share the pretrained model to [email protected]?

I test your weights(only AEI-NET) and it seems great, but for Asian face the performance is not as goods as this. So I want to finetune on Asian face set.

Should I use With_Identity('../washed_img/', 0.8) to fintune? And what's the classes in With_Identity()?

if not fine_tune_with_identity:
    dataset = FaceEmbed(['../celeb-aligned-256_0.85/', '../ffhq_256_0.85/', '../vgg_256_0.85/', '../stars_256_0.85/'], same_prob=0.3)
else:
    dataset = With_Identity('../washed_img/', 0.8)

给我一份Pretrained model 吧，谢谢啦。 我的邮箱是[email protected],感激不尽

In the paper, the author said " The ratio of training samples that have Xt! = Xs is 80% when training the AEI-Net, it is 50% when training the HEAR-Net. "
But, I noticed that you set same_prob=0.8 in class FaceEmbed, which means that Xt != Xs is 0.2. Is it right?

Hi @taotaonice ,

I try to train your model on a K80 GPU with 12GB of memory
If I use batch_size 16, the program ends with "CUDA out of memory" error
I set batch_size 6, I check the program uses more than 10GB of memory.

Could you tell me the environment you can train with batch_size 16
and could you give some hints, how can we train the model on multi GPUs, cause I have two K80 cards.

Thank you very much!

How many epochs does it cost to obtain reasonable results?

Hi Taotanice,

Firstly thank you for providing the pretrained generator model.

Secondly, I've tried running the demo_inference.py and the results are pretty inconsistent and not that great.
I also tried adjusting and running the code from online_preview.py, which has a bit more going on with the mask and everything, but without much success. I did find that opening target image in BGR mode made the switch more aggresive but resulted in more artifacts.

I'm wondering what is your primary mode of testing the performance of the model?

Can someone please write up instructions for using?

Thanks!

hi，你好，
感谢开源，最近也研究了一下FaceShifter，效果真的惊人，也想复现一下效果
我用vggface和FFHQ进行训练的，统一resize到256的，训练机器P40，训练有几天了，效果一直不理想，不清楚问题出在哪里，后面会仔细研读您的代码，找找问问所在
不知道是否可以开源AEI_net和HEAR_net 的 pretrain model，如果不方便开源，是否可以发我邮箱，非常感谢!
[email protected]

maybe use asian face dataset to train FaceShifter and Arcface?
increase the identity loss?

Hi @taotaonice

I am building the acceleration module, things I found, point to using conda
But I do not use conda, could you give me some hints to build this module?

Thank you very much!
Hai Le

Why doesn't the code support high resolutions such as 1024x1024?

I'm still struggling with AEI learning: I had to reduce the batch size to 5 due to my card memory, but it collapse between 10 and 14 epochs. I'm trying another optimiser (ranger) which hasn't collapsed so far, but it's painfully slow.
I may try other tricks like different optimisers for G and D or gradient accumulation...
Currently my question is: why have you commented out torch,no_grad in get_attr function ? I can't understand why Y should contribute to gradient calculation.

I see that your setting is 2000 epochs. Did you train your model in 2000 epochs?
Thank you, looking forward to hearing from you.

我简单研究了下论文， HearNet训练部分还没有完成么？

Thank for sharing your code.
The reconstruction loss implemented at:

Happy Chinese New year!!!
I wan to run the traing, Can you teach me how set the Training Datasets?
Would you mind sharing your weights so that I can do some tests and continue training using datasets.

I've just noticed the requirements of the pretrained model weights. I can provide our group one. Links are below.
Weight 1
Weight 2
In our project, we loaded those weights and had a try on this model. You can have a look at this repo. Click open in Colab for the modelC at the bottom of the readme.
Or you can go there via

Hi ,
I met this error when running the trai_AEI file

Traceback (most recent call last):
File "train_AEI.py", line 131, in
ZY, Y_feats = arcface(F.interpolate(Y_aligned, [112, 112], mode='bilinear', align_corners=True))
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 539, in call
result = self.forward(*input, **kwargs)
File "/home/user/FaceShifter/face_modules/model.py", line 141, in forward
x = m(x)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 539, in call
result = self.forward(*input, **kwargs)
File "/home/user/FaceShifter/face_modules/model.py", line 76, in forward
res = self.res_layer(x)
.......
......
File "/home/user/.local/lib/python3.6/site-packages/apex/amp/utils.py", line 63, in maybe_half
return x.half()
RuntimeError: CUDA error: an illegal memory access was encountered

Anyone met this error, and could show me the hint?

Thank you!!

Could you please expalin why you crop Xs to Xs[19:237, 19:237] before sending to arcface ? The code lies in Line 100 of train_AEI.py ?
embed, Xs_feats = arcface(F.interpolate(Xs[:, :, 19:237, 19:237], [112, 112], mode='bilinear', align_corners=True))

First, thanks for your great work. I have a question about the discriminator.
In the paper, the author said they used the same multi scale discriminator as the StyleGAN. But I think StyleGAN doesn't use a multi scale discriminator.
In your implementation, you seems to use the multi scale discriminator from pix2pixhd, but you optimized it with a hinge loss. This looks strange to me since the multi scale discriminator outputs not the simple fake/real prediction but a feature map. So is it reasonable to optimize it with the hinge loss?

Hi, thanks for your awesome work and sharing. I wanted to test your model I would really appreciate if you could share the pretrained weights. thank you so much.
[email protected]

Line 102 , network/AEI_Net.py

m = self.up1(z_id.reshape(z_id.shape[0], -1, 1, 1))

Is z_id supposed to be a tuple, a numpy array or a tensor ?

I converted z_id into a numpy array, and then I get the following error:
conv_transpose2d(): argument 'input' (position 1) must be Tensor, not numpy.ndarray

Hello, thanks for sharing this implementation.
I'm learning about face swap and am interested to train with the weights you've tested with, can you kindly share them here?