Images aren't saved when using flag -s about pytorch-studiogan HOT 7 CLOSED

[Flow of StudioGAN]

CUDA_VISIBLE_DEVICES=0 python3 src/main.py -t -e -l -c CONFIG_PATH -s -iv -knn -itp -tsne -fa

0. current_iter =  0
1. While current_iter > total_iter: 
2.     Repeat: Train -> Temporary Evaluation(-s, -iv, -knn, -itp, -tsne, and -fa modes do not work since It spends lots of time)
3.     current_iter += 1

4. Final Evaluation with the best checkpoint
5. -s
6. -iv
7. -knn
8. -itp
9. -fa
10. -tsne

CUDA_VISIBLE_DEVICES=0 python3 src/main.py -t -l -c CONFIG_PATH

0. current_iter =  0
1. While current_iter > total_iter: 
2.     Repeat: Train
3.     current_iter += 1

You can save images after training is done.

And, I will add an image canvas saving module to see generated images during training.

Thank you.

from pytorch-studiogan.

@mingukkang Ok, training is done, images are saved. But seems like training labels do not correspond to saved images labels. As far as I know, StudioGAN uses ImageFolder from Pytorch under the hood that sorts labels alphabetically.
I used specific images divided into 4 classes. After training procedure, samples folder has the next structure - 0, 1, 2, 3. Each folder per class. But turns out that some labels are flipped, you know, not corresponds to the original one. For example, image with label 0 (I know it should be exactly that label) may be marked as label 3. Is this a bug in code (sorting labels on save step, etc.) or I should've trained it for more iterations?

Unfortunately, I can't test it with more obvious data like MNIST, when you can exactly name the label just by looking at the image, because I already spent $65 on AWS in 3 days 😅

from pytorch-studiogan.

Hi,

Thank you so much for using our code in your experiment:)

Unfortunately, it seems to be the same phenomenon as the issue.

We met this kind of problem since we had evaluated GAN's performance using traditional ways: IS, FID, and Precision and Recall.

Best,

Minguk

from pytorch-studiogan.

Hello,

I have added an execution of "run_image_visualization" method during evaluation.

Now, you can directly see generated images at ./figures/RUN_NAME/generated_canvas.png.

Please refer to Link for more details.

Thank you.

Best,

MInguk

from pytorch-studiogan.

How to save images during training?

from pytorch-studiogan.

Hi,

You can save generated images using -s flag only on evaluation time.

Before adding DistributedDataParallel, StudioGAN had a part of saving images during training.

But, DDP made a trouble with it, and I determined to remove the saving module for StudioGAN to work all protocols.

I understand that visualizing images is a straightforward way to identify success of GAN training, so I will update this ASAP.

Thank you.

Best,

Minguk

from pytorch-studiogan.

@mingukkang Hi! I don't get it completely) You mean I can only save images after training is done by running script again only for evaluation with -s flag? Because my script is already running with these flags -iv -s -t -e. But there's no samples folder, only ROC figures. Am I doing something wrong?

from pytorch-studiogan.