G loss increases much suddenly? about improved_wgan_training HOT 2 CLOSED

I experienced the same outcome in my implementation when I added learning rate decay every 25k iterations. Whenever the learning rate decreases the loss of G goes up, the real loss of D goes up, the fake loss of D goes down (- loss G) but the overall loss of D continues to increase as normal.

Any idea what could cause this @igul222 @martinarjovsky ? Is there some correct way of doing this learning rate decay? In the paper, it is mentioned that you implemented learning rate decay for the ResNet-101.

Did you manage to find a fix @LynnHo ?

from improved_wgan_training.

@crisbodnar I have found recently in my work that layer normalization may cause this phenomenon, and instance normalization may help.

from improved_wgan_training.