Problems with Replacing ReLU with eLU about improved_wgan_training HOT 11 CLOSED

from improved_wgan_training.

I have also experienced the same effect and ended up reducing the learning rate to compensate for it.

from improved_wgan_training.

I also experienced the same effect. Reducing learning rate does not have any effects on this issue.
I observed that W perturbing and diverged when I only trained critic networks. Any thoughts?

from improved_wgan_training.

@hiwonjoon , have you tried using weight norm in your conv1d? also have tried decreasing beta1?

from improved_wgan_training.

@NickShahML Can you explain why decreasing beta1 should help?

from improved_wgan_training.

my (very rough, hand-wavy) intuition: beta1 is a momentum term. if you think of momentum as using past gradients as an estimator for the current gradient, it follows that momentum might not be helpful on loss surfaces with sharp curvature. gradient penalty introduces a lot of this through multiplicative interactions between weights in the loss fn. this makes optimization with momentum less stable sometimes. (eLUs seem to be tricky to optimize for similar reasons). note that none of this means you can't make it work -- you'd just need to drop the learning rate so much that it's probably not worth it.

from improved_wgan_training.

Yea, I've found that dropping the learning rate from ELU does work though you have to drop it so much that they aren't worth it. You could try SELU instead but I've experienced the same effect.

from improved_wgan_training.

For curiosity's sake, would SELU eliminate the need of normalization in the Discriminator? @NickShahML

from improved_wgan_training.

@Jiaming-Liu I don't know if SELU would necessarily eliminate the need to normalize but in theory it should.

from improved_wgan_training.

@rkjones4 There is a theoretical reason. By adding the gradient penalty in the objective during the critic training, the resulting gradient update contains terms of second order derivatives of the network's activation functions. For non continuous second order derivatives this can lead to a collapse of the training. Remember that ELU has a non continuous second order derivative. This non continuity ruins the objective by producing strange behaviours in the gradient penalty.
Just have a look on the latest version of: https://arxiv.org/pdf/1704.00028v1.pdf

from improved_wgan_training.

from improved_wgan_training.