Additive Noise Reparameterization about variational-dropout-sparsifies-dnn HOT 5 CLOSED

Hi, Daniil!

It's absolutely the same construction. Besides clipping T.exp(log_alpha) * self.W * self.W can be replaced with T.exp(self.log_sigma2 - T.log(self.W ** 2)) * self.W * self.W that is exatly log_sigma_2.

The beautiful thing about ART is that you have to change the only parametrization while evaluation is the same.

from variational-dropout-sparsifies-dnn.

Yep, It's the same thing, but if we will look at this equation without local reparameterization we will see that in current implementation you have implied_weight = W + W^2 * alpha * eps and thus variance of gradient will remain the same

from variational-dropout-sparsifies-dnn.

Actually no. Let's move to math to see it clearly. We are interested in d implied_weight / d log_sigma2 :

implied_weight = W + W^2 * alpha * eps
implied_weight = w + w^2 * exp(log_sigma2) * 1/exp(T.log(self.W ** 2)) * eps 
implied_weight = w + exp(log_sigma2) * eps 

After several lines, we can see d implied_weight / d log_sigma2 does not depend on alpha that's why thought "additive reparameterization takes no effect in the case of big alpha" is wrong.

from variational-dropout-sparsifies-dnn.

Yep, you are absolutely right. I forgot that alpha is obtained from sigma and W.
Thank you for help!

from variational-dropout-sparsifies-dnn.

Wellcome!

from variational-dropout-sparsifies-dnn.