j-min / dropouts Goto Github PK

Hi, if I'm not mistaken, the alpha in your math is the variance, so one should scale the randn variables by sqrt(alpha)?

Thanks for posting this. However I found you can make things much faster by using the gpu for generating your random numbers as below:

class GaussianDropout(nn.Module):
    def __init__(self, alpha=1.0):
        super(GaussianDropout, self).__init__()
        self.alpha = torch.cuda.FloatTensor([alpha])
        
    def forward(self, x):
        """
        Sample noise   e ~ N(1, alpha)
        Multiply noise h = h_ * e
        """
        if self.train():
            epsilon = torch.cuda.FloatTensor(*x.size()).normal_() * self.alpha + 1

            epsilon = Variable(epsilon)

            return x * epsilon
        else:
            return x

Hello, thank you for the implementation code for dropout methods. I have a question that whether the alpha in your variational dropout changes, because I see the dropout rate is a constant which is not related to alpha.

Good afternoon,
I have been working on your implementation of Variational Dropout and something seems weird to me in the forward pass

            # Clip alpha
            self.log_alpha.data = torch.clamp(self.log_alpha.data, max=self.max_alpha)
            alpha = self.log_alpha.exp()

The code above seems to clamp the log of alpha and not the alpha itself which is not what we expect. Do I misinterpret you code or the theory here ?