elife-asu / innlab Goto Github PK

How do I train INN with my dataset?

I faced this problem "The algorithm failed to converge because the input matrix is ill-conditioned or has too many repeated singular values (error code: 767)" during the training of JacobianLinear layer. It seems that the input feature need to fit this requirement.

In utilities.default_net(), we have a default activation function (currently, we use nn.SELU). Which one works best?

When we need to add other loss components, the log(p) and log|det(J)| have to be averaged to make it easy to balance. This issue also make the https://github.com/ELIFE-ASU/Macro be hard to handle.

If we just want INN.Conv2d output the first value, where can we set the value of "compute_p" ?

We need a test program for checking Jacobian calculation.

Dear authors,

Thank you for creating such a great project. I would like to know how I can train an INN? Is the training process the same as normal pytorch?

Best,
Lin

In INN.BatchNorm1d, the forward function is:

def forward(self, x, log_p=0, log_det_J=0):
        
        if self.compute_p:
            if not self.training:
                # if in self.eval()
                var = self.running_var # [dim]
            else:
                # if in training
                # TODO: Do we need to add .detach() after var?
                var = torch.var(x, dim=0, unbiased=False) # [dim]

            x = super(BatchNorm1d, self).forward(x)

            log_det = -0.5 * torch.log(var + self.eps)
            log_det = torch.sum(log_det, dim=-1)

            return x, log_p, log_det_J + log_det
        else:
            return super(BatchNorm1d, self).forward(x)

Do we need to requires var has gradient information? It seems not training BatchNorm1d, but training modules before it. Is there any references on this?

Hi @Zhangyanbo!

Thank you for creating this great package! Would you mind adding a license to it, so that the legal implications are clear when it is used by third party projects?

Christoph

The output of RealNVP can be very high. This is because it have torch.exp is the forward process:

y = x * torch.exp(self.logs(x)) + self.t(s)

Is there any method that do all requirement below at the same time:

1. avoid very large number
2. avoid very small number