Nan Loss after a couple of epochs about adabins HOT 3 CLOSED

Hi,

First of all very nice paper and architechture! Ive been playing with the training code and trained a model with my own data. However after a couple of epochs im getting a Nan loss and after some debugning it seems like its the output of the model returns a nan tensor. I played a little bit with gradient clipping to see if it would help but it does not seem to be the case.

Best regards

Hi
Have you found a solution.I'm facing the same problem

from adabins.

One reason might be that at some point loss has a 'nan' value. What is the minimum depth you are using?
The code uses SILog and if your depth has zeroes (or negative) this will lead to nan values.

If that's the case, you can:

1. Clip (from below) your depth values to some minimum positive value (like 1e-3).
2. You can change the loss in line 19 in loss.py (

   AdaBins/loss.py

   Line 19 in 2fb686a

   g = torch.log(input) - torch.log(target)

   )
   to :

alpha = 1e-3
g = torch.log(alpha + input) - torch.log(alpha + target)

Let me know if this solves the issues.

from adabins.

from adabins.