Training with object detection , Loss : Nan about machinelearning HOT 10 OPEN

@michaelgsharp my best guess is there's overflow when calculating focal loss?

from machinelearning.

Is there a solution or suggestion? please

from machinelearning.

@Cilouche which coco dataset are you using, could you share a link?

from machinelearning.

data : https://drive.google.com/drive/folders/1-dQPRdQ-MRp6mrPhnpng5pcgJsTZMg23 ,

I used this site https://drive.google.com/drive/folders/1-dQPRdQ-MRp6mrPhnpng5pcgJsTZMg23 to convert them to coco format

from machinelearning.

@Cilouche which site? The site link seems to be the same with the data you share

from machinelearning.

Yes sorry the site :
https://roboflow.com/convert/pascal-voc-xml-to-coco-json?ref=blog.roboflow.com

I've also noticed that once the database is large, there's a loss Nan
example:
data = 100, epoch=8; all is well except for the precesion is low ~ 0.69

but from data ~= 1200 images, epoch= 5, 8 , 11 ; losses converge rapidly towards Nan

from machinelearning.

Update

I got it reproduced on my second training, thanks

Origianl post

Hey @Cilouche some updates here: I can't reproduce the NaN loss error using your dataset in the latest model builder main branch. Maybe it's already been fixed.

We haven't released model builder yet, but you can verify the latest bit in mlnet cli > 16.18.2 by installing mlnet-win-x64 and try object detection there. mlnet cli and model builder shares the same AutoML service so if you didn't see NaN error from mlnet cli, then you probably also won't see NaN error from model builder

steps to verify

• install the latest mlnet-win-x64 （https://www.nuget.org/packages/mlnet-win-x64)
• start object detection there

mlnet object-detection --dataset /path/to/coco.json --device cuda --epoch 1

from machinelearning.

Any suggestion or solution to bypass this problem, please ?
Thanks

from machinelearning.

Try a smaller batch size, Maybe 1?

And GPU training doesn't produce NaN loss, is that also an option for you(training on GPU)

from machinelearning.

It's works on GPU thanks.

from machinelearning.