using any regularization causes ever increasing loss about caffe HOT 5 OPEN

This is a little weird! Are you able to train the baseline without any regularization? Caffe is relatively old, and you should consider switch to others like pytorch.

from caffe.

Yeah that's really weird. Training without regularization leads to a useful loss and a kinda good accuracy (~90%). But I honestly don't understand why one of the normal regularization methods would cause this behavior. I am currently fine tuning this baseline with SSL and see where this goes. It also started with a really high loss (e+12) and is currently working its way down (e+10).

I know that caffe is getting old but I am currently working on my bachelor thesis where I am comparing interesting sparsification methods and I think SSL is a really interesting approach based on the fact that you don't need specialized hardware to get a acceleration from it.

from caffe.

@aradar Thanks for having interest in SSL. SSL can be easily applied to the frameworks supporting autograd, such as tensorflow and pytorch. You just need to add group Lasso regularization to the cross entropy and that's it. We have an RNN code for your reference.

from caffe.

Thank you for the reference code! I will look into implementing it myself for tensorflow.

But wouldn't I also have to implement sparse convolution ops for tensorflow to also get the speedup on a normal GPU and CPU?

from caffe.

@aradar if you remove structures (such as filters and channels), then you won't have to. You will just need to create a smaller DNN with the learned structures (such as fewer filters and channels) and initialize them by non-zeros.

from caffe.