Allow turning off saving best model after epochs (DONE) and regression prediction instability (TODO) about ludwig HOT 9 CLOSED

Yes, it saves the best weights if the performance on validation improves. I can think about adding a --skip_save_best_model parameter, but then the model will never be saved. There could be some use cases where that could be useful (hyperparameters search for instance). I may also try to find a way to keep the best model weights copied in memory and just saved at the end. Will add this to the list of enhancements.

from ludwig.

Ludwig saves both the latest weights and the best weights. You can turn off saving the latest weights with skip_save_progress_weights, but you can't turn off saving the best weights, because, well, that's the whole purpose of training. Why would you want not to save the best weights?

from ludwig.

I meant by saving the best weight to the disk. Isn't it what it does? @w4nderlust

from ludwig.

Thanks, this will improve the performance for small training problems. Also, I've sent you a jupyter notebook and test data set for a regression problem showing the performance issue. What matters more is the regression results are so different between Ludwig and Keras with the same parameter setting, in which Keras results make more sense. If you want me open a new issue to describe it, I can. But I suspect it could be due to the way how biases are initialized. @w4nderlust

Yes, it saves the best weights if the performance on validation improves. I can think about adding a --skip_save_best_model parameter, but then the model will never be saved. There could be some use cases where that could be useful (hyperparameters search for instance). I may also try to find a way to keep the best model weights copied in memory and just saved at the end. Will add this to the list of enhancements.

from ludwig.

Those are two separate issues (speed and predictions), but will look at both of them, no need to open another issue. Thanks so much for the effort.

from ludwig.

Thanks a lot. Just a bit more information. I have also tried to fake a linear regression problem with Y = a * X + b + c*normal(0, 1) and create a single neuron/single layer for training. The results are very sensitive to how I initialize the network.

Those are two separate issues (speed and predictions), but will look at both of them, no need to open another issue. Thanks so much for the effort.

from ludwig.

So with the pushed fix the speed issue is in large part solved. Now there are parameters to turn off saving of intermediate models, progress and logs. Now on small models the speed compared to native TF is comparable (still slightly slower because of keeping track of statistics and generation of placeholders, but pretty close), while for bigger models, bigger datasets and bigger batch sizes the difference is negligible.

Still working on the prediction issue.

from ludwig.

from ludwig.

The regression isntability issue should have been solved. Please reopen if it is still an issue.

from ludwig.