Add cross-validation for small datasets about farm HOT 8 CLOSED

Hey @tripl3a, this is a great idea and we think it should be a functionality inside of farm. Especially for small datasets you want some more stable evaluation criteria. It wont be super straightforward to implement though : )

A fast way would be to do it outside of FARM:
Pytorch doesn't have native CV code yet: pytorch/text#486
There is a Pytorch wrapper that does: https://github.com/skorch-dev/skorch It seems to be able to pass a Pytorch module (our adaptivemodel is one) to a Classifier and also Pytorch Datasets. If you want to try this, you could create a separate function in experiment.py, something like run_experiment_crossvalidated(n_folds=5, args). We have no experience with this package unfortunately.

If you want to integrate CV inside FARM, we should have a call and discuss how we proceed in more detail since it will create a lot of changes deep inside processor, data silo and trainer classes.

Do you have any preference or any thoughts about this?

from farm.

OK I have an example xval for doc classification, but it depends on the changes made in pull request 165, so I will wait for that one to get merged or further worked on.

from farm.

Thanks again @Timoeller for the feedback and for taking the time to have a call with me yesterday! As discussed, I don't have the capacity to implement such deep changes in FARM at the moment. Instead, I'll take a look at skorch and keep you posted.

from farm.

Thanks again @Timoeller for the feedback and for taking the time to have a call with me yesterday! As discussed, I don't have the capacity to implement such deep changes in FARM at the moment. Instead, I'll take a look at skorch and keep you posted.

@tripl3a have you been successful to use skorch and would you be able to share code / insights?

from farm.

@johann-petrak No, I haven't tried it yet.

from farm.

From a quick look I had it does not look trivial to adapt/wrap the FARM dataset, model, optimizer etc so that it can be used like skorch needs it.

from farm.

I will try to come up with some example code for this but without using skorch, just using some of the sklearn functions.

from farm.

Merged in #167. Thanks @johann-petrak!

from farm.