Add resampling by cross-validation about mlj.jl HOT 4 CLOSED

Short but important comments:
(i) sd/sqrt(n_folds) is not a good estimator of anything, unless all the training and test pairs are non-overlapping!
(ii) I would strongly discourage that the resampling strategy itself produce summaries of any kind. This makes sense as the predictions or loss samples may be needed by meta-learning (e.g., ensembling, tuning) strategies, or evaluation (e.g., CI computation, comparative hypothesis tests)

from mlj.jl.

Yes, the individual fold estimates are not really independent and sd/sqrt(n_folds) is not ideal. However, in practice, this is the thing routinely reported by data scientists, no? In view of your objection (ii) I suppose we needn't argue this point. I am happy to drop the se method.

At present, there is an evaluate method for every resampler and this returns a single loss estimate. Questions:

1. Perhaps we instead make it a vector of estimates (a singleton for Holdout), yes?
2. I only envisaged tuning strategies that make use of a single estimate (or the mean of multiple estimates). Are you suggesting we may want to support tuning strategies that require more than a single estimate? This would complicate the API. If you are suggesting this, could you give some examples?

from mlj.jl.

However, in practice, this is the thing routinely reported by data scientists, no?

Yes, mostly those who use sklearn because sklearn reports it.
But wrong is still wrong...

1. Perhaps we instead make it a vector of estimates (a singleton for Holdout), yes?

can you explain what you mean here?

2. Are you suggesting we may want to support tuning strategies that require more than a single estimate?
   This would complicate the API. If you are suggesting this, could you give some examples?

The point is that
(a) mean is somewhat arbitrary. Why force a needless aggregation into the design which is irreversible.
(b) tuning strategies may often use the mean (i.e., a single aggregate), but reasonable evaluation strategies use the full sample of losses, e.g., to get confidence intervals, or measures that are not mean losses (e.g., auroc)
(c) there are some popular tuning strategies which don't only use the mean prediction loss, e.g., some variants of Bayesian optimization (to get an uncertainty), or boosting based ones (to get weights).

from mlj.jl.

This is now implemented, with all estimates returned as a vector. The current tuning strategies do use the mean, but this can be easliy changed later or new strategies can be addded.

Closing this. Feel free to open an issue for specific tuning strategy enahancement or new strategy.

from mlj.jl.