AutoMLSequentialWrapper about neuraxle HOT 4 CLOSED

I look to your first draft and this looks good for most AutoMl algorithm like RandomSearch, TPE, GaussianProcess, Neuroevolution/GeneticAlgo, etc.

Just to make sure, when guessing the next best params (or when fitting) does the auto_ml_strategy has access to the probability distribution class (pdf, rvs, etc.)? Since TPE needs the probability distribution function (pdf) and needs to sample from probability distributions, to guess the next best params. And other auto_ml might will probably needs to rvs at least.

As for the trial crashes, I know that hyperopt used some kind of status to know if the trial has succeeded or not. We could use some kind of success flag also. It should come from the score_function or the validation_technique which needs to output a status flag.

Most of the automl technique can run in parallel. For instance, TPE needs the past history to suggest new points, but it is still faster to run multiple run in parallel and update the hyperparams_repository less quicker than waiting for each to complete before starting the next trial. As cited by this article : "The consequence of parallelization is that each proposal x∗ is based on less feedback. This makes search less efficient, though faster in terms of wall time". For parallelization, there should have some share hyperparams_repository for any Bayesian Optimization technique (GaussianProcess, TPE, Spearmint, etc.) and probably Genetic Algo also to have access to past history.

There is only hyperband that it is a bit particular. The algo speed up the search by first allocating fewer resources (epochs) to not promising training run and to allocate more and more resources on more promising training run each times. You will not be able to do Hyperband with this abstraction, but Hyperband is not the same kind of abstraction anyway.

So I think that this is a really good starting point.

from neuraxle.

@Eric2Hamel Thanks for the thoughts!

Good catch for the error status, that would for sure be a good thing to add status to the hyperparams_repository, perhaps that adding a try-catch (or a timeout when/if distributed?) would be interesting.

For the PDFs I thought about it and I'd need to add those methods in each of our Distribution classes, I'm not really sure of how to express/formulate those PDFs in the code of those distributions (help wanted). For now the auto_ml_strategy already have access to those distribution classes because we pass to it the result of the call of wrapped_pipeline.get_hyperparams_space which is an HyperparameterSpace dict containing the distribution instances.

Also yeah, probably that on suggesting the next trial to do that it could suggest a percentage of data to try on or something like that such as doing something like Hyperband, that'd be an interesting method to add to the auto_ml_strategy. However, if in Hyperband the next trials are the continuation of the first ones, then it'd perhaps need a new class different than AutoMLSequentialWrapper. I'd like to have suggestions on this too.

You also somehow remind me that I didn't include yet the wrapped_pipeline.inspect() method that I discussed in the conference where after training it'd be possible to save extra features of the trained model (e.g.: average and std of neurons' weights, loss' train curve and validation train curve, etc). That'd be another thing to add to the hyperparams_repository.

from neuraxle.

from neuraxle.

@Eric2Hamel Nice! You can definitely add the pdf of the hyperparameter distributions. I'd suggest doing it for one in a first PR for a first review and then proceeding to every other ones. I'm mostly unsure of how for example Hyperopt and other algorithms would query the PDF to use it (e.g.: min and max and mean? Continuous PDF function? Discrete array given a PDF resolution? etc.).

Also, it more and more sounds like Hyperband and Hyperband-like algorithms would be another meta wrapper step think. I think that to continue training it'd need to use a kind of minibatch pipeline and limit the number of epochs and resume from there with pipeline serialization between each steps.

from neuraxle.