Can this architecture find hyperparameters for multi-algorithm in parallel? about syne-tune HOT 4 CLOSED

Can you elaborate a bit more on your use-case? Do you want to search jointly for the right algorithm and hyperparameters?

from syne-tune.

the use-case is maybe there is one dataset, and I want to parallel find 5 algorithms hyperparameters, these 5 algorithms is independent each other. It means I start 5 ray actors for 5 algorithms to find hyperparameters. Before when I use ray.tune to do it. 5 ray.tune will mix the hyperparam each other.

from syne-tune.

What you could do is to define one categorical hyperparameter that defines the algorithm. Then you define all hyperparameters for your algorithms. For example, it could look like this:

config_space = {
    "algorithm_type": choice(['rf', 'xgb']),
    "learning_rate_xgb": loguniform(1e-6, 1e-1),
    "n_estimators_rf": randint(10, 500),
    "min_samples_split_rf": randint((1, 50))
}

Now, in your train function, you first check which algorithm to use and then pass it the corresponding hyperparameters. For example:

if algorithm_type == 'xgb':
    model = xgboost.XGBClassifier(learning_rate=learning_rate_xgb)
elif algorithm_type == 'rf':
    model = RandomForestClassifier(min_samples_split=min_samples_split_rf, n_estimators=n_estimators_rf)

This might be not the most efficient way, but as long as you use a random-search based method such as ASHA it should work.

from syne-tune.

Feel free to reopen if there are further questions

from syne-tune.