Convenience transformation for config spaces about syne-tune HOT 8 CLOSED

Hello Andreas, I am closing this one, but feel free to reopen it with a feature request. We could indeed provide more convenience tooling, for example an abstraction for derived parameters (they'd have to be functions of the original parameters).

from syne-tune.

Hello,
Syne Tune does not support conditional configuration spaces. This is mostly because we've been focused on Bayesian optimization, where supporting conditional spaces is difficult.

For your first example, I'd use two variables, x [randint] and x_is_none [choice]. For the second, as you said, I'd encode the difference as non-negative. Again, we do not support fixed constraints between the parameters, as this would need constrained optimization of the acquisition function.

from syne-tune.

I think it would be nice to have some sugar that allows encoding factors or exponents; often you want powers of two, and/or factors of difference. These can be taken into account downstream but if synetune could do the translation internally, that would make the use easier.
Indeed, I can encode the factor myself but then the space that synetune operates in is different from the space I want to see in MLFlow.
It wouldn't require any changes to the BO to allow this translation, and it would also allow encoding things like powers of two more naturally.

Anyway, thanks for your replies so far, I'm actually running it on a real system now, let's see how it goes.

from syne-tune.

Hi Andreas,

Regarding power of two of exponents, it is already supported (unless I misunderstood your question).

For instance,

from syne_tune.config_space import logfinrange
hp = logfinrange(lower=8, upper=128, size=5, cast_int=True)
print(hp.sample(size=10))

what do you mean by factor of differences?

from syne-tune.

Oh it is, thank you, I overlooked that, I think I tried lograndint but that wasn't the right one. Thank you!
What I mean, it something like "I want the hidden size of the transformer to be somewhere between 1x and 4x the encoding size" or something like that. Basically you want a conditional distribution of one parameter given another, and I think one of the simplest forms of that is that one is a random multiple of the other.

from syne-tune.

Oh I see, we do not support hyperparameters that are function from the others at the moment. I think it would make sense to support general conditional search space but it is a larger effort, in the meantime having one "ratio" as an hyperparameter is probably the easiest I would say.

from syne-tune.

I guess I was wondering how many of the conditional space cases can be covered by a reparametrization, which wouldn't require any changes to the optimizer, only to how the search space is presented to the optimizer.
I guess that's always possible but the question is whether any auxiliar variables are required and how easy it is to learn for a GP in the transformed space?

from syne-tune.

I believe the auxiliary variables should be learnable without too much difficulty for a GP but of-course it depends on the exact range being used, let us know if you observe something strange.

from syne-tune.