Ambiguity in the query about jahs_bench_201 HOT 3 OPEN

I guess the surrogate is fine with out of domain values but i agree that an out of domain check would be helpful. I ended up creating my own configs with a validate() and manually specifying the bounds to counter this.

from jahs_bench_201.

This is a very valid point, thank you for bringing it up! The way the number of epochs is currently being handled is definitely sub-optimal and will be revised in the upcoming updates.

Regarding the domain checks, this is something that requires some discussion.

• For the surrogate, there is technically no reason for forbidding out-of-domain values as long as a user is aware that our underlying model, XGBoost, is not very suitable for extrapolation. My personal preference would be to only raise a warning when out-of-domain values are encountered. Adding the option to enforce domain bounds and/or raise an error instead would then be a nice-to-have for users who want that.
• For the tabular dataset, domain checks actually make no sense, since the dataset can only be queried on the specific values that were originally sampled in any case. The thought behind the current design was that exposing the ConfigSpace implementation of the search space is enough to provide the requisite information to any algorithms that need the domain bounds, but maybe this can be improved.

What is your opinion?

from jahs_bench_201.

Hi, thanks for the response:)

For the tabular, I don't think we need the check, but for the surrogate model, it would be grateful to have warning at least because XGBoost gives, by nature of the decision tree, the same prediction on each parameter which is rounded to the original range.

from jahs_bench_201.