mlos_bench: parameter constraints about mlos HOT 9 OPEN

Regarding your observation, I think it's mostly a documentation issue.
This PR in ConfigSpace seems to indicate that Forbidden clauses between Hyperparameters, not just constants, is possible.

https://github.com/automl/ConfigSpace/pull/245/files

How I might handle this is something like the following:

• a separate config structure to define conditions and forbidden clauses
  • I can probably help with the json schema when we get to it, but I think something like the following to start with might be OK:

    {
       "constraints": [
          {
            // "op_name": ["param_name_1", "param_name_2"] --> param_name_1 {op_name} param_name_2
            "less_than": ["param_a", "param_b"]
          },
          {
            // Would have to be converted to a reverse "ForbiddenGreaterThanRelation" internally
            "less_than_equal": ["param_c", "param_d"],
          },
          {
            "equal": ["param_x", "param_y"],
          },
          ...
       ]
    }

• allow them to span different TunableGroups definitions (e.g., VM Environment TunableGroups and App Environment TunableGroups)
• when converting from TunableGroups to ConfigSpace, apply the Forbidden constraints only for Tunables that exist in the set (e.g., if a Condition exists declaring vm_ram > buffer_pool_size, but vm_ram is not in the included TunableGroups, issue a Warning but omit it)

Since not all backend optimizers support ConfigSpace (e.g., FLAML) we might also need to implement an internal "rejection" method inside mlos_bench for this (e.g., automatically report a bad value back when a suggested value returned by the optimizer is invalid)

from mlos.

Thanks for getting back to me about this, I will go over the linked documentation/PRs and reach out once I have a better idea of the approach.

from mlos.

Some things to note:

• TunableGroups provide reusable parameter specifications for individual Environments (e.g., VM, OS, App)
• They can be combined in an Experiment to do cross Environment co-tuning.
• Constraints may similarly need to be specified both within a TunableGroup and across TunableGroups. For instance:
  • app.min_file_size < app.max_file_size
  • app.buff_size_mb < (vm.ram_size_gb - 1) * 1024

This will require some care in thinking through what the config specification looks like as well as how to handle it in either TunableGroups vs ConfigSpace conversions or else automatic enforcement in the mlos_bench.Optimizer side of things.

from mlos.

Hi @bpkroth,

I had a look at the Conditions and the ForbiddenCluases. However, I am unsure how to implement conditions between hyperparameters. I think the conditions work on the child hyperparameter based on the parent hyperparameter and value. On the other hand, the forbidden clauses work on a hyperparameter/value(s) pair. From what I understand, we can't directly add a condition between two hyperparameters, is this correct or am I missing something here?

Thanks,
Pooja

from mlos.

I haven't thought through the full implementation in mlos_bench yet, but it likely starts with something here:

• https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/config/schemas/tunables
• https://github.com/microsoft/MLOS/blob/main/mlos_bench/mlos_bench/tunables/tunable_groups.py
• https://github.com/microsoft/MLOS/blob/main/mlos_bench/mlos_bench/optimizers/convert_configspace.py

from mlos.

@motus any other thoughts?

from mlos.

Related:
https://microsoft.github.io/FLAML/docs/Use-Cases/Tune-User-Defined-Function/#more-constraints-on-the-tuning

from mlos.

FWIW: automl/ConfigSpace#352

from mlos.

Hi @poojanilangekar, I am maintaining ConfigSpace and hopefully we can get towards making this possible :)

from mlos.