Giter Club home page Giter Club logo

Comments (5)

Neeratyoy avatar Neeratyoy commented on June 16, 2024 1

Thanks for bringing the issue to notice!

It has been updated and the PyPi v0.0.4 should have this change included.

from dehb.

Neeratyoy avatar Neeratyoy commented on June 16, 2024

Hi @drpriscu

To simply compare DEHB and DE, if you run DEHB with deferred then you should compare it with DE. If you run DEHB with immediate you should compare it with AsyncDe with immediate updates.

For reproducing experiment results from the paper, the experiments branch is the closest code version.

Closing this for now, as no code issues have been detected yet. Please feel free to reopen with the code snippet that needs to be looked into.
Thanks!

from dehb.

drpriscu avatar drpriscu commented on June 16, 2024

Thank you for your response. I would like to run DEHB and DE with immediate, as according to the paper, DE was outperformed by Random Search because it used a deferred strategy, whereas DEHB performed better because it uses an immediate update design for DE:

"DEHB uses the immediate update design for DE, wherein it updates the population immediately after a DE selection, and not wait for the entire population to evolve. We posit that this feature, along with lower fidelity search, and performing grid search over population sizes with Hyperband, enables DEHB to be more practical than classical-DE.".

According to the experiments branch, the DEHB class stores the async_strategy with the value immediate in de_params, which is sent to the AsyncDE class. However, the code issue in the pip version is that it never stores the async_strategy in the DEHB class, so when the DEHB class calls AsyncDE and passes the de_params, AsyncDE always implements the default async_strategy value of deferred.

Could you please update DEHB to implement the async strategy value passed so that AsyncDE uses the desired value?

dehb.py: lines 41-57

# DE related variables
self.mutation_factor = mutation_factor
self.crossover_prob = crossover_prob
self.strategy = strategy
self.fix_type = boundary_fix_type
self.max_age = max_age
self.de_params = {
    "mutation_factor": self.mutation_factor,
    "crossover_prob": self.crossover_prob,
    "strategy": self.strategy,
    "configspace": self.configspace,
    "boundary_fix_type": self.fix_type,
    "max_age": self.max_age,
    "cs": self.cs,
    "dimensions": self.dimensions,
    "f": f
}

line 357:

self.de[b] = AsyncDE(**self.de_params, budget=b, pop_size=self._max_pop_size[b])

de.py: lines 548 - 551

class AsyncDE(DE):
    def __init__(self, cs=None, f=None, dimensions=None, pop_size=None, max_age=np.inf,
                 mutation_factor=None, crossover_prob=None, strategy='rand1_bin',
                 budget=None, async_strategy='deferred', **kwargs):                 

from dehb.

Neeratyoy avatar Neeratyoy commented on June 16, 2024

Hi @drpriscu

This issue is open again. Please feel free to link the issue and post a PR.
I can take a look and merge it to master if all looks good.

Thanks!

from dehb.

drpriscu avatar drpriscu commented on June 16, 2024
class DEHBBase:
    def __init__(self, cs=None, f=None, dimensions=None, mutation_factor=None,
                 crossover_prob=None, strategy=None, min_budget=None,
                 max_budget=None, eta=None, min_clip=None, max_clip=None,
                 boundary_fix_type='random', max_age=np.inf, async_strategy='deferred', **kwargs):
        # Benchmark related variables
        self.cs = cs
        self.configspace = True if isinstance(self.cs, ConfigSpace.ConfigurationSpace) else False
        if self.configspace:
            self.dimensions = len(self.cs.get_hyperparameters())
        elif dimensions is None or not isinstance(dimensions, (int, np.int)):
            assert "Need to specify `dimensions` as an int when `cs` is not available/specified!"
        else:
            self.dimensions = dimensions
        self.f = f

        # DE related variables
        self.mutation_factor = mutation_factor
        self.crossover_prob = crossover_prob
        self.strategy = strategy
        self.fix_type = boundary_fix_type
        self.max_age = max_age
        self.async_strategy = async_strategy
        self.de_params = {
            "mutation_factor": self.mutation_factor,
            "crossover_prob": self.crossover_prob,
            "strategy": self.strategy,
            "configspace": self.configspace,
            "boundary_fix_type": self.fix_type,
            "max_age": self.max_age,
            "cs": self.cs,
            "dimensions": self.dimensions,
            "f": f,
            "async_strategy": self.async_strategy
        }
class DEHB(DEHBBase):
    def __init__(self, cs=None, f=None, dimensions=None, mutation_factor=0.5,
                 crossover_prob=0.5, strategy='rand1_bin', min_budget=None,
                 max_budget=None, eta=3, min_clip=None, max_clip=None, configspace=True,
                 boundary_fix_type='random', max_age=np.inf, n_workers=None, client=None, async_strategy='deferred', **kwargs):
        super().__init__(cs=cs, f=f, dimensions=dimensions, mutation_factor=mutation_factor,
                         crossover_prob=crossover_prob, strategy=strategy, min_budget=min_budget,
                         max_budget=max_budget, eta=eta, min_clip=min_clip, max_clip=max_clip,
                         configspace=configspace, boundary_fix_type=boundary_fix_type,
                         max_age=max_age, async_strategy=async_strategy, **kwargs)

from dehb.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.