Comments (5)
Thanks for bringing the issue to notice!
It has been updated and the PyPi v0.0.4 should have this change included.
from dehb.
Hi @drpriscu
To simply compare DEHB and DE, if you run DEHB with deferred then you should compare it with DE. If you run DEHB with immediate you should compare it with AsyncDe with immediate updates.
For reproducing experiment results from the paper, the experiments
branch is the closest code version.
Closing this for now, as no code issues have been detected yet. Please feel free to reopen with the code snippet that needs to be looked into.
Thanks!
from dehb.
Thank you for your response. I would like to run DEHB and DE with immediate, as according to the paper, DE was outperformed by Random Search because it used a deferred strategy, whereas DEHB performed better because it uses an immediate update design for DE:
"DEHB uses the immediate update design for DE, wherein it updates the population immediately after a DE selection, and not wait for the entire population to evolve. We posit that this feature, along with lower fidelity search, and performing grid search over population sizes with Hyperband, enables DEHB to be more practical than classical-DE.".
According to the experiments branch, the DEHB class stores the async_strategy with the value immediate in de_params, which is sent to the AsyncDE class. However, the code issue in the pip version is that it never stores the async_strategy in the DEHB class, so when the DEHB class calls AsyncDE and passes the de_params, AsyncDE always implements the default async_strategy value of deferred.
Could you please update DEHB to implement the async strategy value passed so that AsyncDE uses the desired value?
dehb.py: lines 41-57
# DE related variables
self.mutation_factor = mutation_factor
self.crossover_prob = crossover_prob
self.strategy = strategy
self.fix_type = boundary_fix_type
self.max_age = max_age
self.de_params = {
"mutation_factor": self.mutation_factor,
"crossover_prob": self.crossover_prob,
"strategy": self.strategy,
"configspace": self.configspace,
"boundary_fix_type": self.fix_type,
"max_age": self.max_age,
"cs": self.cs,
"dimensions": self.dimensions,
"f": f
}
line 357:
self.de[b] = AsyncDE(**self.de_params, budget=b, pop_size=self._max_pop_size[b])
de.py: lines 548 - 551
class AsyncDE(DE):
def __init__(self, cs=None, f=None, dimensions=None, pop_size=None, max_age=np.inf,
mutation_factor=None, crossover_prob=None, strategy='rand1_bin',
budget=None, async_strategy='deferred', **kwargs):
from dehb.
Hi @drpriscu
This issue is open again. Please feel free to link the issue and post a PR.
I can take a look and merge it to master if all looks good.
Thanks!
from dehb.
class DEHBBase:
def __init__(self, cs=None, f=None, dimensions=None, mutation_factor=None,
crossover_prob=None, strategy=None, min_budget=None,
max_budget=None, eta=None, min_clip=None, max_clip=None,
boundary_fix_type='random', max_age=np.inf, async_strategy='deferred', **kwargs):
# Benchmark related variables
self.cs = cs
self.configspace = True if isinstance(self.cs, ConfigSpace.ConfigurationSpace) else False
if self.configspace:
self.dimensions = len(self.cs.get_hyperparameters())
elif dimensions is None or not isinstance(dimensions, (int, np.int)):
assert "Need to specify `dimensions` as an int when `cs` is not available/specified!"
else:
self.dimensions = dimensions
self.f = f
# DE related variables
self.mutation_factor = mutation_factor
self.crossover_prob = crossover_prob
self.strategy = strategy
self.fix_type = boundary_fix_type
self.max_age = max_age
self.async_strategy = async_strategy
self.de_params = {
"mutation_factor": self.mutation_factor,
"crossover_prob": self.crossover_prob,
"strategy": self.strategy,
"configspace": self.configspace,
"boundary_fix_type": self.fix_type,
"max_age": self.max_age,
"cs": self.cs,
"dimensions": self.dimensions,
"f": f,
"async_strategy": self.async_strategy
}
class DEHB(DEHBBase):
def __init__(self, cs=None, f=None, dimensions=None, mutation_factor=0.5,
crossover_prob=0.5, strategy='rand1_bin', min_budget=None,
max_budget=None, eta=3, min_clip=None, max_clip=None, configspace=True,
boundary_fix_type='random', max_age=np.inf, n_workers=None, client=None, async_strategy='deferred', **kwargs):
super().__init__(cs=cs, f=f, dimensions=dimensions, mutation_factor=mutation_factor,
crossover_prob=crossover_prob, strategy=strategy, min_budget=min_budget,
max_budget=max_budget, eta=eta, min_clip=min_clip, max_clip=max_clip,
configspace=configspace, boundary_fix_type=boundary_fix_type,
max_age=max_age, async_strategy=async_strategy, **kwargs)
from dehb.
Related Issues (20)
- Checkpointing optimization run HOT 2
- Improve doc strings and add type annotations HOT 2
- Handling min_budget = max_budget situation HOT 4
- Continuously develop unit tests
- Keep active brackets in a dictionary HOT 1
- Implement ask-and-tell interface for DEHB HOT 1
- Use of deprecated np.int in dimension type checking HOT 3
- [Bug] `isinstance(self, Client)` will never be `True` HOT 6
- Configuration and budgets in each brackets HOT 6
- Using Dask ```Client``` as context manager
- No support for Constant type in vector_to_configspace() and configspace_to_vector() HOT 3
- Update CHANGELOG with the historic releases
- Updating and populating the documentation
- Implement versioning for documentation HOT 1
- Introduce IDs for evaluated configurations HOT 2
- Continuing configuration evolution when run budget is exhausted. HOT 2
- Implement parallel DE
- [Bug] Seeding is not applied to configspace HOT 3
- Is there any way to hide non-critical logs to stdout? HOT 8
- ERROR - Failed to communicate with scheduler during heartbeat. followed by TimeoutError: No valid workers found HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dehb.