Comments (11)
Hello again, @quanvuong!
-
You can use
render(ax_client.get_optimization_trace())
to see the objective values so far (API reference). We will add parameter values on hover to this plot, to make it easier to understand what the most promising parameters so far are. The resulting plot currently will look something like this:
. -
Length scale / sensitivity analysis –– in the works, thank you for bringing it up!
-
Extracting visualization plot –– we will look into a more easily serializable way of providing the plots, thanks for the feedback!
-
Will add parameter values on hover to plot shown in 1), would that, in combination with the contour plot, be sufficient for the purpose of detecting whether the search space needs adjustment?
from ax.
Hello, @quanvuong! Just for context, which one of our APIs are you using and would want to see this functionality implemented in?
from ax.
Hi!
I'm using the Service API. The Service API allows me to manually logs the hyper-parameter values and their corresponding function evaluation values. But it would be nice if this feature was already implemented.
from ax.
@quanvuong, just so that I understand, does the following code not achieve this purpose? What should we add?
from ax.service.ax_client import AxClient
from ax.utils.measurement.synthetic_functions import branin
from ax.utils.notebook.plotting import render
ax = AxClient()
ax.create_experiment(
parameters=[
{"name": "x", "type": "range", "bounds": [-5.0, 10.0]},
{"name": "y", "type": "range", "bounds": [0.0, 15.0]}
]
)
for _ in range(6):
params, idx = ax.get_next_trial()
ax.complete_trial(idx, branin(*(params.values())))
ax.get_best_parameters()
render(ax.get_contour_plot())
Just for ease of reference, this is what it looks like executed in a notebook:
from ax.
Thanks!
For example:
-
The contour plot does not show the function evaluation values, the corresponding hyper-parameter in a ranked manner. This would be useful during optimization to get a sense of what is the most promising hyper-parameter values so far.
-
Length Scale or some sort of sensitivity analysis. Before running BayesOpt, I might not know what the important hyper-parameters are and what hyper-parameters the performance is relatively insensitive too. It would be super useful to show how sensitive the performance is with respect to each hyper-parameter. This would inform subsequent experiments and understanding.
-
I might run BayesOpt inside a docker image or on a server. So it's not easy to extract out visualization plot. In this case, textual logs would be useful as well.
from ax.
Another feature that would be helpful is:
- Viewing the hyper-parameter values of the best function evaluations so far. This helps to see if the optimized hyper-parameter values hit their respective bounds, meaning that the bounds for some of the hyper-parameters are too tight. This usually means that better hyper-parameter values can be found by loosening the bounds the hyper-parameter values are searched over.
from ax.
yes, that would be great. Thanks!
from ax.
Parameter values on-hover are now part of the optimization trace plot for the Service API and more textual logs are now enabled in the latest stable version, 0.1.6. Examples in the tutorial: https://ax.dev/tutorials/gpei_hartmann_service.html.
Sensitivity analysis plot and extraction of plots in serializable way are still in the works.
from ax.
We have a pending change to the service API:
render(ax_client.get_feature_importances())
Will show a ranked bar chart of feature importances for different metrics.
from ax.
Also, @quanvuong, while we consider a slightly more elegant option you can serialize and visualize plots as follows:
from ax.
I believe this should address all the issues brought up in this task: textual logs and parameter values on hover (included in the latest stable release), sensitivity analysis plots (included on master), and a workaround for serializing the plots. Closing the issue, but feel free to follow up, @quanvuong!
from ax.
Related Issues (20)
- [Bug] Generation Strategy equality check error without call to repr HOT 1
- ax.dev, GitHub readme, and loop tutorial examples are ignoring `minimize` kwarg HOT 2
- Get Started code gives the same result regardless of random seed HOT 3
- Trouble with searching documentation (Algolia, API docs, etc.), for example `get_next_trials` HOT 1
- Issue when starting an AxClient with out-of-design points HOT 2
- cannot import name 'TrainingData' HOT 2
- applying complex constrains HOT 2
- Ax is not not starting as many workers as I'd like to; sometimes, get_next_trials returns 0 new trials HOT 5
- Evaluating custom candidates HOT 2
- Input Feature Selection - Does the relevant code exist? HOT 6
- [Feature Request] support constraints on `ChoiceParameters` HOT 4
- Extending Models.THOMPSON with an extra parameter HOT 1
- There are some questions when i use the Ax HOT 7
- Space characters in the objective name AND specifying a threshold leads to an error message: "AssertionError: Outcome constraint should be of form `metric_name >= x" HOT 2
- Pandas deprecation warning when deserializing AxClient JSON HOT 2
- AX seems to get stuck with Ray
- `StandardizeY` transform requires non-empty data." when using SAASBO
- Plotting outside of a notebook HOT 1
- Setting search space step size in Ax Service API HOT 10
- Problem when Sobol falls back to HitAndRunPolytopeSampler HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ax.