Comments (4)
Hi @thchang, you are indeed correct that this is not the proper way to calculate hypervolume, and instead it should work on the boundaries of metrics, not mins or maxes. Therefore it requires knowing the metrics used and their boundaries in the hypervolume
calculation function.
@NeoChaos12 are you still up for maintaining this?
CC: @DaStoll
from jahs_bench_201.
Thanks for the reply!
In case it helps, for now I've been using the following as my reference point to get started:
val_acc_min = 0.0
latency_max = 10.0
but I've only tried this with the CIFAR 10 benchmark, and haven't investigated the others yet
from jahs_bench_201.
Hi @thchang,
Thank you for bringing this to our attention! This sounds reasonable, though I would like to hear @worstseed's opinion on this issue.
@eddiebergman I may not be able to step back into an active maintainer role for another while, maybe a couple of weeks, though I do still have a long to-do list for the repo.
Best,
Archit
from jahs_bench_201.
Hello @thchang,
I want to start off by apologizing for the delay in responding to your questions. We value your interest and feedback, and we're sorry for any inconvenience caused by the delay.
Your insights regarding the reference point for the hypervolume calculation in our benchmark are quite perceptive. Currently, we are using the minimum of all observed values, which is a common practice when there isn't an explicit worst-case scenario. However, your point about the potential for bias is well-taken. Ideally, we would indeed use worst-case scenarios from the benchmark as reference points. This approach would prevent methods that sample poor configurations from potentially inflating their hypervolume scores. We are grateful for your suggestion and will seriously consider revising our approach to provide a more accurate comparison of different solvers.
Regarding your second point about the objectives in the multi-objective benchmark, you are correct. The objectives are to maximize validation accuracy and minimize latency. We acknowledge that this could have been clearer in our evaluation protocol. Thank you for bringing this to our attention; we'll make sure to clarify this in our documentation.
We appreciate your valuable feedback and your interest in our work. Please do not hesitate to reach out if you have any further questions or suggestions.
Best regards,
Maciej
from jahs_bench_201.
Related Issues (14)
- Incompatible checksums error HOT 8
- pip install error HOT 1
- Ambiguity in the query HOT 3
- Termination in colorectal_histology due to memory overflow HOT 3
- Bug in argument check (metrics) HOT 1
- Wrong path when loading tabular data HOT 2
- Moving away from pickles
- Provide a fixed tag version once stable HOT 2
- Data download
- Add a `py.typed` to export type information
- Export the ability to create the `joint_config_space` as a function HOT 1
- Issue with config and trajectory HOT 1
- XGBoost and MacOS incompatibility HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jahs_bench_201.