Comments (3)
Hey,
there are three different scorings:
i) Training loss (e.g. cross_entropy
) that is used for the optimizer when training an NN
ii) Metrics for scoring (e.g. accuracy_score
), these are logged in results.json under their name
iii) The loss considered by BOHB. This is the main metric (could be accuracy_score
) transformed to a metric that is minimizable, that is 1-accuracy_score
. This loss is logged as loss
.
You can change ii) via the optimize_metric
argument in the autonet config.
from auto-pytorch.
I wrote a code with new optimization metric
def qloss(y_true, y_pred, qs):
e = y_true - y_pred
return np.maximum(qs * e, (qs-1)*e).mean()
metric_ql = functools.partial(qloss, qs = np.array([[0.1, 0.5, 0.9]]))
def qinputloss(y_true, y_pred, ws):
w_q = ws[0,0]
w_input = ws[0:1, 1:]
q = y_true[:, 0:1]
e_q = np.abs(y_true[:,0]-y_pred[:,0])
e = y_true[:, 1:] - y_pred[:, 1:]
#print(w_input)
#print(q*e, (q-1)*e)
ret = w_input * np.maximum(q*e, (q-1)*e)
#print(ret)
return np.mean(ret)+ np.mean(w_q*e_q)
metric_qil = functools.partial(qinputloss, ws=np.array([[0,1]]))
metric_selector = autonet.pipeline[MetricSelector.get_name()]
metric_selector.add_metric('metric_ql', metric_ql, loss_transform=False, requires_target_class_labels=False)
metric_selector.add_metric('metric_qil', metric_qil, loss_transform=False, requires_target_class_labels=False)
results_fit = autonet.fit(X_train=train_x_wq,
Y_train=train_y_wq,
loss_modules=["qinput_loss"],
optimize_metric = 'metric_qil')
I have not test it fully, but it looks working.
At first, I tried closure type function generation like the below
def QLossMetric(qs):
if len(qs.shape) < 1:
raise ValueError('qs should be 1-dim or 2-dim array')
elif len(qs.shape) == 1:
qs = qs.reshape(1,-1)
elif len(qs.shape) > 2:
raise ValueError('qs should be 1-dim or 2-dim array')
def qloss(y_true, y_pred):
e = y_true - y_pred
return np.maximum(qs * e, (qs-1)*e).mean()
return qloss
but somehow it did not seem to work properly for QuantileLoss
or QinputLoss
. I do know that function's default arguments sould not be mutable, and it seemed to be somewhat related to that but I could not quite grasp the reason so I changed to something like the above.
(I am rather an R-expert than python expert... so...)
Any suggestion is welcome here.
from auto-pytorch.
Looking at your code, it looks like you are adding your metrics to the metric selector (which is used to choose configurations with BOHB) but also add loss_modules
(which is the module used as training loss).
If you want to just use the metric, you should omit selecting the loss module (which will use cross_entropy per default). If you actually want to use a different training loss, you have to add it to the respective node.
from auto-pytorch.
Related Issues (20)
- [Time series Forecasting] Continuous Ranked Probablity Score (CRPS) loss for probablity network ouput type HOT 2
- Improve `include_components` documentation HOT 1
- Cannot run time-series example on GPU HOT 1
- Score function error on example code HOT 2
- Prediction on one sample produces error HOT 1
- The predict method of the base task expects to build a logger from a temporary folder
- [Feature request] time series classification HOT 2
- Question about how to maximize the search space HOT 1
- AutoPytorch selects only Dummy model. HOT 4
- Generate a leaderboard for each model?
- ImportError: cannot import name 'AutoNetClassification' from 'autoPyTorch' HOT 1
- I ran autopytrch 0.2 on multiple datasets every time it picks the Dummy model HOT 1
- Error in using command "from autoPyTorch.api.time_series_forecasting import TimeSeriesForecastingTask" HOT 2
- TypeError occurs when import TabularRegressionTask
- Image Classification: API not finished
- Can't install on Google Colab HOT 4
- Errors on running example_tabular_classification.py HOT 1
- Errors installing autoPyTorch HOT 1
- [ERROR] [Client-AutoPyTorch:RefitLogger:1] Prediction for lgb failed with run state StatusType.TIMEOUT.
- Multi-GPU support
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from auto-pytorch.