Larger max runtime error about auto-pytorch HOT 22 CLOSED

Great that you could resolve your issue.

"On the sidehow do I avoid Exception: Budget exhausted before training started"

Increase the minimum budget or switch to budget type epochs.

Let me explain why this exception occurs:

When you use time as a budget, we also include the time for preprocessing. If preprocessing your data takes more time than the current budget, the budget will be exhausted before the training started.

Furthermore, we reduce the budget by 10 seconds to compensate for training wrap-up (computing validation performance, etc.). If your current budget is 60 seconds for example and preprocessing takes 50 seconds, this exception will be raised.

If you use cross validation the budget will be divided among the splits. If the wrap-up of your first split takes less than 10 seconds, we will give the other splits more time.

We evaluate each network in a subprocess which we kill after the budget (+ some grace time) is exhausted. The compensation time explained above is necessary to avoid that a subprocess will be killed because the wrap-up took a bit too long. The compensation time is that high because we are working on supporting image data, where the wrap-up might take longer.

Please tell us if you have suggestions to improve this approach.

from auto-pytorch.

Hi,

Thank you for your interest in Auto-PyTorch.
Could you please provide a mini example to reproduce the error?

Have you tried to increase the max_runtime as stated in the error message?

Best,
Marius

from auto-pytorch.

Hi,

do you call fit() with numpy arrays for the arguments X_train/Y_train/X_valid/Y_valid?

It seems like you call fit() with lists which might cause the issue.

Cheers,

Matthias

from auto-pytorch.

Thank you @mlindauer. I did increase the max_runtime from 1800 to 2700, but still had issues.

You are right @urbanmatthias. The problem was the model was first fitting lists instead of numpy array, which I fixed to some extent I guess.

Is it possible to make auto-pytorch work with lists as well?
I am excited about the whole auto-pytorch idea, but these issues with list is weighing down my interests and excitement.

Sincerely,
Enoch

from auto-pytorch.

Have you tried to convert your lists to numpy arrays before calling fit?

We are currently not checking the types of the arguments you provide to fit (we will in the future).

fit() is expecting numpy (or scipy) arrays, so the best way to use it with lists is to simply convert the lists to numpy arrays yourself, before calling fit.

Nevertheless, a mini-example would help us a lot to diagnose your issue.

from auto-pytorch.

Yes, I have fixed the errors. Thank you so much for @urbanmatthias and @mlindauer.

"We are currently not checking the types of the arguments you provide to fit (we will in the future)."
Yes, that is what I was suggesting. It will be really helpful and save lot of time debugging, when you have so many lines of code or script.

P.S.
On the sidehow do I avoid Exception: Budget exhausted before training started

from auto-pytorch.

Thank you for the feedback. I am working with a large text dataset and preprocessing and feature extraction take s a lot some time before training starts.
I suggest the budget should not include the time for preprocessing. I am not sure if this would have a consequence of something else though. Just a straight forward suggestion.

Thanks for all the assistance so far.

from auto-pytorch.

Thank you for your suggestion. We will include an alternative budget type that will only include training time.

from auto-pytorch.

Thank you too, and I will really appreciate that. I tried the epoch budget type and it worked without exhaustion problem, like you said.

from auto-pytorch.

The budget type "training_time" should now be available after the last commit.

from auto-pytorch.

Hello all,

Yesterday I began using AutoNetMultilabel for a multilabel task problem I have, but no matter the runtime type or max_runtime I use, I always get :
RuntimeError: No models fit during training, please retry with a larger max_runtime.

Am I doing something wrong, or the issue is with the algorithm?
Also, there is no documentation on AutoNetMultilabel to serve as a guy. I'm counting on your usual co-operation. Thank you.

Best,
Enoch.

from auto-pytorch.

Hi,

if you not models are fitted during training, although your max runtime is high enough, something else must have gone wrong.

Maybe all the training runs crashed? This would obviously also result in no fitted models.

Can you give us the full output and ideally a small example to reproduce the error?

We are working on improving the documentation. The API of multilabel classification should be identical to multiclass classification.

Hope this helps.

Cheers,

Matthias

from auto-pytorch.

Thank you for the feed.
I'm using a codebase, but I think the Issue was as a result of an overflow... some zeros where fed to a log function.

I will update you, asap when there's a new development.

Best,
Enoch.

P.S. Really want to cite this library

from auto-pytorch.

Hello, I am still getting an error using AutoNetMultilabel
I just pass similar parameters as done in AutoNetClassification, but I first get
"Warning: hexagon layer count does not match 3 != 4" and subsequently
"Error parsing results. Check results.json and output for more details. An empty results.json is usually caused by a misconfiguration of AutoNet."

RuntimeError Traceback (most recent call last)
in
8 log_level="debug",
9 loss_modules=['bce_with_logits', 'bce_with_logits_weighted'],
---> 10 early_stopping_patience=3)

~/miniconda3/envs/google-crash/lib/python3.6/site-packages/autoPyTorch/core/api.py in fit(self, X_train, Y_train, X_valid, Y_valid, refit, **autonet_config)
140
141 if not self.fit_result["optimized_hyperparameter_config"]:
--> 142 raise RuntimeError("No models fit during training, please retry with a larger max_runtime.")
143
144 if (refit):

RuntimeError: No models fit during training, please retry with a larger max_runtime.

from auto-pytorch.

Hello @mlindauer and urbanmatthias, please I really need your help on this.
I keep getting the following warnings when running the AutonetClassification and also in the Regression and Multlabel:

RuntimeWarning: overflow encountered in multiply in sklearn.
x = um.multiply(x, x, out=x)

RuntimeWarning: overflow encountered in power in sklearn.
out[~pos] = -(np.power(-x[~pos] + 1, 2 - lmbda) - 1) / (2 - lmbda)

RuntimeWarning: overflow encountered in double scalers in scipy.optimize

Someone raised that issue some time ago here, but I can't find a fix for me.
I have been struggling with this for weeks now and I really need your help. My school project depends on this.

from auto-pytorch.

@LMZimmer could you please look into it

@etetteh could you please provide a minimal example such that we can reproduce the issue locally.

from auto-pytorch.

Thank you, but I think reproducing the issue locally may not be possible because I am using a dataset, which is not publicly available, MIMIC-III (you only get acces after an MIT course) and the code base here
The only thing is instead of the logistic regression I am using AutoNet where appropriate (that is either regression or classification or Multilabel)
Please, I have sent you a mail.

from auto-pytorch.

Please, let me know if you need any help in reproducing the results and I would gladly assist.
Counting on your usual cooperation

from auto-pytorch.

Hello @etetteh,

I have to reproduce your error, could you provide some more information? In particular:

1. What is the type and shape of your dataset? Can you reproduce the error with a handcrafted dataset of similar type (e.g. simply with zeros)?

2. Which config are you using for AutoPyTorch?

from auto-pytorch.

The dataset are csv files with each row being a csv file. There are functions created to read the data, and are contained in the link to the code base I sent. It's not easy to handcraft a similar dataset. I can give more guidance to this through my email etetteh[at]aimsammi[dot]org, because of the dataset. @mlindauer has a copy of the data.

This is the autopytorch config:

auto_net = AutoNetClassification(budget_type='epochs', min_budget=3, max_budget=30, num_iterations=3, log_level='debug')

auto_net.fit( optimize_metric='auc_metric',
cross_validator='stratified_k_fold',
early_stopping_patience=3,
loss_modules=['cross_entropy', 'cross_entropy_weighted'],
log_level="debug",
X_train=train_X,
Y_train=train_y,
X_valid=val_X,
Y_valid=val_y,
cuda=True
)

I am replacing the logistic regression used here with the autnet defined above.

The script to run the code is this:
python -um mimic3models.in_hospital_mortality.logistic.main --output_dir mimic3models/in_hospital_mortality/logistic

from auto-pytorch.

Thank you so much @LMZimmer. You've been very helpful. I will now proceed to close this issue.

from auto-pytorch.

I am also using autopytorch0.0.2 and have the same problem
RuntimeError: No models fit during training, please retry with a larger max_runtime.
Do you remember how it was resolved?
thank you

from auto-pytorch.