Enable using validation dataset from gridSearchCv in Callbacks. about skorch HOT 4 CLOSED

In general, callbacks are used for training only. The validation step in grid search is inference (it calls predict or predict_proba), therefore, callbacks are not involved. If you could more explain in more details what you would like to achieve, I might be able to suggest a solution.

from skorch.

Hi Benjamin Bossan, Thanks for the feedback! In my case, I want to use EarlyStopping( where valid-set is required )when train the net while I am using randomizedSearchCV to tune params of the net. I am considering use the valid-set from cross-validation in randomizedSearchCV for EarlyStopping rather than doing a train_split again in skorch. Please let me know if it's not clear explained. Thanks!

from skorch.

Yes, I understand your intent. What you describe is not possible with callbacks. It is also not recommended. By using the validation data from grid search for early stopping, you actually leak information from the validation data into the training process. This could lead to overfitting. Therefore, it is best to use the train_split from skorch: it is based on the training data and therefore there is no leakage.

from skorch.

I see. Thanks for pointing it out! I will close it then.

from skorch.