Comments (11)
Thanks for pointing this out @aschl. We are currently re-working quite substantially the backtesting functionality, and indeed our plan is to introduce a stride
parameter like you suggest. We will also introduce moving windows for the training set (instead of expending only, as now). These should come in a release sometime soon.
from darts.
Hello @aschl, while we don't have a date yet for the release of this feature. If this something you would like to try as soon as possible the stride parameter has already been added to the backtest_forecasting
method on the develop branch. Keep in mind that we are currently working on a quite heavy refactor of the whole backtesting.py
file and the API for backtesting might change in the upcoming release.
from darts.
I checked backtesting.py
in the develop
branch. If I understand it correctly, the stride
parameters determines the distance between two consecutive predictions. This might be helpful if e.g. you only want to predict January values (and skip Feb-Dec forecast). I think there are quite a few use cases when you need all prediction points but would like to train the model only on every x
time steps. This would increase efficiency quite a lot without necessarily loosing predictive power (adding a few more samples to a series of few hundert observations doesn't substantially improve the model if the distribution remains similar).
The idea behind scikitlearn's TimeSeriesSplit()
is exactly this. You split the time series in k
cross-validation pieces and retrain the model only k
times. This way you obtain a relatively unbiased estimate of the forecasting error with reasonable computational efforts.
(I see that you implemented a retrain
argument, which goes into this direction. This would however only allow to either retrain for each prediction or only once with the first sample.)
from darts.
Yes. Exactly.
I am currently busy with some other projects. Not sure when I will find time to come back to this. Please keep me updated.
from darts.
Can I take at this one up if this is still open?
from darts.
You're right @deltamacht. Gridsearch is still pretty basic... There's however a stride
parameter in gridsearch()
which maybe goes some way into doing what you're looking for.
from darts.
Awesome! Looking forward to seeing this feature. The idea of incorporating the moving window is also great and very useful.
When do you approximately plan to release this?
from darts.
Hi @aschl and sorry for the delay answering your last comment. If I understand correctly, your suggestion would be to replace retrain:bool=True
with something like retrain_every_n:int = 1
, am I correct ?
If that's the case, I think you make a good point. Feel free to contribute if you can! Otherwise I'll put it on our radar.
from darts.
@pravarmahajan sorry for the late reply. We would be very happy to receive a PR to introduce a "retrain every N steps" feature to historical forecasting / backtesting!
from darts.
Just looping back to this. As I understand, #1139 addressed the concern on retraining every n steps in the retrain
behavior in backtest()
, but this parameter isn't exposed in the gridsearch method. With regards to the discussion above about having some behavior that would be similar to Sklearn's TimeSeriesSplit
, am I correct in thinking that this type of cross validation isn't easily specified in the gridsearch
method at present?
from darts.
Implemented in v0.22.0 🚀
from darts.
Related Issues (20)
- Allow broadcasting for element-wise operations on TimeSeries HOT 1
- [BUG] Cannot convert a MPS Tensor to float64 dtype as the MPS framework - GlobalNaiveAggregate HOT 4
- Adding Future Covariates to the BlockRNN
- [BUG] DOC `series_reduction` explained twice in docstring HOT 2
- How to obtain quantiles from a probabilistic `TimeSeries`? HOT 1
- [Question] Why do Global Baseline Models require fitting before prediction? HOT 2
- [Question] How to perform rolling predictions? HOT 3
- Mixed precision support for Torch Models
- `TFTModel.predict()` with `predict_likelihood_parameters=True` HOT 2
- Update examples
- Shap explanation for recursive multi-step forecasting HOT 4
- TFTExplainer: getting explanations when trained on multiple time series HOT 3
- Missing attribute when loading models trained in older darts version
- [BUG] TimeSeries.from_group_dataframe incompatible with integer timestamps
- [BUG] No activations in BlockRNNModel output MLP
- How to implement TIDE Model with similar lags to LGBM HOT 3
- [BUG] LinearRegression get_estimator method's required arguments appear to do nothing HOT 3
- [BUG] sklearn KNeighborsRegressor can't be fitted with darts RegressionModel HOT 2
- TiDE Model Stops At A Specific Epoch HOT 2
- Backtest with custom forecast dates HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from darts.