Comments (9)
Penalty parameters generally range from 1e-1 to 1e-6, and yes it's super arbitrary.
If you are looking at automatically performing these fits in a more robust manner, check out this post #17 (comment) where I look for a variance ratio. You probably need at least 20 data points for that variance ratio to work. I think this is a very novel way to automatically fit these models (and I really need to write a paper on this).
So the Bayesian optimization is trying to minimize the sum of square of residuals (mypwlf.ssr
) while penalizing the model complexity (number of line segments). As the number of line segments goes to infinity, the sum of square of residuals goes to zero. Also as the number of line segments goes to infinity, the penalty on model complexity also should go to infinity. It's a dance with the devil.
I would just try lambdas = [1e-1, 1e-2, 1e-3, 1e-4, 1e-5, 1e-6]
and see which one gives you the best visual fit. Do this for a couple cases in your data set, and then just fix the penalty parameter to that value.
from piecewise_linear_fit_py.
Thanks Charles,
Knowing the range helps, not that it will help my problem. I managed to run the model with 1e-1, but I fear my set is too noisy to benefit from smaller values. I have daily nighttime light intensities for one location and am trying to fit the piecewise linear function through the data, but the variance is very high.
The function with fixed lines (see below) runs fine up to 4 lines, but introducing more lines is increasing the run time exponential, and setting the maximum elements to 20 takes 1.5hr on my laptop in a Jupyter Notebook.
from piecewise_linear_fit_py.
I used 12 line elements because this is the first point in the optimised graph, using the suggested 19 doesn't make a visual difference, and the optimising values are very similar (if not identical)
from piecewise_linear_fit_py.
The variance is very high in your case, and you may benefit from trying this #17 (comment) but replace x
and y
with your own data. It should be biased to use very few line segments. (it should also run much faster than the Bayesian optimization routine).
from piecewise_linear_fit_py.
Thanks Charles,
I will check the issue, and compare the results.
One other thing I am going to try is to smooth my data with a 7-day moving average, this will remove most of the noise in the data. I tried this averaging already, to get the data stationary, and the 7 day period gives the best results (clear trend).
The reason I am trying your method is to have the piece-wise linear lines to check if there is a trend change after a known date. I can use the (known) date, but that won't prove that there is a trend change, that will (in my understanding) only show a different trend.
from piecewise_linear_fit_py.
This is a view on the smoothed data:
The event date is at the beginning of August, but what I was expecting is not the decline before the event; it should have been much more abrupt (in theory), so there is something else happening before the event (likely the COVID-19 confinements are interfering with the NightTime Light intensities in the area)
Best,
Gijs
from piecewise_linear_fit_py.
Hi Charles,
Quick update, your method #17 is giving some promising results. The method suggests 2 lines, but I think 3 segments are telling the story better.
from piecewise_linear_fit_py.
What was the F ratio for both cases?
F = sigma_hat / sigma
Maybe it's better to pick the one that is closest to 1.0, since one over and the other is under.
from piecewise_linear_fit_py.
Hi Charles,
Did you check my pull request? Please let me know, thanks.
from piecewise_linear_fit_py.
Related Issues (20)
- .fit() fails with 1 segment HOT 5
- How to force the fit process to have a fixed Intercept? HOT 1
- Can I get y_values if I have only x and slopes values? HOT 1
- Limit the slope of each segment of the curve HOT 1
- Re-constructing Piecewise PWLF HOT 2
- Set Slope of Segment to 0 HOT 2
- How to fit multiple functions simultaneously HOT 3
- Hi, i want to make sure that there are no fitted lines between points that are too far apart i.e. set a min value( fragment optimization) how to achieve this? HOT 1
- Why last beta is always positive? HOT 3
- How to prevent poor fitting HOT 4
- Forcing the segments to be integers and not float
- p values does not seem accurate HOT 3
- Error for coefficients of linear equations HOT 7
- assure the slopes to be lower and lower HOT 1
- divide by zero error in calc slopes if two break points are the same, or if a breakpoint is on the boundary HOT 2
- Issue using .fit() HOT 5
- How to plot segments with fit_breaks information HOT 4
- support random seed on init
- How to calculate prediction intervals? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from piecewise_linear_fit_py.