Comments (4)
if the plan is to only use the dependency for CLasso then I'm not super highly in favour
fair enough --- although if eventually going that route, I will say I've had good experiences using Clarabel.jl
I'll start to play with fminbox a bit. with the gramian training ready, box constraints are definitely the next most important thing for me!
from mljlinearmodels.jl.
so let's say you wanted to do a linear regression with positive coefficients, you could either write that as something where you project on R+ at every step (would be easy to implement with the current code but likely not guaranteed to converge and might be slow) or start in R+ and add a penalty that explodes (e.g. logarithm) when things get close to 0. Extension to box is the same.
Note also that there is an LBFGS algorithm that supports box constraints but I don't think optim.jl implements it, it might implement something similar to it though (e.g. this pr: JuliaNLSolvers/Optim.jl#584, I haven't dug into that).
So maybe my suggestion would be to start with what optim has in store, see if it can be made to work, also see if there are existing "standard" implementations of regression with + constraints in python or other that could be used as a baseline.
Note: looks like fminbox implements a primal barrier that would amount to what I was suggesting in my first paragraph https://github.com/JuliaNLSolvers/Optim.jl/blob/5fa5d61a9f2ba9fc534b4e74b5659df45afd59c4/src/multivariate/solvers/constrained/fminbox.jl#L182-L206
from mljlinearmodels.jl.
is projecting to R+ at each step what sklearn does here ? https://github.com/scikit-learn/scikit-learn/blob/286f0c9d17019e52f532d63b5ace9f8e1beb5fe5/sklearn/linear_model/_cd_fast.pyx#L568C5-L568C33 it looks like it but I'm not entirely sure. I guess they are doing coordinate descent rather than gradient descent anyway
in particular I have this algorithm in mind for the constrained lasso https://arxiv.org/pdf/1611.01511.pdf
from mljlinearmodels.jl.
is projecting to R+ at each step what sklearn does here ? https://github.com/scikit-learn/scikit-learn/blob/286f0c9d17019e52f532d63b5ace9f8e1beb5fe5/sklearn/linear_model/_cd_fast.pyx#L568C5-L568C33 it looks like it but I'm not entirely sure. I guess they are doing coordinate descent rather than gradient descent anyway
oof that's not an easy one to read, could be used as an example of why people should move to Julia... yeah it's coordinate descent, I don't fully understand what they're doing in there so would rather not comment too much. Projected gradient descent is more like:
- take a normal GD step (for which we have code), or in fact any admissible step btw, just that GD makes sense and has a better chance to lead where you want
- project orthogonally onto R+ (should be something like vector minus a dot product or something similar)
- do that again
tI think this will be ok in simple cases but might be pretty bad in some cases (lots of non-admissible steps that get projected with the next step that looks very similar) and is not guaranteed to converge afaik. Might still be good to implement as a comparison.
in particular I have this algorithm in mind for the constrained lasso https://arxiv.org/pdf/1611.01511.pdf
right so just as a note, ADMM tends (in my experience) to be a pretty poor algorithm, hard to set up, and even when set up right it can be beaten pretty handily by simpler methods (but people thought it was cool because it can be somewhat parallelisable and lots of papers followed up on it however I don't think it's as good as it's famous).
QP might be interesting but typically needs a dedicated QP solver, from a quick search there's a few in Julia but it'll come at a cost of an an additional dependency which would need to be justified (e.g. if the plan is to only use the dependency for CLasso then I'm not super highly in favour).
To me it would be quite interesting to just try fminbox on the algorithms that are here, see if that works well, try to compare with standard-ish implementations in python / R to see if we have something that's as good or better and then move on from there; it might be that it leads to something that is vastly superior to the article you quote (which wouldn't surprise me).
from mljlinearmodels.jl.
Related Issues (20)
- Docs for L1+std HOT 1
- `LogisticClassifier` lambda and gamma are easy to confuse HOT 1
- Problems when only one class manifest in the training target HOT 5
- What is the new scale_penalty_with_samples=true doing? HOT 12
- TagBot not working HOT 7
- Ncvreg wrapper HOT 1
- `predict` throws an error for `MultinomialClassifier` on crabs dataset
- Unexpected behaviour with LogisticClassifier HOT 5
- Add MLJ compliant doc strings to all models
- Help with documentation review HOT 9
- Test proxgrad with gamma=0 HOT 1
- Add MLJTestInterface tests
- cov-only training for L2Loss HOT 3
- Testing quantreg vs MLJLinearModels HOT 7
- should scale_penalty_with_samples = true be defualt? HOT 4
- Expand docs to include cut and past examples for different Regressors HOT 3
- generalized / group lasso HOT 1
- Feature request: Support for tables HOT 3
- Huber breakpoint parameter in quantile form? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mljlinearmodels.jl.