Comments (4)
mrocklin
commented on June 8, 2024
I don't think that dask.dataframe has anything to do with the dask.learn project. Dask-learn strictly handles model parallelism on objects that satisfy the sklearn API. Does this answer your question @data-steve ?
from dask-searchcv.
jcrist
commented on June 8, 2024
Sorry for the lack of response here. You can pass any dask object (e.g. array/dataframe/delayed object) to the *SearchCV.fit method, but the enclosed estimator methods will only receive the "computed" version of that object. So if you pass in a dask.dataframe.DataFrame, your fit/transform methods will get a pandas dataframe. In general this library is for fitting many many models on small-medium data, so this isn't seen as a problem, as the benefit of using dask for data-parallelism in these cases is small.
from dask-searchcv.
data-steve
commented on June 8, 2024
what constitutes small to medium data for dask? In terms of rows or cols or both?
~ Steve
Sent via telepathy
… On Apr 5, 2017, at 5:43 PM, Jim Crist ***@***.***> wrote:
Sorry for the lack of response here. You can pass any dask object (e.g. array/dataframe/delayed object) to the *SearchCV.fit method, but the enclosed estimator methods will only receive the "computed" version of that object. So if you pass in a dask.dataframe.DataFrame, your fit/transform methods will get a pandas dataframe. In general this library is for fitting many many models on small-medium data, so this isn't seen as a problem, as the benefit of using dask for data-parallelism in these cases is small.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
from dask-searchcv.
jcrist
commented on June 8, 2024
I guess what I mean here is "anything you'd use scikit-learn in memory with". We don't do anything to parallelize across data or do anything out-of-core, we just parallelize across fitting multiple estimators. What that means is computation dependent.
from dask-searchcv.
Related Issues (20)
- Efficiency for GridSearchCV on large graphs HOT 46
- Asynchronous algorithms and "Good enough" RandomSearchCV HOT 7
- Memory Error when number of worker increase in daskgridsearch HOT 13
- pip installable HOT 4
- Works with stock Pipeline now? HOT 4
- Maybe add verbose parameter to "RandomizedSearchCV" ? HOT 9
- Wish: support multiple metric scoring as in scikit-learn 0.19
- Compatibility with scikit-learn 0.19 HOT 6
- TypeError: can't pickle NotImplementedType objects on sklearn.metrics.make_scorer and FeatureUnion HOT 8
- Failure on model pipeline that succeeds using stock scikit-learn HOT 7
- bayesian optimization for hyperparameter tuning HOT 1
- COMPAT: /Users/taugspurger/Envs/dask-dev/lib/python3.6/site-packages/scikit-learn/sklearn/base.py:114: DeprecationWarning: Estimator Pipeline modifies parameters in __init__. HOT 1
- AttributeError: 'unicode' object has no attribute 'version' - LooseVersion with Py 2.7 HOT 6
- Incompatibility With Keras Scikit-Learn Wrapper HOT 6
- Multi-threading or -processing doesn't work for simple sklearn Pipeline HOT 12
- Fold dask-searchcv into Dask-ML HOT 11
- BaseSearchCV throws IndexError for particular sized optional arguments to BaseSearchCV.fit HOT 2
- Implement partial_fit support in DaskBaseSearchCV HOT 2
- dask-searchcv incompatible with Dask v0.18 HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dask-searchcv.