Joblib, parallel_backend, and performance about dask-ml HOT 5 CLOSED

Thanks, I'll try to debug this later today or tomorrow. Can you say a bit more about

1. The graph backing X and y. Do they involve disk IO? Do all the workers share a file system?
2. The estimator you're fitting? This probably isn't the issue, but it may help with debugging.

from dask-ml.

1.) X and y are already in memory - no additional disk IO after they've been read from std.io to a dataframe earlier in the process. The workers in the above are 10 containers spread across three physical hosts, so each worker on average shares a filesystem with 2 other workers.

2.) The estimator here was RandomForestClassifier

from dask-ml.

Sorry for the delay on this! Let's try to narrow this down to see if it's just the scheduler that's not working properly. Could you setup a cluster / client and try out the following:

import dask.distributed
import pandas as pd
import numpy as np
import dask_ml.joblib

from sklearn.externals import joblib
from sklearn.model_selection import RandomizedSearchCV
from scipy import stats

from sklearn.base import BaseEstimator

class DummyEstimator(BaseEstimator):
    def __init__(self, parameter=None):
        self.parameter = parameter
        
    def fit(self, X, y=None):
        return self
    
    def predict(self, X):
        return np.zeros(len(X))
    
    def score(self, X, y=None):
        return 0

search = RandomizedSearchCV(DummyEstimator(), {"parameter": stats.uniform}, cv=3, n_iter=20, verbose=10)

%%time
N = 100_000
X = pd.DataFrame(np.random.randn(N, 10))
y = pd.Series(np.random.uniform(size=N))

addr = client.scheduler_info()['address'] 
with joblib.parallel_backend('dask.distributed', addr,
                             scatter=[X.values, y.values]) as pb:
    search.fit(X, y)

For me, that finishes in ~4 seconds, just using a local cluster.

from dask-ml.

FYI, if you're able to try out and joblib master and dask.distributed master, things may have improved in the last couple weeks. Nothing specific to this issue, but we were making changes to that code and it might have fixed things magically :)

from dask-ml.

Hey Tom - Sorry I sorta dropped off the face of the planet. I appreciate your responses - I might not get around to re-checking this out for a bit - so I will close this issue for now :)

from dask-ml.