Spark Broadcast exceeding executor memory with large training data set about spark-sklearn HOT 4 CLOSED

Yeah pre_dispatch doesn't have any effect; it's there to make this a drop-in replacement for scikit's implementation, but the execution context is quite different.

A broadcast will only be copied once to each executor in Spark. Spark effectively 'pulls' but it won't matter as all the tasks need this data. The grid here reuses the single instance of the X broadcast. I don't think that's the issue unless there's something subtler at work.

However this fix might be relevant, which would come in the next version: 03df085

Else, maybe you can say more about what runs out of memory where.

from spark-sklearn.

PS I merged #100 which removes "pre_dispatch". Reopen or comment if the next release doesn't resolve this, as I think the issue is broadcasting the split indices.

from spark-sklearn.

@srowen There is a restriction on the size of X due to cloudpickle limitations. I'm seeing the following error when the size of X exceeds 2 GB.

PicklingError: Could not serialize broadcast: OverflowError: cannot serialize a string larger than 2 GiB

How do we use GridsearchCV with larger training datasets?

One option I can think of is to not use broadcast variable for X and y. Instead, load this as a pandas data frame and write to a file. Every executor would read from that file.

from spark-sklearn.

Yeah, in that case you would need to side-load the data somehow. This is a limitation of Spark/cloudpickle, as you say.

from spark-sklearn.