Printing variable value in notebook when I set value to a variable about ispark HOT 5 OPEN

Thanks a lot for report, I've been aware of this for a long time. The major cause is that Spark interpreter only expose its API to retrieve the last variable being set, regardless of whether its the last line of a code or not. It's possible to use deeper non-API functions to retrieve it but the gain doesn't justify the risk. Could you append it with ;"" to override it with an empty output?

from ispark.

Yes I did that, but I don't know this workaround solved this issue or not.
Because when I trained an svd and I wanted to serialize V matrix ISpark threw out of memory exception.
The same job in a spark shell run smoothly.
Maybe you know why.
Thanks for helping!

from ispark.

ISpark has much higher memory consumption than spark-shell for obvious reason (message queue, visualization etc.) Make sure you increase both --driver-memory and --executor-memory to accomodate your dataset.

I doubt if display too much is the cause of out of memory: in this case the websocket will timeout first before that happens.

from ispark.

Hm... Strange because I set driver-memory to 32G and the actual usage was about 23G then I started to save object file and the notebook died with oom.

from ispark.

IMHO save object file is done on workers/executors. SO that's expected

from ispark.