leverage cloudpickle for missing functionality in dill about dill HOT 6 CLOSED

You are talking about cloudpickle, which is fairly impressive, but I'm fairly sure has smaller language coverage then dill has. I do know of one or two things that cloudpickle can handle that dill is still working on… but I don't see anything in cloudpickle that I would want to absorb as part of the dill codebase -- especially not with the LGPL licensing issue. That being said, dill predates cloudpickle, and I'm sure that there has been cross-pollination of ideas over the years.

Have you tried dill? dill handles everything you have mentioned above. However, if you are finding something that dill can't handle for you, please let us know.

from dill.

That all sounds reasonable; just so long as everyone is aware, rather than missing out... One small blow in the battle against unnecessary duplication of effort. Necessary duplication of effort I am completely OK with. ;-)

Thanks for the response, and for the work on dill more generally.

from dill.

Aaron Staley here; author of cloudpickle.

As discussed here (http://mail-archives.apache.org/mod_mbox/spark-dev/201307.mbox/%3CCE1B73E9.F573D%[email protected]%3E), I earlier released cloudpickle.py and debugpickle.py + dependencies under the BSD license, so feel free to use them in whatever code you want.

-Aaron

from dill.

@usaar33: Hey Aaron, thanks for the pointer on the license. You did some really nice work in cloudpickle. I wasn't aware of debugpickle (or I don't remember if I was aware of it). I'll have to check it out. The apache spark team has discussed my and your code, and has pinged me a few times on some corner cases cloudpickle handles that they want to see in dill.

@howthebodyworks: In this case, I don't think the duplication was necessary... it just happened quite some time ago, and before collaborative environments (like github) were really as prevalent.

from dill.

From the Spark side, I'd be very interested in collaborating on improvements to dill and cloudpickle.

We've made a handful of changes and extensions to our copy of cloudpickle.py. For example, @davies has a pull request that adds PyPy support to cloudpickle (apache/spark#2144).

from dill.

@JoshRosen: I'm a bit sticky about quality and impact of what goes in… but that's more of a mild warning than anything else. I'd also be very interested in such a collaboration.

from dill.