Use Py4j to communicate between Python and Client.scala about knit HOT 7 CLOSED

Definitely agree. I had similar thoughts -- adding more CLI options is annoying to say the least. I've experimented with this a bit in the past: https://github.com/ContinuumIO/nutchpy/tree/master/nutchpy but was a little less diligent with tests then

We'll also want higher bandwidth communication between Client and AM especially for things like dynamic container management (ongoing discussion here: dask/distributed#128)

My initial thoughts were also Py4J but do you think it's worth considering other options ? Jepp/JPype or Thrift/Protobuf ?

More extensive thoughts on scala/python interaction:
https://github.com/A-OK/Snakes-and-Ladders/

from knit.

I was thinking to use PY4j solely because spark is also using it. I guess it should be fast enough for dynamic allocation, but sending large binary objects is probably not a good idea.

Spark has a similar comment:
https://cwiki.apache.org/confluence/display/SPARK/PySpark+Internals
"large data transfers are performed through a different mechanism."

As for the other options, I guess Thrift stands out, as that "protocol" is used to communicate between the other yarn stuffs (AM -> RM etc.). But then again, I guess simply starting another PY4j gateway in the AM, and registering its rpc port using

rmClient.registerApplicationMaster("", PY4J gateway port, "")

would allow us to connect from python directly to the AM.

We do need to bind to 0.0.0.0 on the AM, and have some way of "securing it", as described here
https://www.py4j.org/faq.html#are-there-any-security-concerns-with-py4j.

from knit.

@quasiben i've implemented a poc at #51

from knit.

@NielsZeilemaker, this is great! Thanks for your contribution/PR here. I just wanted to let you know that @quasiben is out of the office this week, but we can review this when he gets back.

from knit.

Sure no problem, I've had a lot of fun trying this stuff out. For now, I'll leave it as it is and wait for @quasiben to return.

from knit.

This looks fixed t me - OK to close?

from knit.

Oh, yes! Thanks for cleaning up.

from knit.