Can we run ConnectedComponents algorithm in production using this library? about gelly-streaming HOT 3 OPEN

Hi @kant777,

Thank you for your interest in gelly-streaming. Note that this is an experimental library and it is not integrated with Flink's state management and checkpoints currently. So, if fault-tolerance is important for your application, I wouldn't suggest using it. That said, it is not so hard to implement the necessary features and I can give you pointers if you'd like to do that.

Regarding your second question, keyBy(0) does not change the parallelism. The 0 here indicates the key as the position in the input tuple, i.e. this refers to the partition index.

-V.

from gelly-streaming.

Hi @vasia

Sure, please send me the pointers to implement fault tolerance.

Also, I am trying to understand what this line is doing? and why it is needed? because if I remove the line the reduce is giving me the right connected components as expected.

from gelly-streaming.

@kant777 @vasia interested in the pointers to make this library fault tolerant as well.
Were those shared or possibly implemented?

from gelly-streaming.