billfarber / kafka-marklogic-source Goto Github PK

Implement a "long polling" option so that the connector will wait for a record to be available before breaking the connection.

This will primarily involve developing a server-side endpoint that can handle long polling.

Improve the efficiency of the connector so that it can process sets of documents efficiently with respect to the database.

As a connector,
I want to have multiple copies of the kafka-marklogic-source running,
So that I can pull documents out of MarkLogic as fast as possible.

Given documents that are in the "ready" collection,
When the connector queries the database in update mode (for setting the collection to "processing",
Then the connector doesn't lock the entire set of documents,
and the connector only queries a subset of forests,
and the connector only locks the target documents in those forests.

Use a synthetic semaphore to lock the queried documents in a subset of forests.

As a connector,
I want to not update records in MarkLogic until delivery is confirmed,
So that records are not skipped.