Transaction Support Timeline about goblin HOT 6 OPEN

Hi Conrad,

Sorry for the slow response. I have had limited internet connection over the last week.

During the original development of Goblin, yes, we had planned on supporting transactions. This was put on hold for various reasons. I know that you can perform transaction management using the Gremlin Console, but I am not entirely sure that you can using the Java remote connection/GLV. Regardless, I am currently in the process of moving and I will be out of the country for several weeks. Therefore, any work I would put in here would be during the second half of July.

If you need transaction management now, I would consider using the aiogremlin client to manage sessions and transactions by submitting raw scripts to the server. If you want to integrate this in the Goblin, and you need the support earlier than I can provide it, you could always open a PR and I could work with you on it while I am out of town.

Finally, please note that I will be releasing new versions of aiogremlin and Goblin (for tp 3.2.5), later this week, so you may need to upgrade.

Let me know what you decide, and we can work towards getting something going for you.

from goblin.

I checked and couldn't find an explicit description for transactions in GLV documentation. It also isn't supported in the official gremlin-python.

I can probably wait, but I would also be happy to help out. I don't know that it will save you much time though. I looked through the source and the Bytecode already accumulates multiple steps. From what I could tell, it gets submitted to the cluster in the request message directly.

Depending on which graph server is being used, transaction support is slightly different. Gremlin will create a new transaction for each request and automatically commit/rollback. Multi-threaded transactions can be created explicitly if supported. The issue is how to support transactions across multiple requests and allow multi-threaded transactions.

I could imagine having a Transaction class that gets instantiated via Session.tx() which has a traversal property for accessing all the methods that add steps to the byte code. It would not send the request to the server unless flush or commit are called. When a gremlin session is created, there is an optional parameter that configures whether transactions persist between requests. They recommend going sessionless where possible, so we could expose that as an extra parameter on the Session constructor that does sessionless by default. I realize that your Session wasn't designed to correspond directly to a database session, but perhaps we could make that correspondence or create a new class that is a database session? They recommend using uuid as the session identifier; we could easily start a database session when the object is created and use the existing machinery.

Multi-threaded transactions are a different story because they aren't explicitly discussed in gremlin. Perhaps individual developers should handle that depending on their server.

You probably have a better idea of how you want to extend the code, so I am happy to have you decide what's best.

from goblin.

Hi Dave,
Do you know when you will implement transactions?

from goblin.

I hope to discuss this with some of the TinkerPop PMCs this fall. I'm not sure where it will go. I will definitely post updates when I know more.

from goblin.

@davebshow Any update on transactions or the project in general? Doesnt seem to be updated in a while.

from goblin.

I'm copy and pasting this from another ticket, its details about how I fixed this issue (sort of) in a fork of the project, happy to share any code.

Side note (ill repost this on the relevant ticket as well). The fork I have going locally adds transaction support though likely not in the way the original ask intended, but I think the only elegant way possible given the underlying gremlin implementation.

Basically I took advantage of the fact that the only way to get a traversal is either 1) use the raw driver and pass text-based traversals through it (unless im mistaken this is impractical for an OGM, though maybe I'm missing something). 2) Treat each traversal itself as its own transaction and accept that you cant manually create transactions that spread across multiple invocations of traversals (this appears to be the official answer)

So I decided to exploit feature #2 and implement a sort of optimistic locking that works with one caveat, it is only useful when adding new edges and nodes it isnt useful for modifying them in place. This may seem like a big caveat, and it is, but it also is useful when along side another feature I implemented: per-element immutability. In that case if an element is set as immutable then it can be created, but never destroyed nor modified after its initial creation.When these two new features I added are used in tandem you basically get something that is basically optimistic locking that works on immutable graphs.

Let me explain further. Its very typical in immutable graphs that you still need to maintain integrity. This means that a single node or edge may be inconsistent on its own and only makes sense when connected in a valid pattern, a simple example being a graph that must be connected to be valid. As such you may want to create a set of edges and nodes but with the guarantee they either all get created or none of them. Thus the optimistic locking I mentioned.

How it works:

I created a new attribute on Element classes called __immutable__ which takes an enum that has one of three values: {OFF, SIMPLE, TRANSACTIONAL}. When the value is OFF it behaves classically, this is also the default value if nothing is set. SIMPLE only creates immutability but makes not transactional guarantees. TRANSACTIONAL gives the combined functionality of both I mentioned. When this mode is set then transactions occur automatically through the flush method. Whenever a flush occurs a randomly generated UUID is created as a transaction identifier. Then all element updates throw an exception and only allow element creation. When being created every element is given a property called "dirty" which contains the transaction ID. If all saved elements succeed to write to them database then a final traversal is called, written especially a single traversal, which will drop the property dirty from all elements where the ID is matched, this effectively makes the edges and nodes live. However if an exception occurs a rollback traversal is called which drops all the dirty elements.

Finally at startup the system can call a method which scans the DB for any dirty values that may be left over from a crash and removed them as well.

Using this mechanism I was able to get guaranteed DB integrity. So far its only used in single-application-single-database configuration. But I plan to adopt it for multiple application front ends. For that I'd need to do some integrity checking as part of the commit traversal.

I can pull out the code for this if you are interested, right now its just something i use locally.

from goblin.