What operations are concurrency safe and I can call from different Goroutines? about go-git HOT 9 CLOSED

go-git is not thread-safe, this is due to not having mutex in the write operations. But also, could be potentially problematic for reading operations since some areas are using caches, and this may cause problems.

We didn't make evaluation of this, and personally I never used it in a concurrent mode for a single repository. But it's an interesting topic.

from go-git.

@kaihendry thank you for sharing some code to make it easier to reproduce the problem.

I managed to get your code working with 10 concurrent go routines by using memory storage with three additional changes:

• #175 (this will remove the panic)
• fluxcd#14 (this will ensure the results' accuracy)
• Adding a wg.Wait() at the end of your code.

Here's a full end to end using the Flux' fork: https://gist.github.com/pjbgf/45eb717606f4857661e071f6249f4225
I will do a few more tests and performance benchmarks before I submit fluxcd#14 upstream (here).

I don't think this will resolve all concurrent use cases, but could be a start.

from go-git.

Thank you very much. I fortunately had a strong abstraction layer to go to Git calls. I landed up putting a single Git access mutex. I lock and unlock when actually access.
The only problem was re-entrant access but I was able to unwrap those calls and make some of them private and leave them unlocked trusting calling calls being thread-safe. The public calls are inside lock/unlock.
So for now, I am ok.

from go-git.

Noticed it wasn't thread safe myself, as I'm working on a project that gets matching file objects, and passes them to large group of goroutines to concurrently scan each object for specific tokens (scanning for TODO's and similar), to be more efficient, it reads the file as an io.Reader rather than all at once, which means there are concurrent calls to plumbing/object.GetBlob() to get the reader as well as in the TreeWalker, that are definitely not concurrent safe. It seems after getting the io.Reader, I don't think I have to worry about concurrency, so it's mainly the function calls to the library.

Will wrap the calls against these with a mutex for now, but would definitely be nice to see the library thread-safe.

from go-git.

Hi, I've hit fatal error: concurrent map read and map write and I don't even understand what the write is, since I'm reading in my loop: https://gist.github.com/kaihendry/748b57905d35403751be2f4aac6cd40f#file-main-go-L75-L95

Is there some way to speed up reading the commit's line Addition/Deletions?

Is concurrency just ruled out? src-d/go-git#457

Thank you!

from go-git.

It appears about 4x faster (200 it/s versus 50 it/s) when using 10 concurrent go routines in local random repo! 🔥

Oddly with https://github.com/kaihendry/lesshero/blob/main/main_test.go it doesn't seem faster at all though. I must be doing something wrong!

from go-git.

To help us keep things tidy and focus on the active tasks, we've introduced a stale bot to spot issues/PRs that haven't had any activity in a while.

This particular issue hasn't had any updates or activity in the past 90 days, so it's been labeled as 'stale'. If it remains inactive for the next 30 days, it'll be automatically closed.

We understand everyone's busy, but if this issue is still important to you, please feel free to add a comment or make an update to keep it active.

Thanks for your understanding and cooperation!

from go-git.

I think it should be reopened because there still exists such problems.

from go-git.

@d4x1 there is a big overlap between this and #773, so instead of having two issues on the same subject, let's track the work on the issue that is open instead.

from go-git.