Creating process pipe: Too many open files about org-rifle HOT 16 CLOSED

@NightMachinery quick heads up triggered by org-roam/org-roam#1601 (comment). I am able to avoid the "Too many open files" error by disabling git-gutter. Maybe that works for you too.

from org-rifle.

This has nothing to do with org-rifle. As the backtrace shows, the problem appears to be caused by your using global-magit-file-mode.

from org-rifle.

Thanks for taking a look.

This has nothing to do with org-rifle

I was aware of that, hence my statements: "I realize that this is ultimately an issue with how Git is run..." and "I was just wondering if this has happened for other org-rifle users". Thanks to your investigation this is even more likely given that global-magit-file-mode is t by default. I tried setting global-magit-file-mode to nil in my user config (and even changed the default in magit-files.el and removed the matching .elc file), but to no avail, it is still called as in above bt. I understand that you are not supposed to give support on magit issues here, but do you have an idea offhand? Again, I just wanted to bring this up as I assumed other org-rifle users might run into this as well (and even more likely with global-magit-file-mode having default value t).

from org-rifle.

I just found https://github.com/alphapapa/org-ql, quickly gave it a spin and so far it works for the searches I had in mind for org-rifle. So either way, this is solved for me as I plan to dig more into org-ql. Thanks for all your great contributions to the ecosystem!

from org-rifle.

I'm far from an expert on these kinds of issues, but I'm guessing there's more to this than meets the eye, probably some combination of using Windows, using MinGW Emacs (as opposed to Cygwin or native win32 Emacs), native Windows git (as opposed to Cygwin or MinGW git), etc.

On non-Windows platforms, AFAIK Emacs does not keep files open after loading them into a buffer. I don't know if it also works that way on Windows. But since there are at least 2 or 3 different builds of Emacs that you could use on Windows, and some of them (like Cygwin) are more "Unixy," I don't know if they all behave the same way.

Are we even sure that global-magit-file-mode is the issue? Try disabling the mode (use customize-option) and then see if it still happens. Also, try just writing a function to open a large number of files in Emacs and see if the problem happens.

Another question: how many Org files are you actually opening at once?

I'm glad that org-ql works for you, and it has surpassed org-rifle in all ways but one, so that's what I mostly use now as well. But if the problem is caused by opening many files in Emacs, that could happen just as well with org-ql as with org-rifle, as well as any other code that opens many files in Emacs.

And, as usual, you should try to reproduce the problem using emacs -q to see if it's caused by something in your config.

The bottom line, I think, is that 1) Windows is Windows, and 2) there are too many variables at play here, and 3) the problem isn't caused by org-rifle. So there's not much help I can offer you.

I'd suggest that you produce an Elisp file that you can load into an emacs -q session that reproduces the problem with the minimal amount of code (i.e. it probably shouldn't even use org-rifle, it should just find-file on a bunch of files), and then post that to reddit.com/r/emacs or emacs-devel and ask for help, being sure to mention that it's a Windows/MinGW problem.

Thanks for all your great contributions to the ecosystem!

Thanks for the kind words. I'm glad they're useful to you.

from org-rifle.

Thanks for all the pointers and thoughts.

Another question: how many Org files are you actually opening at once?

About 300. I am using https://github.com/org-roam/org-roam where the tendency is to use many small files instead of a few big files.

I will look into reproducing with a minimal Elisp file.

from org-rifle.

About 300. I am using https://github.com/org-roam/org-roam where the tendency is to use many small files instead of a few big files.

Yes, in general, this is why Org tends to work better with fewer, larger files rather than the one-note-per-file paradigm. When an Org file is opened in a buffer, hooks are run, which can vary depending on the user's configuration, and all that overhead can add up.

Tools like org-tree-to-indirect-buffer, org-rifle, org-ql, org-bookmark-heading, etc. allow subtrees to be handled almost as if they were individual files, so using lots of small files tends to be a disadvantage. org-roam works around that by having its own index and cross-linking tools, so a tool like org-rifle, which is designed for the fewer-files paradigm, doesn't integrate as well with org-roam's many-tiny-files paradigm.

I'm curious, since I don't use org-roam: why are you using org-rifle to search your org-roam files? I thought org-roam had its own indexing and search tools designed to work well with its workflow.

from org-rifle.

When an Org file is opened in a buffer, hooks are run, which can vary depending on the user's configuration, and all that overhead can add up.

Yeah, I see now how that can lead to trouble quickly and I should have seen it coming as I lean to - but am not consistently on - the minimalist side of things.

I'm curious, since I don't use org-roam: why are you using org-rifle to search your org-roam files? I thought org-roam had its own indexing and search tools designed to work well with its workflow.

I used helm-ag and org-rifle a lot before starting out with org-roam. Everything worked smoothly as I used fewer, larger files as you described. Now that I use org-roam more and more, notes start to add up as single files. When I am in org-roam mode, I use its indexing and search tools which work flawlessly. Now the way it is set up and supposed to be set up is to have an org-roam-directory where all the notes live. In my setup it is a subfolder of my old org file hierarchy. So now occasionally I need to do searches on all the org files for e.g. a heading tag, because I don't remember whether I noted it down in org-roam or the "old system". So it is about this transition phase to org-roam that I need to figure out how best to apply workflows and supporting tools. I am now with Emacs long enough to know that the tinkering won't ever stop, but I am aiming for a local optimum right now. I heard about https://github.com/Wilfred/deadgrep a while ago and tried it out yesterday - I like the persistent search interface (frequent helm-resume user here...) and it works so far for simple things and is quite snappy as well. I think I will reach for org-ql for when I want to do more detailed and org-aware queries and use deadgrep in the meantime as an helm-ag and org-rifle alternative.

Now I am curious, did you tinker with org-roam already or do you plan to?

from org-rifle.

Thanks for the explanation. Now I understand better what you're trying to do.

I'm keeping an eye on org-roam. It's a very cool project and seems well-executed. But I prefer the fewer-larger-files paradigm, thinking of files more as notebooks and outlines within them as pages and notes, rather than the lots-of-small-scraps-of-paper, zettlekasten-style paradigm. It seems to me that, by treating headings as notes, Org can already do what org-roam does, and there are also a variety of technical issues and drawbacks with using many, tiny files, so I don't see the appeal of the one-note-per-file paradigm.

Roam Research has some cool features and ideas, and I think some of them could work well in Org, but I don't think that necessitates many-tiny-files.

from org-rifle.

Thanks for sharing your opinion and preference on the many-tiny <> few-large files continuum - I will keep it in mind during the exploration of the solution space.

from org-rifle.

@alphapapa Any ideas on searching a big repo such as https://github.com/NightMachinary/r_rational ? helm-org-rifle gives apply: Creating pipe: Too many open files.

from org-rifle.

@NightMachinary as I mentioned in #60 (comment) this already happens with around 300 org files. Your repo contains a few thousand org files so is expected to fail in the same way. I think there won't be any further input on top of the prior conversation in this issue.

from org-rifle.

@alphapapa FYI and as a follow-up on #60 (comment) in case you did not notice already: org-roam/org-roam#1401 where org-roam will move away from the one-file-per-note approach.

from org-rifle.

@alphapapa Any ideas on searching a big repo such as https://github.com/NightMachinary/r_rational ? helm-org-rifle gives apply: Creating pipe: Too many open files.

As @fleimgruber said, I don't think any more can be added. If you're on Windows, that's probably part of the problem. Whether you are or not, I don't know what this package has to do with pipes, so there's probably a different package or configuration issue causing it. And @fleimgruber said that org-ql seems to work on a large set of files where org-rifle didn't, so try that.

@alphapapa FYI and as a follow-up on #60 (comment) in case you did not notice already: org-roam/org-roam#1401 where org-roam will move away from the one-file-per-note approach.

Yeah, thanks, I was glad to see that. I might end up using some of its features someday.

from org-rifle.

@alphapapa I tried helm-org-ql, and it didn’t work either. (I didn’t even try it on that huge repo, it didn’t work on my normal notes directory.) I think any elisp solution will not scale, unfortunately.

BTW, I’m on macOS, so no Windows bugs.

from org-rifle.

I think any elisp solution will not scale, unfortunately.

On the contrary. For example, one org-rifle user uses it for a large Org-format database in his work at a museum, and he searches thousands of files with it. Once the files are opened in Emacs, searches are quick.

I don't know what problem you are having, but I doubt it is unique to this package. MacOS is, of course, a proprietary platform, and its support for FOSS software seems to be declining.

Anyway, even though I don't use MacOS, I know that there are a variety of different Emacs distributions that people use. To even begin to troubleshoot would require much more information and some experimentation on your part, e.g. trying different Emacs distributions, checking versions of supporting GNU tools, etc.

My best advice, as you may anticipate, is to use a FOSS OS that works well with standard FOSS tools.

from org-rifle.