Too many open files ? about pywb HOT 3 CLOSED

Hm, it looks like the file handle may not be getting closed, should have a fix soon. Thanks for the report. I assume this happened after loading the page many times?

from pywb.

Not really.
It seems to happen when there are many .idx files (more than 1000, in my case 1509).

from pywb.

Ah ok, that is a lot of .cdx files. I checked in a fix to ensure cdx files are getting closed, however, I think this may exceed the limit of open files on your machine.

For each lookup, all the files need to be open at the same time, so having that many .cdx files is generally not recommended. It is much better to merge them into a few files or even a single file if possible.

The cdx-indexer can write a single cdx file for many warcs:

cdx-indexer -s /tmp/merged.cdx /path/to/warcs/

Or, if you prefer to merge manually, you can do something like this:
export LC_ALL=C; sort -m *.cdx > /tmp/merged.cdx and use the /tmp/merged.cdx

from pywb.