OutOfMemoryException Building Index - Need to make the enumeration more lazy about examine HOT 3 CLOSED

Hi, yeah it should support that but it could be due to how the SimpleDataIndexer queues up it's data for indexing. The way it currently works is it will lookup all of the data that needs to be indexed and serialize it to the Examine format (which is in memory) and then the Examine indexing thread will consume the queue. Chances are that the memory is being filled up with millions of serialized items waiting to be indexed.

But it should be smarter and perform it's data lookups in batches (let's say of 1000) and ensure the batches only start enumeration at the time of indexing instead of queuing up in memory. This is possible now with a bit of work but Examine should be managing this by default OOTB.

I started on making this happen a while ago and must have got side tracked. I'll update this issue name so I don't forget about this for the next release!

In the meantime, I think you might have to do your index rebuilding in batches with timeouts as to not fill up the the mem with serialized versions of your data. Once your index is built though it's strongly advised to not actually rebuild it ever (unless there's data inconsistencies) and just keep it up-to-date with your source data.

from examine.

In case anyone else is running into this, I've found that if I page the query into 15,000 results at a time then I can get the index to consistently rebuild for my data set.

from examine.

I think this issue is what I've fixed here: #41

1c497dc#diff-a4636988629a819865ddf6634cf261f1R78

from examine.