Scalability of filtering operations about django-zombodb HOT 1 OPEN

Hi @thclark, thanks for the issue.

zombodb in effect evaluates that filtered queryset, then sends a list of ids to ElasticSearch in order to filter the potential results

In fact it's the opposite. The 'name:Hut' search is executed on ElasticSearch side. Then the results are filtered with the additional SQL filters (WHERE). Note results can be limited to avoid heavy ES searches.

Haystack and basically every other search tool I've checked will suffer from similar problems: you need a list of ids to combine SQL filtering with searching (on a separate Search Engine).

But be aware nothing prevents you to ensure you're using only ElasticSearch for searches and completely avoid the use of SQL WHERE / .filter. In fact, that's recommended per docs:

It’s fine to call filter/exclude/etc. before and after search. If possible, the best would be using only a Elasticsearch query.

For that, you just need to use filter everything with the ES syntax. Try the dsl_search method.

However, I agree that's not clear enough, so I think we should separate that into a new warning explaning better what's going on behind the scenes. I'll leave this issue open due to that.

I'ld personally suggest you trying django-zombodb if you already have the infrastructure to support it. Be aware of zombodb's limitations though: https://github.com/zombodb/zombodb/blob/master/THINGS-TO-KNOW.md

from django-zombodb.