Question: How to increase performance on entity resolution? about zentity HOT 1 CLOSED

@pmartin-96

I recommend this webinar (slides) where I discuss sizing and capacity planning for Elasticsearch. Below are some steps you can take following the guidance of that presentation.

Improving the performance of individual searches by tuning the resource configurations of Elasticsearch data nodes:

• Prefer SSD over HDD storage devices
• Prefer local storage over SAN, and SAN over NAS
• Increase memory (RAM) on the Elasticsearch host to read more of the data from cache instead of from disk. If your data is ~4GB, consider allocating >8GB RAM (4GB JVM heap, 4GB for the OS) to fit the entire data set in the file system cache (i.e. the non-heap memory). You may also consider testing the index.store.preload setting, which preloads the contents of your index entirely into the file system cache for faster searches. By default, data is cached only after it's read for the first time.

Parallelizing searches to minimize end-to-end duration of multiple resolution jobs:

• Add nodes and replicas to increase search throughput. For example, add two data nodes to the cluster and set the number of replicas on your index to 2 to employ all three nodes in your searches. Assuming your index has one primary shard (which I recommend given the size of your index), then only one node will execute a given search even with the other replicas present. Having multiple and replicas won't improve the speed of any single search by itself -- but it will let you run more searches per second. So if you can run your resolution requests in parallel (rather than one at a time in series) then this could drastically cut your processing time overall.

I don't think an upgrade will be necessary, although I think it's always a good idea to stay on the latest stable releases when possible. I've checked the release notes for zentity and Elasticsearch, and I don't see performance enhancements that would likely help your desired outcome.

from zentity.