How much can this handle? about elastiflow HOT 4 CLOSED

I wanted to provide an update on this issue. I have done some testing in an environment where the ingestion rate averages 500 flows/sec over 24 hours (day time peaks are over 1000/sec). Reports covering a 24 hour period, which is 42 million records, are still rendering within 5 secs.

This was a 3 nodes cluster installed on bare-metal. 64GB RAM and SSD storage, which is really a "starter cluster".

I hope this gives you a bit more insight into how the solution can scale.

from elastiflow.

This is a question that can only be answered with "it depends". While you could have 1TB of data on a single Elasticsearch node, spreading it across 3 nodes will provide much better ingest and query performance (and 3 nodes is also the starting point if high availability and data redundancy are important.

Keep in mind that indexing data in Elasticsearch is not the same as writing records to a traditional SQL DB. The process of indexing data optimizes it for search... i.e. querying. So instead of having to scan through millions (or even billions) of rows to find those that contain data for a particular server, the index contains direct pointers to documents which refer to the server. This allows many types of queries to complete extremely fast compared to SQL DBs or many other storage mechanisms.

The guidance that I can provide is that most of the dashboards will render within 5 secs, even when querying 50-60 million flow records. For some environments that may be 30 days of data, in others that may be only 30 minutes (or less). Fortunately the Elastic Stack provides deployment options for either case, but you need to have an idea of your requirements first.

from elastiflow.

KK, thanks for quick reply! I think it might be worth giving it a try then

from elastiflow.

Thanks for update. I definitelly have better insight now. Currently, I'm experimenting with ClickHouse but want to give this a try too.

from elastiflow.