[Query] Performance impact of geo lookup about elastiflow HOT 3 CLOSED

The developer of the Netflow codec for Logstash provides the following guidance:

For high-performance production environments the configuration below will decode up to 15000 flows/sec on a dedicated 16 CPU instance. If your total flowrate exceeds 15000 flows/sec, you should use multiple Logstash instances.

This used to be:

For high-performance production environments the configuration below will decode up to 6000 flows/sec on an 8 CPU instance. If your total flowrate exceeds 6000 flows/sec, you should use multiple Logstash instances.

On my quad-core laptop using the ElastiFlow pipeline I can process 4000-5000 flows/sec, which is inline with, maybe even a bit better than, the guidance above.

Ultimately you will have to test in your environment. It is however worth mentioning that the ElastiFlow pipeline is designed to lookup ONLY those IP addresses which are part of the public address space (as determined by a CIDR filter). There is no GeoIP lookup for flows that are from private IP to private IP. If your traffic patterns involve a lot of external traffic, then the penalty of the GeoIP lookups may be more noticeable.

from elastiflow.

Thanks for the quick reply!

from elastiflow.

FYI: I am running Logstash (Elastiflow) on a VM, 4 vCPU of 2.3Ghz, using htop i am seeing 100% cpu on all 4 CPU (mainly from logstash), but strangely my "thoughput" goes from 1250 - 2000 flows/second depending on time of day. I have relatively many public ips, so i disabled geoip lookup (commenting out postprocessing)

from elastiflow.