not all fields filled about elastiflow HOT 4 CLOSED

Which version of Logstash, or perhaps more important, which version of the Logstash codec are you using? I have found the codec handles a lot of scenarios that you may encounter, but sometimes vendors will send data structured in an unexpected manner, which appears to be what you are seeing. You will need to open an issue for the Netflow codec (https://github.com/logstash-plugins/logstash-codec-netflow), and ideally provide a PCAP of the flows so that it can be investigated in detail. I am also happy to test such a PCAP myself if you can provide it.

from elastiflow.

I have opened an issue as you recommended. logstash-plugins/logstash-codec-netflow#87 and hopefully provided enough information to troubleshoot. Thank you

from elastiflow.

This OP's error is originating from Logstash's bad feature. UDP-layer do not send metadata to codec, meaning codec cannot analyze netflow-v9 template per host. So every template with same template ID will overwrite the template in logstash's memory.

Router A template 256 = xxx.yyy.zzz
Router B template 256 = aaa.bbb.ccc

So netflow will try to interpret netflow packet from A with later template received from B. Router B's netflow packet will be decoded correct unless A sends new upgraded template and then B's flows cause errors in the log described.

But above to be problem you need multiple devices sending netflow v9 templates. Do you have many firewalls?

To fix this you need to patch udp.rb to include "metadata" configuration option and capability to send this metadata upwards. Netflow-codec already understands it on code-level so only udp.rb and conf.d file needs to be fixed.

My elastiflow 10_input.logstash.conf
input {
udp {
type => "netflow"
metadata => true
port => 2055
codec => netflow {
versions => [5,9]
cache_save_path => "/var/lib/logstash/netflow_cache"
}
}
}

from elastiflow.

Resolved by #29 and c6d01da

from elastiflow.