Support for IOS XR flow sample rate records about elastiflow HOT 17 CLOSED

Good catch. I will make the change. Thanks.

from elastiflow.

Actually it is an issue. I have been sent some sample data by a user so that I can create a fix. I was hoping to get to it this week, but it may slip to next week due to other projects. Stay tuned.

from elastiflow.

Fixed in this commit. Will be included in next release.

from elastiflow.

Hi,
with Netflow v9, sapler information, has been exporter as meta-data, with an option-table, not included in data flow. In particular using Cisco IOS XR:

https://www.cisco.com/c/en/us/td/docs/routers/crs/software/crs_r4-2/netflow/configuration/guide/b_netflow_cg42crs/b_netflow_cg42asr_chapter_00.html

In my case, sampler (1:1024) table is exported every min, and correctly imported from netflow.flow_sampler_random_interval, in to index "netflow.sampler_interval".

for this reason, [flow][sampling_interval] does not exist in data flow record, and the value of byte and pkt, still remain the same because it is always "0" for

if [flow][bytes] and [flow][sampling_interval] > 0

i think it shoud be exported, and be valuated every time a sapler-data flow is received, not for each data flow.

Example of sampler-data flow record:

Example of data flow record:

in semple term: my value still 33.2MB and not 33.2GB
I hope I explained myself :-)

from elastiflow.

I understand your issue. Unfortunately I don't think there is any way to solve this in a Logstash pipeline. It would have to be something that the netflow codec handles, similar to how it handles v9 templates. I would recommend opening an issue on logstash-codec-netflow here...
https://github.com/logstash-plugins/logstash-codec-netflow/issues

@jorritfolmer does a great job maintaining the codec, and it would be worth having him look at it. In the meantime I will reopen this issue here as this is worked through.

from elastiflow.

First of all, I have not yet congratulated, rob, for your work. Very good job!
now, back to our issue, it looks the same problem mentioned from jorritfolmer HERE.
In Netflow v9, data information and sampler info, are exported divided in two different record, Anyway using data flow, without keep in mind sampling it is useless unless sapler is 1:1.....
In my opinion I think is a best way, for now use an enviromer variable to value the sampler interval, as:
ELASTIFLOW_SAMPLER_INTERVAL
meanwhile we try to understand how to get the fields from different events together.
what do you think about?

from elastiflow.

This sampler calculation logic, and other advanced operations on the netflow data, can be better implemented in an Elasticsearch script

Elasticsearch already has the relevant data, and supports simple and arbitrary complex operations on the data.

from elastiflow.

Hey there all,

I'm a little confused about where this is at. I'm in the same position as @jnikoletich but I'm not sure if the resolution is something that @robcowart would implement in any way or if, as @jorritfolmer mentioned, using an Elasticsearch script is how this would ultimately get addressed?

from elastiflow.

I haven't had time to look at it yet. Sorry.

from elastiflow.

Just FYI, I'm seeing the same thing with IPFIX templates/flow data. Just commenting to be sure it gets addressed in that pipeline as well.

from elastiflow.

The manual sampling rate has only been applied to logstash/elastiflow/conf.d/20_filter_20_netflow.logstash.conf. Like @telsin, I see the same issue with IPFIX flow data from Juniper MX 480 devices.
Adding the workaround code you committed in 9a139c3 to elastiflow/conf.d/20_filter_30_ipfix.logstash.conf allows for correct values to be calculated.

from elastiflow.

A question, does elasticflow extrapolate the sampled netflow rate to full/normal rate now ?

from elastiflow.

@robcowart I can also confirm that applying the sampling dictionary (as described in #52 (comment)) to the ipfix filters in 3.4.1 gets sampling working for MX480 running JUNOS 16.1.

If you would be so kind as to apply this change there as well, as you intended in #52 (comment).

Thank you!

from elastiflow.

Hi @robcowart , looks like workaround introduced in 9a139c3 does not work for flows with IPv6 entries received from NetFlow v9 exporter.
flow.sampling_interval is 0, regardless of setting in dictionary.
Flows with IPv4 entries work as expected, counters for bytes and packets are adjusted.

from elastiflow.

Is there anyone in this issue that can send me a PCAP of the options template and data from one of these flow sources? I want to make sure this is being handled properly in the new ElastiFlow Unified Flow Collector, but I need some data for testing. You can send to me directly at [email protected].

from elastiflow.

The limitations of Logstash complicate this issue. For example a dictionary of IP to sample rate mappings treats the IP as string, not an IP address type. This makes it challenging especially for the IPv6 addresses. The new ElastiFlow Unified Flow Collector handles this better. First it supports option data, so that the device can report on the sampling rates directly and the collector can adjust accordingly. When an external dictionary is used, the IP entries are handled as true IP address data types which ensures accurate matching of entries to the raw data. I recommend taking a look at the new collector.

from elastiflow.

This issue is being closed as this legacy version of ElastiFlow is now deprecated and is to be archived. Please try the new ElastiFlow, request a free Basic Tier license, and join the ElastiFlow Community Slack. Thank you.

from elastiflow.