In https://bonsai.sensu.io/assets/sensu-plugins/sensu-plugins-network-checks, there is the ability to monitor ports to see if they're listening/open. We should expand this plugin to provide parity with the other check collection.

We very often have the following error message:

The sensu agent must then be stopped, the cache folder emptied and then restarted.
This only happens with this asset.
Furthermore, the asset is packed incorrectly, so that it is not in the hash root folder but directly in /var/cache/sensu/sensu-agent/

Error executing network-interface-checks: error opening metric file /var/cache/sensu/sensu-agent/network-interface-checks

agent version
v6.10.0

Server
Sensu version 6.10.0

---
api_version: core/v2
metadata:
  created_by: sensu-flow
  labels:
    sensu.io/managed_by: sensuctl
    sensu.io/workflow: sensu-flow
  name: network-interface-health
  namespace: gsdd
spec:
  check_hooks: null
  command: >-
    network-interface-checks --state-file {{
    .annotations.network_interface_monitoring_state_file | default
    "/var/cache/sensu/sensu-agent/network-interface-checks" }}
    --exclude-interfaces {{ .annotations.excluded_network_interfaces | default
    "\"\"" }}
  env_vars: null
  handlers: []
  high_flap_threshold: 0
  interval: 30
  low_flap_threshold: 0
  output_metric_format: prometheus_text
  output_metric_handlers: null
  output_metric_tags:
    - name: entity
      value: '{{ .name }}'
    - name: host.name
      value: '{{ .name }}'
    - name: namespace
      value: '{{ .namespace }}'
    - name: os
      value: '{{ .system.os }}'
  output_metric_thresholds:
    - name: drop_in_rate
      null_status: 0
      tags:
        - name: interface
          value: '{{ .annotations.default_network_interface | default "eth0" }}'
      thresholds:
        - max: '0.0'
          min: ''
          status: 1
    - name: drop_out_rate
      null_status: 0
      tags:
        - name: interface
          value: '{{ .annotations.default_network_interface | default "eth0" }}'
      thresholds:
        - max: '0.0'
          min: ''
          status: 1
    - name: err_in_rate
      null_status: 0
      tags:
        - name: interface
          value: '{{ .annotations.default_network_interface | default "eth0" }}'
      thresholds:
        - max: '0.0'
          min: ''
          status: 1
    - name: err_out_rate
      null_status: 0
      tags:
        - name: interface
          value: '{{ .annotations.default_network_interface | default "eth0" }}'
      thresholds:
        - max: '0.0'
          min: ''
          status: 1
  pipelines: []
  proxy_entity_name: ''
  publish: true
  round_robin: false
  runtime_assets:
    - sensu/network-interface-checks:0.2.0
  secrets: null
  stdin: false
  subdue: null
  subscriptions:
    - linux
    - linux/network
  timeout: 10
  ttl: 0
  max_output_size: 0
  discard_output: false
type: CheckConfig

Overview

Sensu Go network device monitoring for Linux. The Sensu Ruby plugin for network interface monitoring was downloaded over 4.2M times, making it the 5th most downloaded Sensu plugin of all time!

References

• sensu-plugins/sensu-plugins-network-checks (MIT License)
• Telegraf net plugin (MIT License)
• Prometheus node_exporter netdev module (Apache License 2.0)
  • netdev_linux
  • netdev_darwin
  • netdev_bsd
  • netdev_openbsd
• Prometheus windows_exporter net module (MIT License)
• Datadog Agent net module (Apache License 2.0)
• Datadog integration docs

Configuration

• --sample-interval (float) number of seconds to sample (default: 3.0)
• --sum (bool) add additional measurement per metric w/ "interface=all" tag (sum of all interfaces per host); NOTE: this could break certain dashboards which calculate sums of per-interface metrics
• --include-interfaces (string) comma-delimited string of interface names to include (e.g. "eth0,eth1")
• --exclude-interfaces (string) comma-delimited string of interface names to exclude (e.g. "lo,eth1") (default: "lo")

Configuration Validation/Logic

• If --include-interfaces is set, and --exclude-interfaces="lo" (if it's the default), then ignore --exclude-interfaces (or reset it to "")
• If --include-interfaces is set, and --exclude-interfaces != "lo" raise a warning and exit status 3
• If --sample-interval is set to zero (0), only collect one sample and do not calculate the "rate" metrics
• If --sample-interval is set to a value greater than zero, collect two samples --sample-interval seconds apart, and calculate a rate that is effectively (sample2 - sample1) / sample-inteval

Metrics

All metrics should be output in Prometheus format.

There are effectively four metrics (bytes, packets, errors, and dropped packets), x2 for sent/outbound and received/inbound, and x2 for collecting counters and rate gauges.

• bytes_sent (counter) bytes sent
• bytes_sent_rate (gauge) bytes sent per second
• bytes_recv (counter) bytes received
• bytes_recv_rate (gauge) bytes received per second
• packets_sent (counter) packets sent
• packets_sent_rate (gauge) packets sent per second
• packets_recv (counter) packets received
• packets_recv_rate(gauge) packets received per second
• err_out (counter) outbound errors
• err_out_rate (gauge) outbound errors per second
• err_in (counter) inbound errors
• err_in_rate (gauge) inbound errors per second
• drop_out (counter) outbound packets dropped
• drop_out_rate (gauge) outbound packets dropped per second
• drop_in (counter) incoming packets dropped
• drop_in_rate (gauge) incoming packets dropped per second
• mtu (gauge) interface MTU configuration; NOTE: no rate or --sum measurement should be generated for this metric.

Metric Tags

• host.name To be provided via output_metric_tags (i.e. {{ .name }})
• interface collect all of the metrics defined above per interface, with the interface name as the tag value (e.g. eth0)

Health Checks

• n/a (pending support for built-in output_metric_thresholds support in the Sensu Go agent).

Catalog Templates

TODO

TODO

TBD:

• TBD if we should automatically add the host.name tag, or if this should be provided via output_metric_tags?? Let's do it using output_metric_tags.
• TBD how rate metrics should be calculated (e.g. a sample captured during check execution -vs- comparing the current value against the previously collected value)
• How many catalog templates should we produce for this plugin?
• Configure telegraf network monitoring and review metric naming scheme(s)
• Configure node_exporter network monitoring and review metric naming scheme(s) Prometheus metric naming schema is not relevant since the target dashboard is based on Telegraf metric names.

Out of scope:

• MTU healthcheck is replaced by an MTU metric and output_metric_thresholds
• Add support for darwin (macOS) (subset of total available linux metrics)
• Add support for bsd/openbsd (subset of total available linux metrics)
• Add support for Windows (subset of total available linux metrics)
• Bootstrap a separate plugin for network protocol metrics (e.g. TCP/UDP); including check-banner.rb, check-multicast-groups.rb, check-netstat-tcp.rb, check-ports-bind.rb, check-ports.rb
• Bootstrap a separate plugin for DNS; including check-rbl.rb, check-whois-*.rb
• Bootstrap a separate plugin for conntrack metrics

We may need to optionally produce a metric named host_net with a field name label called field, because the new Sumo Logic "Host Metrics" dashboard is based on Telegraf metrics (which are intended for InfluxDB, which supports fields as a distinct concept from tags).

Example:

# HELP host_net network interface metrics
# TYPE host_net counter
host_net{field="bytes_sent", interface="eth0"} 4.669667921e+09 1642040114000
host_net{field="packets_sent", interface="eth0"} 8.096549e+06 1642040114000
host_net{field="err_out", interface="eth0"} 29837 1642040114000
host_net{field="drop_out", interface="eth0"} 31562 1642040114000
host_net{field="bytes_recv", interface="eth0"} 6.667562846e+09 1642040114000
host_net{field="packets_recv", interface="eth0"} 1.0941417e+07 1642040114000
host_net{field="err_in", interface="eth0"} 0 1642040114000
host_net{field="drop_in", interface="eth0"} 0 1642040114000