schweikert / fping-exporter Goto Github PK

prometheus exporter to measure network latency

License: Apache License 2.0

Go 90.41% Shell 4.90% Makefile 2.37% Dockerfile 2.32%

fping-exporter's Introduction

fping-exporter

fping-exporter allows you to run measure network latency using fping and prometheus. Compared to blackbox-exporter, it gives you additionally latency distribution and a packet loss statistics. Also, it is probably better performing thanks to fping.

WARNING: This is currently a work in progress, the code is not production-ready yet

This graph shows the fping_rtt summary as "SmokePing"-like graph in Grafana:

Usage

Start fping-exporter as follows:

  fping-exporter [OPTIONS]

Application Options:
  -l, --listen=[HOST]:PORT    Listen address (default: :9605)
  -p, --period=SECS           Period in seconds, should match Prometheus scrape interval (default: 60)
  -f, --fping=PATH            Fping binary path (default: /usr/bin/fping)
  -c, --count=N               Number of pings to send at each period (default: 20)

Help Options:
  -h, --help                  Show this help message

Configure Prometheus to use this, as you would with blackbox-exporter. For example:

global:
  scrape_interval: 60s

scrape_configs:
  - job_name: test
    metrics_path: /probe
    static_configs:
    - targets:
      - "8.8.4.4"
      - "8.8.8.8"
    relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: 127.0.0.1:9605  # The fping-exporter's real hostname:port.

Metrics

fping-exporter produces the following metrics:

fping_sent_count: Number of sent probes
fping_lost_count: Number of lost probes
fping_rtt_count: Number of measured latencies (successful probes)
fping_rtt_sum: Sum of measured latencies
fping_rtt: Summary of measured latencies

fping-exporter's People

Contributors

Stargazers

Watchers

fping-exporter's Issues

Autobuild on release tag

Hello,

It would be cool if the CI can build the binary for each platform to avoid build on server.
It's really easy to setup on github CI with goreleaser I can create a pull request if you are okay with it.

Best regards,

Possibility to add more args and generate a logfile

Hi,

Could you add the "-B", "-r", "-i" and "-t" options, please ?
Could you also add an option to generate a logfile, please ?

Thanks in advance for your support.

No fping_* metrics, no errors on Ubuntu 20.04

I've set up fping-exporter on an Ubuntu 20.04 machine.

fping is installed and available at /usr/bin/fping.

root@prometheus:/etc/prometheus# apt policy fping
fping:
  Installed: 4.2-1
  Candidate: 4.2-1
  Version table:
 *** 4.2-1 500
        500 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages
        100 /var/lib/dpkg/status
root@prometheus:/etc/prometheus# file /usr/bin/fping
/usr/bin/fping: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=51b0f1cefe44ccb7f091e6d0135b00a45bcaec42, for GNU/Linux 3.2.0, stripped
root@prometheus:/etc/prometheus# getcap /usr/bin/fping
/usr/bin/fping = cap_net_raw+ep

I've tried both the latest release and 7aa628e, Prometheus scrapes them and reports success, but no fping_ metrics are returned.

I've tried sniffing for the packets that are supposed to go out, i.e. tcpdump -ni enp9s0 host 8.8.4.4, but no packets every leave the machine.

I'm running fping-exporter under systemd, but even running it as root on the terminal does not change anything, or log anything.

# /etc/systemd/system/fping-exporter.service
[Unit]
Description=Prometheus fping Exporter
After=network.target

[Service]
User=prometheus-fping
ExecStart=/var/lib/prometheus-fping/fping-exporter_linux_amd64
# or for the current HEAD
# ExecStart=/var/lib/prometheus-fping/go/bin/fping-exporter

[Install]
WantedBy=multi-user.target

I have verified, that the prometheus-fping user can use fping:

# tcpdump -ni ens192 host 8.8.4.4 &
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
# sudo -u prometheus-fping fping 8.8.4.4
11:58:53.721751 IP W.X.Y.Z > 8.8.4.4: ICMP echo request, id 23267, seq 0, length 64
11:58:53.722827 IP 8.8.4.4 > W.X.Y.Z: ICMP echo reply, id 23267, seq 0, length 64
8.8.4.4 is alive

My scrape_config looks pretty innocent:

  - job_name: fping
    scrape_interval: 60s
    static_configs:
      - targets:
        - 8.8.4.4
        - 8.8.8.8
    relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: 127.0.0.1:9605

I even straced the running process, and this is how a single HTTP request looks like:

And I think there is something in there, that would explain this.

fping-exporter dies with message "Error parsing fping output: ICMP Redirect …"

It looks like fping-exporter does not know how to handle "Error parsing fping output: ICMP Redirect …" from fping.

Resolvable host which is down/does not respond to ping

Hi,

While doing different tests, I noticed that when a host is unreachable it only pings the last host in the Prometheus list or even no host at all some times.

mai 03 11:07:33 TIG fping-exporter[29254]: fping error (exit: 2)2020/05/03 11:07:33 new target: host05
mai 03 11:08:11 TIG fping-exporter[29254]: start fping:  [-q -p 30000 -C 5 -i 1 host01 host02 host03 host04 host05]
mai 03 11:13:11 TIG fping-exporter[29254]: fping error (exit: 2)start fping:  [-q -p 30000 -C 5 -i 1 host01 host02 host03 host04 host05]
mai 03 11:18:11 TIG fping-exporter[29254]: fping error (exit: 2)start fping:  [-q -p 30000 -C 5 -i 1 host01 host02 host03 host04 host05]
mai 03 11:23:11 TIG fping-exporter[29254]: fping error (exit: 2)start fping:  [-q -p 30000 -C 5 -i 1 host01 host02 host03 host04 host05]
mai 03 11:28:11 TIG fping-exporter[29254]: fping error (exit: 2)start fping:  [-q -p 30000 -C 5 -i 1 host01 host02 host03 host04 host05]

From man of Fping (link) :

DIAGNOSTICS
Exit status is 0 if all the hosts are reachable, 1 if some hosts were unreachable, 2 if any IP addresses were not found, 3 for invalid command line arguments, and 4 for a system call failure.

PS: I modified the code to have an interval of 1s for my needs.

Allow setting of username/password to accessfping-exporter endpoint

Allow setting of username/password to access fping-exporter endpoint for security

changing number of probes (-c, --count=N) leads to broken graphs

I changed the probe counter -c from default 20 to 30 and the graphs are broken after that.

Between 11:18 and 11:46 the counter was set to 30 instead of 20. Looks like some problems with "filling" the quantile metrics?

Retrieve Targets is too long and reacts weirdly

Between the moment I run fping-export and the moment it runs the ping it takes a long time just to find 5 targets. As shown below it took 5 minutes just for 5 targets...

And oddly in the process of retrieving the targets, I see that it starts by pinging the first host and then it keeps on scanning for targets.

mai 04 12:40:44 TIG fping-exporter[33069]: 2020/05/04 12:40:44 New worker (period: 5m0s )
mai 04 12:40:44 TIG fping-exporter[33069]: 2020/05/04 12:40:44 new target: host01
mai 04 12:40:45 TIG fping-exporter[33069]: start fping:  [-q -p 30000 -C 5 -i 1000 host1]
mai 04 12:41:30 TIG fping-exporter[33069]: 2020/05/04 12:41:30 new target: host2
mai 04 12:42:34 TIG fping-exporter[33069]: 2020/05/04 12:42:34 new target: host3
mai 04 12:43:10 TIG fping-exporter[33069]: 2020/05/04 12:43:10 new target: host4
mai 04 12:43:24 TIG fping-exporter[33069]: 2020/05/04 12:43:24 new target: host5
mai 04 12:45:45 TIG fping-exporter[33069]: start fping:  [-q -p 30000 -C 5 -i 1000 host1 host2 host3 host4 host5]

distribution of probes within time period in worker.go

The delay between probes to an individual target is calculated here:

fmt.Sprintf("%.0f", w.spec.period.Seconds()*500/float64(w.spec.count))

Based on 60 seconds it will set the delay to 1500 ms and with 20 probes, that will be 30 seconds. Tcpdump shows that it starts pinging (after initial delay) in the first 30seconds.

Is there any intention why "500" is used as metric (1000 for ms would be correct), so only the first 30 seconds are used for probing?

I'm trying to find a solution for a continuous probing, so all 20 probes are send within the 60 seconds.

implement automated testing

we need automated testing, preferably using travis-ci

How to graph loss packet ?

Hi,

Well done for your work, it looks promising, I'm currently looking for an alternative solution for smokeping.

With your project, would it be possible to graph packet loss as well ? In the style of smokeping.

BR.

Build fails in Go 1.13 due to older golang-lint presense in mod file

The project doesn't build under Go 1.13 out of the box due to an incompatible/older golangci-lint dependency in the mod file (golangci/golangci-lint#652)

$ git clone [email protected]:schweikert/fping-exporter.git
Cloning into 'fping-exporter'...
remote: Enumerating objects: 40, done.
remote: Counting objects: 100% (40/40), done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 40 (delta 14), reused 32 (delta 9), pack-reused 0
Receiving objects: 100% (40/40), 278.57 KiB | 352.00 KiB/s, done.
Resolving deltas: 100% (14/14), done.
$ cd fping-exporter/
$ go build
go: github.com/golangci/[email protected] requires
	github.com/go-critic/[email protected]: invalid pseudo-version: does not match version-control timestamp (2019-05-26T07:48:19Z)

Use of UDP Echo instead of ICMP Echo

Because we ran into network errors, that were not showing up in ICMP tests in some scenarions with Equal Cost Multipath, Portchannels/Aggregate Ethernet Channels, Virtual Chassis Fabrics and hardware based forwarding on network devices, we're also doing some regular probes against some echo servers in our network. I think it's not showing up with ICMP probably would have something to do with the hashing and/or queuing behavior on these devices and probably simply the large number of available paths.

The source addresses used for the UDP echo monitoring traffic are spoofing protected to avoid the abuseability for DoS. But since TCP with it's retransmission behavior isn't really compareable to ICMP Pings, UDP echo was the only comparable thing for this purpose.

The echo services are currently probed by distributed smokeping instances using echoping.

Now some colleagues thought about using the fping-exporter instead of the distributed smokeping. But the fping-exporter of course only supports the probing via ICMP echo at the moment. Would it be thinkable to support probing against UDP echo (with a randomized source port) also?