I followed your steps and I could get everything working except the Graylog part

I can see the firewall is sending data to graylog on port 1514

`tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes

23:25:56.649243 40:62:31:12:7e:85 > 06:9d:bf:e9:ce:1b, ethertype IPv4 (0x0800), length 233: 192.168.155.254.15740 > 192.168.155.244.1514: UDP, length 191

23:25:56.649245 40:62:31:12:7e:85 > 06:9d:bf:e9:ce:1b, ethertype IPv4 (0x0800), length 255: 192.168.155.254.15740 > 192.168.155.244.1514: UDP, length 213

23:25:56.649245 40:62:31:12:7e:85 > 06:9d:bf:e9:ce:1b, ethertype IPv4 (0x0800), length 255: 192.168.155.254.15740 > 192.168.155.244.1514: UDP, length 213
`
However I dont see anything on the Graylog streams, hence noting on the firewall dashboard on the Grafana.

Graylog is complaining about one error though. Not sure how to resolve it

There is a node without any running inputs. (triggered an hour ago)
There is a node without any running inputs. This means that you are not receiving any messages from this node at this point in time. This is most probably an indication of an error or misconfiguration.You can click here to solve this.

How can I change the Graphana logins?

Changing the 'GF_SECURITY_ADMIN USER/Pass' in the compose.yaml file and recreating the container doesn't work..

I have followed your doc and find that messages are getting processed by graylog, but there are no outputs in grafana firewall panel or browsing the stream "OPNsense / filterlog" in graylog. GeoIP database was installed with direction provided but had to remove the '' in example between commands.

I got every up and running but the map.
I tried to create a new license key and waited for more than 5 mins, using that key to download the new GeoLite2-Country.mmdb, edit execution order in Graylog, and enable all message processors.

Upon importing the Grafana dashboard I encountered an issue with two of the predefined variables: dst_port and src_ip.

This seems to be the reason why none of the Elasticsearch-related panels are showing any data.

The GeoIP curl command to download the database wasn't working for me at all.

I did some looking and found the solution. Looking here the curl command has to look like curl -O -J -L -u YOUR_ACCOUNT_ID:YOUR_LICENSE_KEY 'https://download.maxmind.com/geoip/databases/GeoLite2-Country/download?suffix=tar.gz' -o GeoLite2-Country.tar.gz

With your account_id being available at MaxMind, Login click the "My Account" button at the top and "Account ID" will be in the drop down.

This allowed me to curl the database properly, hope this helps.

Firstly many thanks for your hard work. However after going through your config guide I am getting the following error in the telegraf logs
E! [outputs.influxdb] When writing to [http://localhost:8086]: failed doing req: Post "http://localhost:8086/write?consistency=any&db=telegraf": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Has anyone any idea where I've gone wrong? And more importantly what I would need to do to fix the problem?

I noticed there been some updates to the versions of the containers, so I updated my compose file to reflect this. Mongodb was not happy. I downgraded back to 4.4 to get everything working again. So for anyone already running a 4.4 mongoDB, don't upgrade to 6.0 directly.

Hi there,

telegraf_pfifgw.php reference get_interfaces_info() that looks to be removed from 24.1 (https://forum.opnsense.org/index.php?topic=38504.0) and throw the following error: Fatal error: Uncaught Error: Call to undefined function get_interfaces_info() in /usr/local/bin/telegraf_pfifgw.php:13

Any suggestions?

Thanks,
Vlad

Hello,

I've been working on this dashboard :

based on the data processed by Graylog and stored in elasticsearch

I've added three fields : full country name for both src_ip and dst_ip, and iso country name for dst_ip

It needs the Treemap plugin

and I'm still working on the Action per interface pie chart because I need it to be nested and I don't understand how to do it with Grafana (that's frustrating because I can do it in my sleep with Kibana)

For those of you who want to play with the dashboard, here is the Json for you to import in your grafana setup.
OPNSense trafic detail-1678372350372.json.zip

As of today's update, I get this when telegraf runs the pfifgw.php script.

[01-Mar-2022 16:33:50 America/New_York] PHP Fatal error:  Uncaught Error: Call to undefined function find_interface_network() in /usr/local/bin/telegraf_pfifgw.php:18
    Stack trace:
    #0 {main}
      thrown in /usr/local/bin/telegraf_pfifgw.php on line 18

Maybe its because I've never used Grafana before, but I can't figure out how to login to Grafana from this guide

I've accessed the grafana container and viewed /etc/grafana/grafana.ini and I see the following

• default admin user, created on startup

;admin_user = admin

• default admin password, can be changed before first start of grafana, or in profile settings

;admin_password = admin

but that combination doesn't work when I access the webpage on port 3000, and neither does the login creds for Influxdb or Graylog work on it either.

So I'm obviously missing something obvious here.. but what?

Hello Sir!

Please update this:
curl https://github.com/bsmithio/OPNsense-Dashboard/blob/master/config/telegraf.conf

to this:
curl https://raw.githubusercontent.com/bsmithio/OPNsense-Dashboard/master/config/telegraf.conf

After the upgrade to OPNSense 24.1 the following error messages are shown which results in no Firewall (drops/country/firewall) and Interface information visible in the OPNSense dashboard. The CPU/Memory graphs are fine

I experimented with non-root and root rights for Telegraf in the OPNSense GUI.

Telegraf log error message in OPNSense GUI
E! [inputs.exec] Error in plugin: exec: exit status 255 for command "sudo /usr/local/bin/telegraf_pfifgw.php":

CLI error message in OPNSense
root@firewall:/usr/local/bin # sudo telegraf_pfifgw.php

Fatal error: Uncaught Error: Call to undefined function get_interfaces_info() in /usr/local/bin/telegraf_pfifgw.php:13
Stack trace:
#0 {main}
thrown in /usr/local/bin/telegraf_pfifgw.php on line 13

It looks like some function get_interfaces_info(); has been removed fom require_once("interfaces.inc");

$ifsinfo = get_interfaces_info();

So i put my opnsense into my FAI router DMZ so that every traffic directed at my router goes directly to my opnsense :

I am attacking from a 4G connection my FAI router and it's getting blocked as we can see below :

But on my opnsense grafana dashboard, it doesnt seems to appear :

My WAN interface is vtnet0 while my LAN interface is vtnet1.

Is there something i missed ?

Some pages in graylog were not loading and ended up frozen/in a loop because of ublock origin finding "geoip" within it
Disabling UBlock Origin for the site fixes issue

See: Graylog2/graylog2-server#8470

Hello,

I followed the documentation and there are 2 things to adjust:

Configuring Graylog:
Then download the database file, replace YOUR_LICENSE_KEY with the key you generated above.

Problem: Error write 'GeoLite2-Country.mmdb' to '/usr/share/graylog/data/data/'.

Solution:
sudo docker exec -it graylog /bin/bash
CD /usr/share/graylog/data/data/
chmod 775 GeoLite2-Country.mmdb
rm GeoLite2-Country.mmdb

Execute:
curl "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-Country&license_key=YOUR_LICENSE_KEY&suffix=tar.gz" -o GeoLite2-Country.tar.gz
&& tar -xzvf GeoLite2-Country.tar.gz
&& mv GeoLite2-Country_*/GeoLite2-Country.mmdb /usr/share/graylog/data/data/

Configure Additional Settings:
Now, add your index set from earlier to the "OPNsense / filterlog" stream. Navigate to Streams -> More Actions -> Edit Stream -> select your index set and save.

Problem: Stream 'OPNsense / filterlog' not visible.

Solution:
Open content-pack 'OPNSense Dashboard'
Click 'Install'

Configure Variables
There is no explanation how to change the interfaces, only that you can do it. Maybe some screenshots??

The dashboard is working now and I really like it.

Request:

• How to modify the databases to add support for Docker container monitoring. Earlier I setup Grafana, Prometheus, Node-exporter via this instruction 'https://grafana.com/docs/grafana-cloud/quickstart/docker-compose-linux/' but couldn't make it work with OPNSense Dashboard. How can I make this work side-by-side?

When trying to run the docker compose stack, I get the following errors in the graylog docker logs saying that the graylog.conf file is missing. I was under the impression that I could completely configure graylog with env variables but it seems like it will not start without a conf file.

09:23:31.103 [main] INFO  com.github.joschi.jadconfig.JadConfig - Added configuration bean org.graylog2.configuration.VersionCheckConfiguration@7fc4780b
09:23:31.103 [main] INFO  com.github.joschi.jadconfig.JadConfig - Added configuration bean org.graylog2.plugin.KafkaJournalConfiguration@3b79fd76
09:23:31.103 [main] INFO  com.github.joschi.jadconfig.JadConfig - Added configuration bean org.graylog2.inputs.transports.NettyTransportConfiguration@48c76607
09:23:31.103 [main] INFO  com.github.joschi.jadconfig.JadConfig - Added configuration bean org.graylog.plugins.pipelineprocessor.PipelineConfig@43599640
09:23:31.103 [main] INFO  com.github.joschi.jadconfig.JadConfig - Added configuration bean org.graylog.plugins.views.ViewsConfig@1f81aa00
09:23:31.103 [main] INFO  com.github.joschi.jadconfig.JadConfig - Added configuration bean org.graylog2.system.processing.ProcessingStatusConfig@6b6776cb
09:23:31.103 [main] INFO  com.github.joschi.jadconfig.JadConfig - Added configuration bean org.graylog.scheduler.JobSchedulerConfiguration@1863d2fe
09:23:31.103 [main] INFO  com.github.joschi.jadconfig.JadConfig - Added configuration bean org.graylog.metrics.prometheus.PrometheusExporterConfiguration@1787bc24
09:23:31.103 [main] INFO  com.github.joschi.jadconfig.JadConfig - Added configuration bean org.graylog2.configuration.TLSProtocolsConfiguration@544d57e
09:23:31.103 [main] INFO  com.github.joschi.jadconfig.JadConfig - Added configuration bean org.graylog.plugins.map.config.GeoIpProcessorConfig@55c53a33
09:23:31.103 [main] INFO  com.github.joschi.jadconfig.JadConfig - Added configuration bean org.graylog2.configuration.TLSProtocolsConfiguration@53b7f657
09:23:31.104 [main] ERROR org.graylog2.bootstrap.CmdLineTool - Couldn't load configuration: Properties file /usr/share/graylog/data/config/graylog.conf doesn't exist!

Posting this in case any else runs into same issue.
Noticed I couldn't access graylog while following the tutorial with message "bad gateway".

Issue: My docker host was on Proxmox using cpu type kvm which then causes the mongodb container fail to start:

mongodb        |
mongodb        | WARNING: MongoDB 5.0+ requires a CPU with AVX support, and your current system does not appear to have that!
mongodb        |   see https://jira.mongodb.org/browse/SERVER-54407
mongodb        |   see also https://www.mongodb.com/community/forums/t/mongodb-5-0-cpu-intel-g4650-compatibility/116610/2
mongodb        |   see also https://github.com/docker-library/mongo/issues/485#issuecomment-891991814
mongodb        |
mongodb exited with code 132

Solution: change cpu type to "host" in Proxmox:

see: https://codehammer.io/proxmox-mongodb-5-0-requires-a-cpu-with-avx-support

The n_users metric doesn't get written to InfluxDB and gateway loss is written but is always 0 across all 3 gateways even with 1 running at 100% loss.

All other metrics are written correctly and display correctly. I have gone over all the steps and verified them.

The potential issue with users not being written could be found here:
https://github.com/influxdata/telegraf/blob/master/plugins/inputs/system/README.md
There is no /var/run/utmp and may be related be the cause.

As for the loss, no idea. Any hints to get these metrics working?

Hello,
the configure.md instruct us to configure Elasticsearch but why is this required? As far as I understand only InfluxDB (for general metrics via Telegraf) and Graylog (as a Syslog receiver) is required.

inputs.exec section needed to be modified slightly to allow temperature script to run on my 21.7.7 install:

from:
"/usr/local/bin/telegraf_temperature.sh"

to:
"sh /usr/local/bin/telegraf_temperature.sh"

Love the dashboard and am getting close to getting everything setup and working, but I'm running into an issue that I cannot figure out. I have no gateway stats at all under Network Stats. The Interface Summary for each interface WAN or LAN is blank. WAN interface data is working fine but the LAN Interface just shows - ALL even with regex in place that should cause it to show at least 2 interfaces but it only reports as All so I'm pretty sure something is amiss or missing.
/^(?!enc0$|em0$|igb2$|igb3$)/

I have dual WAN setup, so em0 and igb2; igb3 is an extra interface not currently in use and of course the enc0 interface.

Looking at the data, I'm not seeing any gateway under _measurements, I'm also not seeing a status _field that it looks like the interface summaries need.

Not sure if this has to do with the telegraph error every 10 seconds in the OPNsense log.
2024-03-05T19:18:20Z E! [inputs.exec] Error in plugin: exec: exit status 1 for command "sudo /usr/local/bin/telegraf_pfifgw.php": /usr/local/etc/sudoers:135:55: Alias "PFIFGW" already defined...

Any help would be greatly appreciated.

I have thus far been unable to get this stack working at all.
There two issus which are blocking, and I have been unable to find any solution:

• The graylog container fails to start:

  • 2024-03-18 21:43:50,182 INFO : org.graylog2.bootstrap.preflight.MongoDBPreflightCheck - MongoDB is not available. Retry 18
  • 2024-03-18 21:43:52,184 INFO : org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out
  • I tried adding "GRAYLOG_MONGODB_URI: mongodb://graylog:secret@mongo:27017/graylog" to the graylog env, but no change.
  • I tried adding "MONGO_INITDB_ROOT_USERNAME: graylog" & "MONGO_INITDB_ROOT_PASSWORD: graylog" to the MongoDB container, but no change.
  • No idea where to fom here.

• The installation of telefraf on OPNSense does not work properly. In the configure.md there are instructions to test the install using "sudo telegraf_pfifgw.php". However, this produces:

  • Fatal error: Uncaught Error: Call to undefined function get_interfaces_info() in /usr/local/bin/telegraf_pfifgw.php:13
    Stack trace:
    #0 {main}
    thrown in /usr/local/bin/telegraf_pfifgw.php on line 13

Please let me know if there is any advice on this.

First, I want to thank you for this work. This is the first time I've gotten a dashboard to work because I was always trying to do this on my own without containers. This was so much easier.

A few things I got stuck on that might help others:

• Portainer runs on the same port as graylog, so if you are using this in your container/box, you will need to re-configure the port it uses. Pay attention to the output of docker compose, this is where I saw Graylog failing to bind to port 9000
• The MaxMind API token takes a few minutes to activate, so wait a bit before you run the commands
• Enable all Telegraf monitoring items, in case they are disabled. I had them disabled for some reason
• I missed the "reorder" part of the Graylog configuration. So if you are not getting results in your map, make sure your Message Processors Configuration matches the screenshot

When running telegraf in debug mode I get the follow errors:

Error in plugin: error running pfctl: exit status 1: pfctl: /dev/pf: Permission denied

This was fixed by adding telegraf to sudoers file and adding:
use_sudo = true

after

[[inputs.pf]]

See https://www.reddit.com/r/homelab/comments/9woq6x/opnsense_telegraf_inputpf/ for details.

The metrics for RTT and loss weren't displaying any values. The telegraf_pfifgw.php uses return_gateways_status to query these values.

To enable the query properly go into System > Gateways > Configuration in opnsense and uncheck Disable Gateway Monitoring for each gateway.

Using the command pluginctl -r return_gateways_status you can view the output from this query.

Before

{
    "dpinger": {
        "WAN_DHCP6": {
            "status": "none",
            "monitor": "~",
            "name": "WAN_DHCP6",
            "stddev": "~",
            "delay": "~",
            "loss": "~"
        },
        "WAN_DHCP": {
            "status": "none",
            "monitor": "~",
            "name": "WAN_DHCP",
            "stddev": "~",
            "delay": "~",
            "loss": "~"
        }
    }
}

After

{
    "dpinger": {
        "WAN_DHCP6": {
            "status": "none",
            "monitor": "<Your IPv6>",
            "name": "WAN_DHCP6",
            "stddev": "0.3 ms",
            "delay": "1.2 ms",
            "loss": "0.0 %"
        },
        "WAN_DHCP": {
            "status": "none",
            "monitor": "<Your IP>",
            "name": "WAN_DHCP",
            "stddev": "0.3 ms",
            "delay": "1.0 ms",
            "loss": "0.0 %"
        }
    }
}

Lost a HD, so had to reinstall this from scratch. All is working except elasticsearch. I first noticed this when adding elasticsearch as a data source, it produced the error

Elasticsearch error: Bad Gateway

I tried to connect to the URL via a browser on port 9200 without success. Looking at the container the container is running

but looking at netstat I don't see ports 9200 or 9300 as open

when I look at the logs for the container I see plenty of info messages, but no errors

Am I missing something obvious?

Hi,
As you can see from the preview, i'm missing some informations related to my Firewall.

Ndr. Greylog is not in a container, but in a LXC with UTC timestamp (I've set my opnsense timestamp to UTC, just to be sure). If i check my Firewall log i find a lot of entries

I dot not get interface information when loading the Grafana Dashboard:

When editing a panel and switching from ${datasSource} to my actual data source it starts working:

i followed the steps all the way to where i setup greylog geolocation. when i run the command to download the file with the key i get a error I've tried a few things but I'm not sure what exactly is causing it. i thought maybe something to do with maxmind no longer using legacy keys?. i verified my key works by downloading a config though my browser. please let me know if anyone has any ideas. i google the error and found nothing helpful.

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Hi
I've a few opnsense configured to send data with telegraf and syslog.

If I select one specific Firewall with the filter on top "OPNsense", the Uptime and Firewall will not updatet.
Same with the LAN Interface.

Seems not all Querys have the r.host =~ /^${Host:regex}$/ and in it.

Hi All,

The Active Users Widget showed N/A

"n_users" not found

Kindly advise. TQ

Since I've started using the Suricata dashboard I've noticed that the alerts within the Intrusion Detection part of the OpnSense GUI are no longer listed there. So presumably the action of sending them to Influx or Graylog (I'm presuming Graylog?) results in them being deleted from the system, or at least from that part of the GUI.

This would be fine, except that the Alert Logs part of the Suricata dashboard doesn't tell me whether the traffic was allowed or blocked, or the SID in case I want to tweak the ruleset between alert or drop. This would be a lot of use when troubleshooting issues with specific websites, since without the action information I don't know if its something that was blocked, or noticed but allowed through, and without the SID information finding the specific rule to determine if it is set to alert or drop becomes more difficult (but still possible)

Is this something that I can easily add into the dashboard? Or is there a better approach?

I think the dashboard is really great, but unfortunately, I can't find the right magic with which it would be possible to set an interface other than igb0 for the firewall panel. The relevant OPNsense firewall runs as a VM and has only vtnet instead of igb interfaces.

Perhaps the documentation here is also somewhat inaccurate? It says:
"iface - $iface is the interface variable for the Firewall panels, I have it set to igb0 by default. You can either remove igb0 so you can select all interfaces, or use a comma separated list of interfaces."

Unfortunately, there is no setting in the relevant variable at all that refers to the mentioned default setting igb0.

For those with other than 4 CPU's, Edit the CPU panel in grafana and adjust the line to

r.cpu =~ /cpu[0-7]$/ 

for 8 CPU systems as an example.

Perhaps there is a way to detect from the data, but I'm no expert.

Did anyone configure a dash that reads PfBlockerNG events directly from telegraf?

[[inputs.tail]]
    files = ["/var/log/pfblockerng/ip_block.log"]
    data_format = "grok"
    from_beginning = false
    name_suffix = "_ip_block_log"
    grok_timezone = "Local"
    grok_patterns = ["^%{SYSLOGTIMESTAMP:timestamp:ts-syslog},%{NUMBER:rulenum},%{DATA:interface},%{WORD:friendlyname},%{WORD:action},%{NUMBER:ip_version},%{NUMBER:protocolid},%{DATA:protocol:tag},%{IPORHOST:src_ip:tag},%{IPORHOST:dest_ip:tag},%{WORD:src_port:tag},%{NUMBER:dest_port:tag},%{WORD:direction},%{WORD:geoip_code:tag},%{DATA:ip_alias_name},%{DATA:ip_evaluated},%{DATA:feed_name:tag},%{HOSTNAME:resolvedhostname},%{GREEDYDATA:clienthostname},%{GREEDYDATA:ASN},%{GREEDYDATA:duplicateeventstatus}"]

In the config it asks that you run the command printf 'Defaults!PFIFGW !log_allowed\n' | sudo tee -a /usr/local/etc/sudoers > /dev/null. This however, always returned an error for me PFIFGW: Event not found.

I found that the only way to rectify it was to manually add Defaults!PFIFGW !log_allowed To the sudoers file.

The following needs some extra care, messing with the sudoers file can potentially not go well. Fixing this will just prevent the sudo logs from appearing in the opnsense logging at System > Log Files > General. The solution is as follows.

1. open a root ssh terminal to your opnsense instance
2. Using the command visudo open the sudoers file
   • Its critical you only use visudo to prevent messing with the sudoers file incorrectly
3. Scroll to the bottom using the arrow keys, if you ran the previous commands from the configuration instructions you should see telegraf ALL=(root) NOPASSWD: /usr/local/bin/telegraf_pfifgw.php and Cmnd_Alias PFIFGW = /usr/local/bin/telegraf_pfifgw.php at the bottom
4. Press the I key to enter Insert mode
5. Carefully type out the Defaults!PFIFGW !log_allowed line, copy paste will not work don't try
6. Press Esc Key to enter command mode
7. Type :wq to save and quit the file
8. you can run sudo visudo -c after to ensure it will work

Hope this helps

Im using the compose from here: https://raw.githubusercontent.com/bsmithio/OPNsense-Dashboard/master/docker-compose.yaml
I got everything working in Grafana, except Graylog (and thus the map).

For whatever reason, when I docker compose up, it always fails with:
org.graylog2.bootstrap.preflight.PreflightCheckException: Journal directory </usr/share/graylog/data/journal> has not enough free space (1785 MB) available. You need to provide additional 3334 MB to contain 'message_journal_max_size = 5120 MB'

I tried adding - GRAYLOG_MESSAGE_JOURNAL_MAX_SIZE=10gb to the enviroment table, but it gives the same error, but with the 10gb size

I tried changing volume:
volumes:
- graylog_data:/opt/appdata/influxdb/graylog

But it still gives the error that it goes to </usr/share/graylog/data/journal>

Does anyone have the same issue?

Thanks for great dashboard.
We found that we could apply these simplification with no noticable issues:

1. It is possible to use standard telegraf OPNsense plugin and put the exec input config in /usr/local/etc/telegraf.d/input_exec.conf
2. Run telegraf_pfifgw.php as telegraf user (from OPNsense plugin) seems working perfectly fine, no need to use sudo in command & add to sudoers

I'm not seeing anything in Graylog.

Yet I see UDP traffic on 1514 going to my docker container.

The index also shows plenty of messages inbound.

One thing that I noticed during setup is that stream rule processor is missing. I found documentation that this was forked out of the message filter chain in release 5.0. I have not yet updated Graylog to 5.0.

If samples of other configuration is needed, I will be happy to provide them. I did not yet do so because I feel that the relevance is here within Graylog since I see traffic making it past the edge firewall on my docker host and there is evidence of the messages making it to the container.

Hello,

first of all i want to say thank you for that great dashboard with a great instruction!

Now nearly everything works like a charm. Just the Firewall-Section dont show any data at all.
Is it possible to connect this section with Sensei (Zenarmor) from my OPNsense? If, how?

Best regards

As the title. The Gateway panel shows unavailable and unmonitored
I noticed in another issue Here not adding sudo to telegraf_pfifgw.php the gateway doesn't work. I did exactly whats in the guide. but it's still blank.

Image