This repository contains a complete example that grabs device data from The Things Network, stores it in a database, and then displays the data using a web-based dashboard.
You can set this up on a "Ubuntu + Docker" VM from the Microsoft Azure store (or on a Ubuntu VM from DreamCompute) with minimal effort. You should set up this service to run all the time so as to capture the data from your devices; you then access the data at your convenience using a web browser.
This example uses docker-compose to set up a group of four docker containers:
- An instance of Apache, which proxies the other services, handles access control, gets SSL certificates from Let's Encrypt, and faces the outside world.
- An instance of Node-RED, which processes the data from the individual nodes, and puts it into the database.
- An instance of InfluxDB, which stores the data as time-series measurements with tags.
- An instance of Grafana, which gives a web-based dashboard interface to the data.
To make things more specific, most of the description here assumes use of Microsoft Azure. However, I have tested this on Ubuntu 16 LTS without difficulty (apart from the additional complexity of setting up apt-get
to fetch docker, and the need for a manual install of docker-compose
), and on DreamCompute. I believe that this will work on any Linux or Linux-like platform that supports docker, docker-compose, and node-red. It's likely to run on a Raspberry Pi 2, and it might even run on a C.H.I.P. computer... but as of this writing, this has not been tested.
- The host system is the system running Docker and Docker-compose.
- A container is one of the virtual systems running under Docker on the host system.
- A file on the host is a file on the host system (typically not visible from within the container(s).
- A file in container X (or a file in the X container) is a file in a filesystem associated with containser X (and typically not visible from the host system).
All communication with the Apache server are encrypted using SSL with auto-provisioned certificates from Let's Encrypt. Grafana is the primary point of access for most users, and Grafana's login is used for that purpose.
Access to Node-RED and InfluxDB is via special URLs (base/node-red/ and base/influxdb/, where base is the URL served by the Apache container). These URLs are protected via Apache htpasswd
and htgroup
file entries. These entries are files in the Apache container, and must be manually edited by an administrator.
The initial administrator's login password for grafana must be initialized prior to starting the very first; it's stored in grafana/.env
. (When you start the grafana container the first time, it creates grafana.db
in the grafana container, and stores the password at that time. If grafana.db
already exists, the password in grafana/.env
is ignored.)
Microsoft Azure, by default, will not open any of the ports to the outside world, so you will need to open port 443 for SSL access to Apache.
For concreteness, the following table assumes that base is server.example.com
.
To access | Open this link | Notes |
---|---|---|
Node-RED | https://server.example.com/node-red/ | Port number is not needed and shouldn't be used. |
InfluxDB administrative page | https://server.example.com/influxdb/ | Port number is not needed and shouldn't be used. |
Grafana | https://server.example.com | Port number is not needed and shouldn't be used. |
This can be visualized as below:
- Your host system must have docker-compose 1.9 or later (for which see https://github.com/docker-compose -- be aware that apt-get normally doesn't grab this; if configured at all, it frequently gets an out-of-date version).
- The environment variable
TTN_DASHBOARD_DATA
, if set, points to the common directory for your data. If not set, docker-compose will quit at startup. (This is by design!)${TTN_DASHBOARD_DATA}node-red
will have your local Node-RED data.${TTN_DASHBOARD_DATA}influxdb
will have your local influxdb data (this is what you should back up)${TTN_DASHBOARD_DATA}grafana
will have your dashboards
Within their containers, the individual programs use their usual ports, but these are isolated from the outside world, except as specified by docker-compose.yml
.
In docker-compose.yml
, the following ports on the docker host are connected to the individual programs.
- Apache runs on 80 and 443. (All connections to port 80 are redirected to 443 using ssl).
Remember, if your server is running on a cloud platform like Microsoft Azure or AWS, you need to check the firewall and confirm that the ports are open to the oustide world.
These directions are incomplete. Please also refer to SETUP.md
- Make sure your server has
git
,docker
anddocker-compose
installed, with the versions mentioned above under Assumptions. - Use
git clone
to copy this repository to your host. - Create a
.env
file in your repository with the following settings:TTN_DASHBOARD_DATA=/full/path/to/directory/
The trailing slash is required! This will put all the data file for this instance as subdirectories of the specified path. If you leave this underfined,docker-compose
will print error messages and quit.TTN_DASHBOARD_APACHE_FQDN=myhost.example.com
This sets the name of your resulting server. It tells Apache what it's serving out. It must be a fully-qualified domain name (FQDN) that resolves to the IP address of the container host.TTN_DASHBOARD_CERTBOT_FQDN=myhost.example.com
This should be the same asTTN_DASHBOARD_APACHE_FQDN
.[email protected]
This sets the contact email for Let's Encyrypt. The script automatically accepts the Let's Encrypt terms of service, and this indicates who is doing the accepting.TTN_DASHBOARD_GRAFANA_ADMIN_PASSWORD=SomethingVerySecretIndeed
This sets the initial password for the Grafanaadmin
login. You should change this via the Grafana UI after booting the server.TTN_DASHBOARD_GRAFANA_SMTP_FROM_ADDRESS
This sets the Grafana originating mail address.TTN_DASHBOARD_GRAFANA_INSTALL_PLUGINS
This sets a list of Grafana plugins to install.TTN_DASHBOARD_INFLUXDB_INITIAL_DATABASE_NAME=demo
Change "demo" to the desired name of the initial database that will be created in InfluxDB.TTN_DASHBOARD_MAIL_HOST_NAME=myhost.example.com
This sets the name of your mail server. Used by Postfix.TTN_DASHBOARD_MAIL_DOMAIN=example.com
This sets the domain name of your mail server. Used by Postfix.
Your .env
file should look like this:
# setup for this docker-ttn-dashboard instance, used by docker-compose
TTN_DASHBOARD_DATA=/full/path/to/directory/
TTN_DASHBOARD_APACHE_FQDN=myhost.example.com
TTN_DASHBOARD_CERTBOT_FQDN=myhost.example.com
[email protected]
TTN_DASHBOARD_GRAFANA_ADMIN_PASSWORD=SomethingVerySecretIndeed
[email protected]
TTN_DASHBOARD_GRAFANA_INSTALL_PLUGINS=grafana-worldmap-panel,grafana-clock-panel,grafana-piechart-panel
TTN_DASHBOARD_INFLUXDB_INITIAL_DATABASE_NAME=demo
TTN_DASHBOARD_MAIL_HOST_NAME=myhost.example.com
TTN_DASHBOARD_MAIL_DOMAIN=example.com
### end of file ###
- Setup Apache:
- Follow instructions from
SETUP.md
- Follow instructions from
% docker-compose up
or% docker-compose up -d
- If this fails (for example, Node-RED dies with a "killed" status), check your
.env
file.
- If this fails (for example, Node-RED dies with a "killed" status), check your
- Open Grafana on https://myhost.example.com, and log in as admin. Change your password. Set up your first data source using InfluxDB. See below for more detailed instructions.
- Open Node-RED on https://myhost.example.com/node-red/, and build a flow that stores data in InfluxDB. Be sure to add the trailing slash! Otherwise you'll get a 404 from Grafana. We'll fix this soon.
When designing this collection of services, we had to decide where to store the data files. We had two choices: keep them inside the docker containers, or keep them in locations on the host system. The advantage of the the former is that everything is reset when you rebuild the docker images. The disavantage of the former is that you lose all your data when you rebuild. On the other hand, there's another level of indirection when keeping things on the host, as the files reside in different locations on the host and in the docker containers.
Data files are kept in the following locations by default.
Component | Data file location on host | Location in container |
---|---|---|
Node-RED | ${TTN_DASHBOARD_DATA}node-red |
/data |
InfluxDB | ${TTN_DASHBOARD_DATA}influxdb |
/data |
Grafana | ${TTN_DASHBOARD_DATA}grafana |
/var/lib/grafana |
As shown, you can easily change locations on the host (e.g. for testing). You do this by setting the environment variable TTN_DASHBOARD_DATA
to the absolute path to the containing directory prior to calling docker-compose up
. The above paths are appended to the value of TTN_DASHBOARD_DATA
. Directories are created as needed. Consider the following example:
% export TTN_DASHBOARD_DATA=/dashboard-data/
% docker-compose up -d
In this case, the data files are created in the following locations:
Component | Data file location |
---|---|
Node-RED | /dashboard-data/node-red |
InfluxDB | /dashboard-data/influxdb |
Grafana | /dashboard-data/grafana |
Since data files on the host are not removed between runs, as long as you don't remove the files between runs, your data will preserved.
Sometimes this is inconvienient, and you'll want to remove some or all of the
data. For a variety of reasons, the data files and directories are created owned by root, so you must use the sudo
command to remove the data files. Here's an example of how to do it:
% sudo rm -rf ${TTN_DASHBOARD_DATA}node-red
% sudo rm -rf ${TTN_DASHBOARD_DATA}influxdb
% sudo rm -rf ${TTN_DASHBOARD_DATA}grafana
This version requires that you set up Node-RED, the database and the grafana dashboards manually, but we hope to add a reasonable set of initial files in a future release.
There is one point that is somewhat confusing about the connections from Node-RED and Grafana to InfluxDB. Even though InfluxDB is running on the same host, it is logically running on its own virtual machine (created by docker). Because of this, Node-RED and Grafana cannot use localhost when connecting to Grafana. A special name is provided by docker: influxdb. Note that there's no DNS suffix. If you don't use influxdb, Node-RED and Grafana will not be able to connect.
- On the login screen, the user name is "admin". The initial password is given by the value of the variable
GF_SECURITY_ADMIN_PASSWORD
ingrafana/.env
. Note that if you change the password ingrafana/.env
after the first time you launch the grafana containder, the admin password does not change. If you somehow lose the previous value of the admin password, and you don't have another admin login, it's very hard to recover; easiest is to removegrafana.db
and start over.
- Set the URL (under Http Settings) to
http://influxdb:8086
. - Select the database. There's a default database called "demo", which is always created. (This is determined by the file
influxdb/.env
; if you change that file, you can rename the default database.) - Leave user and password blank.
- Click "Save & Test".
Although the dashboard is already very useful, it's incomplete. Please refer to TODO.md
, and also note that we're considering the following. Check in for updates!
- Add a script to setup the passwords initially for grafana and for access to node-red and influxdb.
- Admin script to show roles and maintain the htpasswd
- Add the auto-update cron script -- right now you have to restart in order to get the SSL certs updated. Not a big deal, as the patches-requiring-reboot interval is shorter than the life of the certs, but still, this should be fixed.
- Switch to phusion for the base image, instead of ubuntu.
- Provide suitable intial files for Grafana and NodeRed, assuming Catena-4450 sensor node.
- The intial script should prompt for the data base name.
This builds on work done by Johan Stokking of The Things Network for the staging environment. Additional adaptation done by Terry Moore of MCCI.