Giter Club home page Giter Club logo

ffmap-backend's Introduction

Data for Freifunk Map, Graph and Node List

Build Status

ffmap-backend gathers information on the batman network by invoking :

  • batctl (might require root),
  • alfred-json and
  • batadv-vis

The output will be written to a directory (-d output).

Copy mkmap.sh-example to mkmap.sh and adapt to your needs, test backend.py for example with:

backend.py -d /path/to/output -a /path/to/aliases.json --vpn ae:7f:58:7d:6c:2a d2:d0:93:63:f7:da

Run backend.py --help for a quick overview of all available options.

For the script's regular execution add the following file to cron in /etc/cron.d/ffmap-backend:

[email protected]
# Freifunk Map Updates
PATH=/usr/sbin:/usr/bin:/sbin:/bin
* * * * *       root nice -n 19 /path/to/mkmap.sh > /dev/null 2>&1
# EOF

Dependencies

Install

on debian jessie:

apt-get install python3-networkx cmake libjansson-dev zlib1g-dev

on debian wheezy:

apt-get install cmake libjansson-dev zlib1g-dev
pip-3.2 install networkx

Running as unprivileged user

Some information collected by ffmap-backend requires access to specific system resources.

Make sure the user you are running this under is part of the group that owns the alfred socket, so alfred-json can access the alfred daemon.

# ls -al /var/run/alfred.sock
srw-rw---- 1 root alfred 0 Mar 19 22:00 /var/run/alfred.sock=
# adduser map alfred
Adding user `map' to group `alfred' ...
Adding user map to group alfred
Done.
$ groups
map alfred

Running batctl requires passwordless sudo access, because it needs to access the debugfs to retrive the gateway list.

# echo 'map ALL = NOPASSWD: /usr/sbin/batctl' | tee /etc/sudoers.d/map
map ALL = NOPASSWD: /usr/sbin/batctl
# chmod 0440 /etc/sudoers.d/map

That should be everything. The script automatically detects if it is run in unprivileged mode and will prefix sudo where necessary.

Data format

nodes.json

{ 'nodes': {
    node_id: { 'flags': { flags },
               'firstseen': isoformat,
               'lastseen': isoformat,
               'nodeinfo': {...},         # copied from alfred type 158
               'statistics': {
                  'uptime': double,       # seconds
                  'memory_usage': double, # 0..1
                  'clients': double,
                  'rootfs_usage': double, # 0..1
                  'loadavg': double,
                  'gateway': mac
                }
             },
    ...
  }
  'timestamp': isoformat
}

flags (bool)

  • online
  • gateway

Old data format

If you want to still use the old ffmap-d3 front end, you can use the file ffmap-d3.jq to convert the new output to the old one:

jq -n -f ffmap-d3.jq \
    --argfile nodes nodedb/nodes.json \
    --argfile graph nodedb/graph.json \
    > nodedb/ffmap-d3.json

Then point your ffmap-d3 instance to the ffmap-d3.json file.

Removing owner information

If you'd like to redact information about the node owner from nodes.json, you may use a filter like jq. In this case, specify an output directory different from your webserver directory, e.g.:

./backend.py -d /ffmap-data

Don't write to files generated in there. ffmap-backend uses them as its database.

After running ffmap-backend, copy graph.json to your webserver. Then, filter nodes.json using jq like this:

 jq '.nodes = (.nodes | with_entries(del(.value.nodeinfo.owner)))' \
   < /ffmap-data/nodes.json > /var/www/data/nodes.json

This will remove owner information from nodes.json before copying the data to your webserver.

ffmap-backend's People

Contributors

baldo avatar codingcatgirl avatar jplitza avatar mweinelt avatar neocturne avatar rubo77 avatar sargon avatar smoe avatar stebifan avatar t-x avatar thisco-de avatar wklaebe avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ffmap-backend's Issues

ZeroDivisionError Exception

lib/nodes.py throws ZeroDivisionError. Somehow d['total'] seems not to be a decimal sometimes - maybe when nodes do not provide the information

Here is the trace:

Traceback (most recent call last):
 File "./backend.py", line 170, in <module>
 main(options)
 File "./backend.py", line 90, in main
 nodes.import_statistics(nodedb['nodes'], alfred.statistics())
 File "/srv/ffmap-backend-new-troisdorf/lib/nodes.py", line 74, in import_statistics
 lambda d: 1 - d['free'] / d['total'])
 File "/srv/ffmap-backend-new-troisdorf/lib/nodes.py", line 61, in add
 statistics))
 File "/srv/ffmap-backend-new-troisdorf/lib/nodes.py", line 74, in <lambda>
 lambda d: 1 - d['free'] / d['total'])
ZeroDivisionError: division by zero

Simple solution (replace line 74 with):

try:
            add(node, stats, 'memory_usage', ['memory'], lambda d: 1 - d['free'] / d['total'])
except ZeroDivisionError:
            add(node, stats, 'memory_usage', ['memory'], lambda d: 0)

Export GeoData in GeoJSON Format

We should export geo informations in GeoJSON format so we can exchange the Map representation in the frontend to any GeoJSON compatible and have a way to import
the dataset in any GeoJSON Framework, e.g. GoogleEarth, OSM.

Blacklist for some nodes or coordinates for the map

I can think of three reasons, why you want to hide the coordinates from the map:

We would need a functionality to blacklist certain nodes on the map for cases

  1. the node is too far away from the community center so you don't want to stretch the map so far to include this node
  2. Some third party legal request forces the community to hide the coordinates from public
  3. the coords are at the wrong place so they create wrong links all across the area that don't really exist and you cannot reach the owner to correct them or the node is placed in a special place where there is no access anymore to.

solutions for those:

  1. the first comment by @mweinelt: ffnord/ffmap-d3#112 (comment)

    It could exclude coordinates for nodes that are farther away than $x km from a given center. The center could also be determined automatically by the use of basic stochastic.

    Someone would have to build that logic though. And maybe present a warning somewhere about bogus coordinates.

  2. Remove the illegal coordinates completely from the generated json data
  3. Those coordinates should be removed from the data too, but only as long as the owner doesn't correct them (also in the backend), so If they are corrected one day, they re-appear on the map

so the blacklist would have to have two options:

  • remove completely
  • remove as long as they are exactly long=... and lat=...

(This was preciedingly posted at ffnord/ffmap-d3#112)

usability: averaging of client number confuses users

Some users of the meshviewer installation from Freifunk Chemnitz seemed to be very confused by fractional numbers for connected clients. It doesn't seem to be easy to understand that the shown values are averages over a timespan.

I personally have no real problem with the current way things are shown. But the persons which had a problem with the current way it is shown haven't yet followed my suggestion to discuss their problems with the presentation with upstream. So I try to step in here to get a discussion with upstream going. Hopefully, some of the other people join the discussion

Right now a node for example looks like this:

test

The fractional parts could easily be removed again by a rounding function like CEIL for the client number

rrdtool 'graph' /opt/freifunk/meshviewer/data/nodes/test_ceil.png \
    '-s' '-7d' \
    '-w' '800' \
    '-h' '400' \
    '-Y' \
    'DEF:clients=/opt/freifunk/meshviewer/data/nodedb/a0f3c1984ca8.rrd:clients:AVERAGE' \
    'DEF:upstate=/opt/freifunk/meshviewer/data/nodedb/a0f3c1984ca8.rrd:upstate:AVERAGE' \
    'VDEF:maxc=clients,MAXIMUM' \
    'CDEF:c=0,clients,ADDNAN,CEIL' \
    'CDEF:d=upstate,UN,0,upstate,IF,0,EQ,clients,UN,MAX,maxc,UN,1,maxc,IF,CEIL,1,MAX,*' \
    'AREA:c#0F0:up\\l' \
    'AREA:d#F00:down\\l' \
    'LINE1:c#00F:clients connected\\l'

test_ceil

This is just an adjustment in the representation. But for example @simonwunderlich seemed to like the idea more to change the CF settings of the clients RRA in lib/NodeRRD.py to MAX. I personally haven't tested this yet but this could also be a way to represent the number of simultaneously connected clients without ending up with fractional clients.

Maybe this was already discussed in the past or someone else has a better idea how to represent clients with the currently available data.


@ambassador86 @deusama This could be relevant for you because it is about data from your server

Preserve roaming when obscure clients

Currently obscuring client mac destroys the roaming feature of ffmap, we should find a way to preserve this feature.

I propose that we should detect roaming in the obscuring code and handle such clients differently.

Make output filterable using libjq

There is library version of the well-known JSON processor jq, and there are python bindings for it. It would be great to specify a jq filter on the command line or in a file which is applied to the/every output JSON file or alternatively to the JSON input from the nodes.

This would solve #78, #44 and all other problems where some kind of modification is required. Those are trickling in one by one and are each asking for a separate option which will blow up the code for some rare use cases.

make backend.py a daemon

Making backend.py a daemon will make additional feature (e.g. gluon-announced support) a lot easier.

support node down areas for graphs over longer timespan

It was noticed that the downtime marks are missing for graphs over longer timespans (e.g. over 7 days instead of just 1d). This seems to be caused by the averaging function of the rrdtool.

Right now the down time area is defined as

'CDEF:d=clients,UN,maxc,UN,1,maxc,IF,*',

So d is defined as max(1,maxc) whenever clients is not defined. But clients seems to become a defined value when the averaging happens.

Right now I am using upstate as workaround for this:

 'CDEF:d=upstate,UN,0,upstate,IF,0,EQ,clients,UN,MAX,maxc,UN,1,maxc,IF,1,MAX,*'

This might have other downsides and thus I haven't created a pull request for this change. The full testcall used right now is:

rrdtool 'graph' /opt/freifunk/meshviewer/data/nodes/test.png \
    '-s' '-7d' \
    '-w' '800' \
    '-h' '400' \
    'DEF:clients=/opt/freifunk/meshviewer/data/nodedb/f4f26dc1f80a.rrd:clients:AVERAGE' \
    'DEF:upstate=/opt/freifunk/meshviewer/data/nodedb/f4f26dc1f80a.rrd:upstate:AVERAGE' \
    'VDEF:maxc=clients,MAXIMUM' \
    'CDEF:c=0,clients,ADDNAN' \
    'CDEF:d=upstate,UN,0,upstate,IF,0,EQ,clients,UN,MAX,maxc,UN,1,maxc,IF,1,MAX,*' \
    'AREA:c#0F0:up\\l' \
    'AREA:d#F00:down\\l' \
    'LINE1:c#00F:clients connected\\l'

Here are is an example from node 60e327fb5176 (before the change, after the change)

original
patched
testdata.zip

running as unprivileged user doesn't work as documented

running ffmap-backend on Debian 8 as unprivileged user doesn't work like it is documented.
as far as i remember it was strange, with the solution from the readme, the alfred socket access wasn't possible most of the time, worked only once in a while

we solved it by

  1. creating wrapper scripts that run batctl, batadv-vis, alfred-json via sudo
  2. create sudo entries for these
  3. change the PATH for the cronjob to include the directory containing the wrapper scripts
  4. change backend.py to copy/use the parent environment variables, see tecff@15553b8 and tecff@a2a5fcb

maybe there's a better way, but the readme should be updated regarding this

Backend Error

Hi,

nachder Installation erhalten wir folgenden Fehler bei der Ausführung des Crons:

Traceback (most recent call last):
  File "./backend.py", line 185, in <module>
    main(options)
  File "./backend.py", line 117, in main
    graph.import_vis_data(batadv_graph, nodedb['nodes'], vd)
  File "/root/ffos/ffmap-backend/lib/graph.py", line 17, in import_vis_data
    *zip(*chain(nodes_a, nodes_b))))
TypeError: map() must have at least two arguments.

Missing geo.latitude produces errors

If a FF-node hat only longtitude but no latitude geo-location. The code fails with this error:
Traceback (most recent call last):
File "/home/fabian/ffmap-backend/backend.py", line 169, in
main(options)
File "/home/fabian/ffmap-backend/backend.py", line 134, in main
json.dump(export_nodelist(now, nodedb), f)
File "/home/fabian/ffmap-backend/lib/nodelist.py", line 10, in export_nodelist
node_out["position"] = {"lat": node["nodeinfo"]["location"]["latitude"],
KeyError: 'latitude'

The result is no data in nodelist.json and wrong data in nodes.json. The meshviewer-frontend show error instead of a map.

RRD.py vs. rrd.py

Hallo,
die Benennung der beiden Dateien rrd.py sowie RRD.py ist nicht eindeutig und verursacht Probleme auf Computern mit Case-Insensitive Filesystems..
Liegt es im Bereich des Möglichen eine der beiden Dateien umzubenennen, und die entsprechenden Aufrufe darauf anzupassen?

Vielen Dank!

Daten für offline knoten archivieren

Wenn die json datei neu generiert wird, und dabei ein knoten rausfliegt, weil er inzwischen offline gegangen ist, wäre es trotzdem interresant die Daten im json als offline zu behalten, damit man, wenn man auf die Seite linkt mit dem namen als parameter (wenn man von einem link eines knotens auf der karte kommt) auch das Info-Fenster links öffnen kann mit infos zu dem Knoten

add socket support (alfred)

Hello,

recently the alfred daemon has been patched to support the configuration for the unix socket path: Link

After this the alfred client alfred-json has also been patched to support this: Link

This allows communities which have currently multiple batman_adv mesh clouds on the same gateway to run one alfred daemon instance for each of the cloud. One implication is that all alfred clients have to patched in order to connect to the correct socket.

So please add support in ffmap-backend for this. Finally, bat2nodes.py should be called with a new command-line-option like -u | --unix-path (alfred daemon style) or -s | --socket (alfred client alfred-json style) in order to return the wanted unix socket to alfred.py and batman.py to make in the mkmap.sh magically stuff like:

./bat2nodes.py -A -s /var/run/alfred0.sock -a somealiases.json -m bat0 -d $DEST
./bat2nodes.py -A -s /var/run/alfred1.sock -a somealiases.json -m bat1 -d $DEST
./bat2nodes.py -A -s /var/run/alfred2.sock -a somealiases.json -m bat2 -d $DEST

And some time later someone should make some brainwork to consider the implications supporting multiple batman_adv mesh clouds in one single map, like e.g. duplicate gateways, vpn links?, etc.

y-labels unreadable on node with many clients

It was noticed on the meshviewer installation of Freifunk Chemnitz that the nodes with a large number of clients have overlapping y-labels. This was caused by the '-y 1:1' setting for rrdtool

tietz_default_settings

This made them remove this setting. The result looked fine for these nodes.

tietz

But nodes with a smaller number of nodes started to have fractional part

test

This freaked out some people and thus I've started to play around with the setting. @simonwunderlich looked at them and told me that '-Y' seemed to look better to him. Here are the previous examples with '-Y' instead of '-y 1:1'

tietz_better_settings
test_better_settings

It is far from perfect but at least doesn't seem to discriminate any of my examples in favor of the other one. Maybe '-Y' could replace '-y 1:1' in lib/NodeRRD.py. But I would like to get more opinions (or better solutions) before a pull request is made.


@ambassador86 @deusama This seems to be relevant for you because it is about your server

Alias Storage

With upcoming alfred support in gluon the need for a central alias information list decreases. Since we won't have any persistent alias informations at some time in the feature, hence any node going offline will immediately vanish from the map.

We should have some persistent store, where we archive the client meta informations for some time. So the client will be visible for maybe a day or two in the map even if connection broke or it is turned off.

Batman-Interface wird als Client im Knotengraph gezeigt

Beim Hamburger-Knotengraph ist mir aufgefallen, dass bei jedem Node immer mindestens ein verbundener Client angezeigt wird.
Daraufhin habe ich mich ein bisschen umgeschaut und festgestellt, dass dieser Client immer die MAC-Adresse des Batman-Interfaces des Nodes trägt und dass das auch keine offiziell vergebenen MAC-Adressblöcke sind. Beim bauen der Clientliste müssten also alle Clients mit Flag "P" noch aussortiert werden.

Edit: Ich sehe gerade, dass die in den vis-Daten mit TT-Label drin stehen und das P-Flag nur in der local transtable enthalten ist. Dann müssen die irgendwie anders aussortiert werden.

Fehlende Clients

Es scheint, als fehlten Clients auf unserer Installation der FF-Karte: http://map.kbu.freifunk.net/
Hi!

Einige Knoten zeigen ihre Clients nicht an. Ich habe mich mit 2 Clients an einen Knoten (samson) angemeldet und auch nach Stunden sind diese nicht auf der Karte aufgetaucht.

Ich konnte verifizieren, dass das Frontend korrekt arbeitet (also keine Clients im json file stehen), habe aber keinen Zugriff auf das Backend.

Hast Du dazu eine Idee?

Unsere Version ist von vor dem 6.5.2013. Gibt/gab es mal ein Problem in der Richtung?

Gruß aus Bonn,
g3ntleman

graph.json undocumented

I'm trying to build an olsr backend, and while nodes.json is documented in README.md, there's no specs for graph.json, could you please provide at least a working example?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.