Giter Club home page Giter Club logo

adsb-data-collector-mongodb's Introduction

adsb-data-collector-mongodb

An asynchronous Python script to continuously feed ADS-B data from dump1090 to a MongoDB database

What does this do?

This is mainly a single script that runs continuously and collects ADS-B data from your dump1090 instance and stores it in organized MongoDB documents.

  • Works with dump1090-fa and should work with dump1090-mutability too, but it's not tested. Please drop me a line if it works for you.
  • Written solely in Python.
  • Uses Python asynchronous coroutines (through asyncio) to minimize delays from fetching and loading operations.
  • Logging facility available
  • Ready for Pushover notifications in case of errors/failure.

Requirements

  • A configured and running dump1090 instance (see links above). This can be yours or a friend's but you should be able to access it from wherever you intend to run this script, if the script is not running on the same machine as dump1090.
  • MongoDB - If you don't already have it, I recommend setting up a free cluster with MongoDB Atlas.
  • Python 3.7+
  • Python libraries
    • aiohttp
    • Motor: Asynchronous Python driver for MongoDB
    • PyMongo
    • Python-dateutil
    • dnspython

Installation and setup

Download the files to the directory where you want to run it. I recommend having a dedicated directory/folder for this.

git clone https://github.com/dbsoft42/adsb-data-collector-mongodb.git

Install the required Python libraries.

pip3 install aiohttp motor pymongo python-dateutil dnspython

Copy the config_template.py file to config.py.

cp config_template.py config.py

Edit config.py in your favourite text editor and change the following parameters.

  • mongodb_conn_str - The MongoDB connection string. If you are using MongoDB Atlas, you can get this by logging in to Atlas, going to your cluster and clicking on the connect button. Get the one for Python.
  • database_name - The name of the database in MongoDB that you will use to store the ADS-B data. I recommend having a dedicated database for this and not using it for any other data.
  • dump1090_url - This is the dump1090 URL which serves the aircraft.json file. Typically this will be in the form of http://hostname/dump1090/data/aircraft.json where hostname is the host name or IP address of the machine where dump1090 is running. If you will be running this script on the same machine where dump1090 is running, you can leave it as localhost.

The file has more parameters for fine-tuning various operations. You can leave these as the defaults or tune them if you like. The file has comments describing in more detail what each parameter is used for.

Do a quick test run.

python3 adsb-data-collector.py

Let it run for a few seconds (or longer if you wish). If all goes well, you should not see any output on the terminal. Check your MongoDB database to see if new documents are being created in the aircraft, flights and status collections. If yes, you are good to go! Stop the running script with CTRL+C.

To run it for the long term, I suggest running in in the background with nohup as shown below, but you can choose your own method. The script will basically run indefinitely once started.

nohup python3 adsb-data-collector.py &

About Logging

The script supports logging to a file using the standard Python logging package. It is disabled by default and can be enabled from the config.py file. Please see the config file on how to enable it and set the other parameters. The files are rotated such that a new file is created at midnight and the old file is renamed with the date stamp. You can configure how may days of old files you want to keep.

About Pushover Notifications

The script also supports sending log messages as Pushover notifications. So you can set it up to notify you of errors or failures. The feature is disabled by default. To enable it, please download the LogPushoverHandler from here and place the LogPushoverHandler.py file in the same directory as adsb-data-collector.py. Then go to config.py and enable the feature. I recommend you keep the log level to logging.ERROR or logging.CRITICAL for the Pushover notifications.

adsb-data-collector-mongodb's People

Contributors

dbsoft42 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

Forkers

source-code-sa

adsb-data-collector-mongodb's Issues

messages dictionary cleanup is not working

The cleanup is not working as the del is happening inside the loop and this results in the following error.

RuntimeError: dictionary changed size during iteration

The keys to be deleted need to be collected and the deletion has to be done outside the loop. The exception handling is preventing the error from showing up, but this is not working.

Error with Logging

Hello,

I have tried to setup the Mongodb and the git clone.

However I am receiving error messages about the logging.

I put ' ' around the Logging.info and logging.Critical to get it to run, but then the error "local variable 'logger' referenced before assignment occurs.

Please let me know if you are aware of any fix.

[-John]

Errors

Feature: Status documents with limited data points

For people with limited space on their Mongo DB, it would help to not keep all data points in their status documents. Only some limited data points like coordinates, altitude, speed, etc can be saved. This can be kept as a switchable feature.

MongoDB errors not caught

Errors from MongoDB are not being caught. They are neither making the script fail nor showing up in the logs.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.