Giter Club home page Giter Club logo

munibot's Introduction

Munibot

Tests

Munibot is a friendly Twitter bot that posts aerial or satellite imagery of administrative regions (tipically municipalities).

munis_cat_scaled

It is written in a modular way so it's easy to customize to different data sources, via the implementation of profiles.

It currently powers the following Twitter accounts:

  • @munibot_es: All municipalities in Spain, shown in random order, with base aerial ortophotograhy from PNOA IGN.

  • @munibot_cat: All municipalities in Catalonia, shown in random order, with base aerial ortophotograhy from ICGC.

  • @communebot: All communes in France, shown in random order, with base aerial ortophotograhy from IGN.

Here's how a sample tweet looks like:

example_tweet

Table of Contents

Usage

Installation

Munibot is available on PyPI and can be installed with pip. It is strongly recommended to install it in a virtual environment:

python3 -m venv munibot
source munibot/bin/activate

pip install munibot

Or alternatively, using pipx:

pipx install munibot

Munibot uses Rasterio and Fiona, which require GDAL. The wheels installed by pip on Linux (and macOS, although I have not tested it) include binaries for GDAL that cover munibot's need so it doesn't need to be installed separately. On other operating systems you might need to install GDAL.

Configuration

Munibot uses an ini file for configuration. You can download the sample ini file included in this repository running:

curl https://raw.githubusercontent.com/amercader/munibot/main/munibot.sample.ini -o munibot.ini

or:

wget https://raw.githubusercontent.com/amercader/munibot/main/munibot.sample.ini -O munibot.ini

By itself munibot is not able to do much. You need to install an existing profile, or write your own.

To install a profile just install its Python package with pip:

pip install munibot-es

Running it

Once munibot is installed, you should be able to run

munibot --help

Munibot assumes that the configuration ini file is located in the same folder the command is run on (and named "munibot.ini"). If that's not the case, you can pass the location of the configuration file with the --config or -c arguments:

munibot -c /path/to/munibot.ini

If at least a profile is available and all the necessary authorization tokens are available in the ini file (see Twitter authorization) just run the following to tweet a new image:

munibot tweet <profile-name>

If you only want to create the image without tweeting it use the create command:

munibot create <profile-name>

Deploying it

You don't need much to run munibot, just a system capable of running Python >= 3.6. Once installed, you probably want to schedule the sending of tweets at regular intervals. An easy way available on Linux and macOS is cron. Here's an example configuration that you can adapt to your preferred interval and local paths (it assumes munibot was installed in a virtualenv in /home/user/munibot):

# Tweet an image every 8 hours (~3 times a day)
0 */8 * * * /home/user/munibot/bin/munibot --c /home/user/munibot/munibot.ini tweet cat >> /home/user/out/cat/munibot_cat.log 2>&1

You can adjust the log level in the munibot ini configuration file.

Writing your own profile

Munibot is designed to be easy to customize to different data sources in order to power different bot accounts. This is done via profile classes. Profiles implement a few mandatory and optional properties and methods that provide the different inputs necessary to generate the tweets. Munibot takes care of the common functionality like generating the final image and sending the tweet.

To see the actual methods that your profile should implement check the BaseProfile class in munibot/profiles/base.py. Here's a quick overview of what you should provide:

  • The geometry of the boundary of a particular administrative unit (given an id). This can come from any place that can end up providing a GeoJSON-like Python dict: an actual GeoJSON file, PostGIS database or a WFS service.
  • The base image (aerial photography or satellite imagery) covering the extent of the administrative unit (given the extent). WMS services work really well for this as they allow to retrieve images of arbitrary extent and size.
  • The text that should go along with the image in the tweet. Generally the name of the unit, plus some higher level unit for reference.
  • A method that defines the id of the next unit that should be tweeted.
  • Optionally, the latitude and longitude that should be added to the tweet.

Once you've implemented your profile class you can register using the munibot_profiles entry point in your package setup.py file:

"munibot_profiles": [
	"<profile_id>=<your_package>.profiles.:<YourProfileClass>",
]

You can check the following examples:

Twitter Authorization

Authentication when using the Twitter API can be confusing at first, but it should be hopefully clear after following this guide.

We need to use what Twitter calls OAuth 1.0a, more specifically PIN-Based OAuth.

Quick summary:

  • You will register for a Twitter developer account
  • With this account you will create an app, which will be used to interact with the Twitter API.
  • The actual bot accounts will authorize this application with write permissions.
  • The munibot app will tweet on behalf of the bot accounts.

Step-by-step setup (you will need munibot up and running so if you haven't yet installed do so first):

  1. Register for a developer account on https://developer.twitter.com (you can use your actual Twitter account or create a separate one)

  2. Create a new Project, and a new Application within it:

    • Select Read and Write permissions
    • Turn on the Enable 3-legged OAuth authentication setting
    • Enter a callback URL (anything will do, we won't use it)
  3. Generate an Access token and secret, and enter them under the [twitter] section in the munibot.ini file:

    [twitter]
    api_key=CHANGE_ME
    api_key_secret=CHANGE_ME
    
  4. Create a twitter account for your bot (eg munibot_xyz)

  5. Run the following command:

    munibot tokens <profile_name>
    

    You should see a message like:

     Please visit the following URL logged in as the Twitter bot account for this profile, authorize the application and input the verification code shown.
    
     https://api.twitter.com/oauth/authorize?oauth_token=XXX
    
     Verification code:
    
  6. Do as suggested, open the link logged in as the bot account (not your own). You should see a page asking you to authorize the application that you created on step 2. Once authorized you should see a big verification code. Enter it in the munibot command prompt.

    Verification code

  7. The command should output the following:

     Done, access tokens for profile <profile_name>:
    
     twitter_access_token=xxx
     twitter_access_token_secret=yyy
    
  8. Enter the tokens above in the relevant profile section in the munibot.ini file:

     [profile:<profile_name>]
     twitter_access_token=xxx
     twitter_access_token_secret=yyy
    

Done! From this moment on munibot should be able to tweet on behalf of the bot account. You can try it running munibot tweet <profile_name>

Development installation

Clone the repo and install the requirements:

git clone https://github.com/amercader/munibot.git
pip install -r requirements.txt
pip install -r dev-requirements.txt

To run the tests:

pytest

With coverage:

pytest -v --cov=munibot --cov-report term-missing

License

MIT

munibot's People

Contributors

amercader avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar

munibot's Issues

Munibot_es failing since 26th March

GetMap requests fail with:

2021-04-02 23:22:17,447 INFO  [munibot.munibot] Start: create image for feature 34070505107 on profile es
/home/adria/dev/pyenvs/munis/src/munibot_es/munibot_es/profiles/es.py:53: RuntimeWarning: Sequential read of iterator was interrupted. Resetting iterator. This can negatively impact the performance.
  geometry = [f["geometry"] for f in src][0]
2021-04-02 23:22:17,861 DEBUG [munibot.image] Received boundaries from profile ((40.654864, -5.137099, 40.704987, -5.070794))
Traceback (most recent call last):
  File "/home/adria/dev/pyenvs/munis/bin/munibot", line 11, in <module>
    load_entry_point('munibot', 'console_scripts', 'munibot')()
  File "/home/adria/dev/pyenvs/munis/src/munibot/munibot/munibot.py", line 97, in main
    create_image(profile, id_, output)
  File "/home/adria/dev/pyenvs/munis/src/munibot/munibot/image.py", line 109, in create_image
    base_image = profile.get_base_image(extent)
  File "/home/adria/dev/pyenvs/munis/src/munibot_es/munibot_es/profiles/es.py", line 70, in get_base_image
    return self.get_wms_image(**wms_options)
  File "/home/adria/dev/pyenvs/munis/src/munibot/munibot/profiles/base.py", line 168, in get_wms_image
    wms = WebMapService(url, version=version, headers=headers)
  File "/home/adria/dev/pyenvs/munis/lib/python3.8/site-packages/OWSLib-0.20.0-py3.8.egg/owslib/wms.py", line 54, in WebMapService
    return wms130.WebMapService_1_3_0(
  File "/home/adria/dev/pyenvs/munis/lib/python3.8/site-packages/OWSLib-0.20.0-py3.8.egg/owslib/map/wms130.py", line 87, in __init__
    self._buildMetadata(parse_remote_metadata)
  File "/home/adria/dev/pyenvs/munis/lib/python3.8/site-packages/OWSLib-0.20.0-py3.8.egg/owslib/map/wms130.py", line 106, in _buildMetadata
    self.operations.append(OperationMetadata(elem))
  File "/home/adria/dev/pyenvs/munis/lib/python3.8/site-packages/OWSLib-0.20.0-py3.8.egg/owslib/map/wms130.py", line 728, in __init__
    url = verb.find(nspath('OnlineResource', WMS_NAMESPACE)).attrib['{http://www.w3.org/1999/xlink}href']
KeyError: '{http://www.w3.org/1999/xlink}href'

Cat profile works fine

France communes

New bot for France's ~34k communes! 🇫🇷

  • Database
    • List: 2021 dump from here, includes id (INSEE code), name and link to Wikipedia.
    • Higher administrative unit (Départements)
    • Lon / Lat
  • Geometries: Nice API here using the INSEE code, so no need to use the WFS.
  • Imagery:
    • Got a key from IGN for the orthos WMS, but need to authenticate with a User-agent header and OWSlib doesn't send it with the getMap requests 🤷‍♂️ Sent a PR but we can patch our own for now
    • Which CRS to use for each region?
  • Code: separate Python package to showcase how to write external profiles. Done: https://github.com/amercader/communebot_fr
  • Deploy 🤖

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.