Giter Club home page Giter Club logo

iitc-scan-scheduler's Introduction

24/7 Intel event scanner and aggregation server application

IMPORTANT: In order to use this application you need a working binary of phantomJS version 1.9.1. You can find prebuilt binaries and sources at: http://phantomjs.org/download.html

CAUTION: By using IITC to access the Ingress API you violate Niantic's ToS. This might lead to a permanent account suspension. Use it at your own risk.

For a quick overview about this application you can find a screencast video I recorded a while back (german only) at YouTube.

Configuration file

IMPORTANT: Rename file config.example.json to config.json and fill in your Ingress credentials. Otherwise the application won't work.

  • All time values are SECONDS. No milliseconds. SECONDS.
  • All memory related values are BYTES.

Services

  • Screenshot capture service
  • Scan schedule editor
  • Event data streaming service

Screenshot capture service

There is a HTTP endpoint exposed at http://localhost:9001/ that serves fullsize screenshots of the involved Intel webpage on demand. The screenshot size depends on what viewport size has been configured. By default the served website that contains the screenshot image (embedded; base64 encoded) will refresh itself every 10s. There is a checkbox for controlling this behaviour.

Served screenshots have all additional UI dialogs/panels (sidebar, chat, additional map/tile overlays) hidden โ€“ only the map, portals, links and fields are visible.

Scan scheduler with editor

There is a HTTP endpoint exposed at http://localhost:9002/scheduler that serves a simple web application for managing what geographic areas will be monitored for events. Although the scanner can be started/stopped via a simple HTTP API, the editor doesn't support this yet (trivial to add).

LeafletJS is displaying OSM/GMaps map tiles to user. Leaflet.draw plugin (with restriction to rectangular elements) is enabled and allows the user to draw rectangles (called "scanner fields") over the geographic area that will be monitored for new events on periodical basis.

The drawn rectangles are persisted in a flat file as GeoJSON data which is used by both, the scan scheduler and the phantomJS iitc-server. (locking!) For now, only one person may use the schedule editor at a time. When the schedule editor is started, it reads the data file and plots its content on the map. When the editor is instructed to persist its state, LeafletJS is used to generate the according GeoJSON data and the file's content is replaced with it.

When the phantomJS server starts monitoring, it parses the aforementioned GeoJSON file and iterates over all contained GeoJSON Feature objects and processes each of them. Therefore it takes its geometric boundaries and calculates how many scanner viewports it takes to cover the according map area, based on the used screen size and the zoom level needed to catch Level 1 portals. Optionally debug tile data can be returned that can be plotted on the map for debugging purposes.

Execution is paused (configurable) after each field/sector scan and before the next re-scan.

Debug mode

If you set the config setting debug to true, the scanner will generate two types of debug data and writes them to a file:

Field sectors

The file data/debug/sectors.json will contain all generated tiles in GeoJSON format. In order to visualize them, just open the schedule editor page, fire up the webdeveloper dialog and enter the following code:

L.geoJson(jsonDataFromFile).addTo(map); whereas jsonDataFromFile is the copy&pasted content of the aforementioned JSON file.

Portal entities

IMPLEMENTATION INCOMPLETE

Right now only the raw and unprocessed event data (received by a listener to "scanner:scanFinished") is written to timestamp-named JSON file that is placed inside the data/scans folder. Note that this data still contains duplicates and isn't reasonably normalized.

Event data streaming service

IMPLEMENTATION PENDING

The data streaming service does no data evaluation or aggregation. It only collects the according data, performs simple de-duplication, filters out personal event data, persists it and exposes an API that can be used by clients to query the data. Data can also be automatically pushed to a pre-configured endpoint.

Pending tasks

  • 2.3.2 Save data for: public+faction chat and fetch data of all portals in range
  • 2.4 Merge/de-dup data from all Ingress tiles
  • 2.5 Filter out personal events ("Your portal is under attack" asf.)
    1. Push newly collected data to ???

TODO

  • [editor] Restrict access to editor by asking for a username/password pair that will be matched against a separate entry in the config file (users:{user1:pass1,...})
  • [client] Make sure that all new events since last scan are gathered (finish when already known event GUID is reached)
  • User may choose zoom level individually for every scanner field
  • Add random jitter factor to pauses between scan operations
  • Store every external script file (iitc, plugins) in private GitHub Gist and introduce ghetto update checker that reloads default page when new versions are available
  • When in debug mode, create GeoJSON file with sector outline and all contained portals so that I can test the scanner behaviour
  • Figure out how to effectively persist data and how to expose it to consumer clients
  • Refactor code and inject context object to prevent nested closures
  • Collect chat data of Resistance faction; create second account
  • Improve algorithm for calculating scanner sectors (reduce overlapping areas and intelligently adjust file size to prevent scanning of outside area)
  • Implement simple locking mechanism for data files
  • Add pid file to prevent multiple iitc-server instances in the same folder
  • General code review

iitc-scan-scheduler's People

Contributors

peta avatar

Stargazers

 avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.