Giter Club home page Giter Club logo

egov-analytics's Introduction

README

This project analyzes of municipal data for real time alerts and forecasts by analysing the data from the eGov SmartCity Public Grievance Redressal (PGR) system.

This is an ongoing collaboration between eGovernments Foundation and DataKind Bangalore, as part of DataKind's DataCorps.

About eGovernments Foundation and DataKind

eGovernments Foundation

eGovernments Foundation transforms urban governance with the use of scalable and replicable technology solutions that enable efficient and effective municipal operations, better decision making, and contact-less urban service delivery.

DataKind

DataKind is a nonprofit organization that brings leading data scientists together with high impact social organizations through a comprehensive, collaborative approach that leads to shared insights, greater understanding and positive action through data in the service of humanity.

About the project

We're using data generated by the Corporation of Chennai (CoC). The city generates these complaints via dedicated phone lines and a portal where citizens can raise a complaint. The data we are using contains complaints from 200 wards across Chennai and 93 types of complaints.

Although the solution is built for the CoC, it is designed to be general enough to work with other municipalities.

There are two goals in mind for this project:

  • Forecasts for overall top complaint types across the city
  • An alerts and notifications system which raises an alert if an anomaly in the inflow of complaints is detected.

Sub-modules

  • EDA - Dashboard: For exploring & plotting data, and preliminary analysis for Time series modeling
  • Alerts Dashboard: Dashboard for viewing detected anomalies at a specific date
  • (WIP) Time Series Dashboard: End to end time series modeling. Forked to a separate repo
  • Gravity: Front-end for the project, where results are tabulated and visualized.
  • Notebooks : Contains several notebooks in which a few complaint types' data are analyzed. Useful for understanding how time series modeling and anomaly detection works
  • eGovs.lib: The R library which contains high level APIs for time series modeling and anomaly detection.

How to run

Installing the R library

  • Install devtools using install.packages("devtools")
  • Execute devtools::install_github("egovernments/analytics",subdir = "eGovs.lib")

Additional instructions are available here

Building models and getting output

  • Create a config file which contains model specifications. An example can be found here in eGovs.lib/R/example_config.json
  • Execute eGovs.lib::execute.all(<path.to.config>, <path.to.output>)

Additional instructions are available here

Running the front-end

Instructions are available here

Generating sample data /*TODO*/

FAQ /*TODO*/:

  • How do I contribute?
  • Something's not working, what do I do?

egov-analytics's People

Contributors

ashwathgovind avatar egov-systems avatar mansiarora1009 avatar mramakrishnan-chwy avatar rithwik avatar sahil-m avatar samarthbhargav avatar srivastava-manu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

egov-analytics's Issues

Getting more done in GitHub with ZenHub

Hola! @mansiarora1009 has created a ZenHub account for the egovernments organization. ZenHub is the only project management tool integrated natively in GitHub – created specifically for fast-moving, software-driven teams.


How do I use ZenHub?

To get set up with ZenHub, all you have to do is download the browser extension and log in with your GitHub account. Once you do, you’ll get access to ZenHub’s complete feature-set immediately.

What can ZenHub do?

ZenHub adds a series of enhancements directly inside the GitHub UI:

  • Real-time, customizable task boards for GitHub issues;
  • Multi-Repository burndown charts, estimates, and velocity tracking based on GitHub Milestones;
  • Personal to-do lists and task prioritization;
  • Time-saving shortcuts – like a quick repo switcher, a “Move issue” button, and much more.

Add ZenHub to GitHub

Still curious? See more ZenHub features or read user reviews. This issue was written by your friendly ZenHub bot, posted by request from @mansiarora1009.

ZenHub Board

EDA for Alerts

Need to come up with a defn for an anomaly in the context of complaints.
Is it a point anomaly, or is it a 'trend' that needs to be detected

Consolidate 2015 data

Need to have data for the entire year. Consolidate the data Ankit fetched, and the one given to us in the sprint into a single file for the analysis.

This will be used in the validation phase to see how well our forecasts work

Add a grid search strategy for ARIMA modeling

Right now, the ARIMA notebook requires manual selection of the (p, d, q) parameters. It should be automated, in the following way:

  • Build a grid of possible (p, d, q) params
  • Split the data into a train, test, and validation set
  • Select [1] one param set, fit a model, compute accuracy[2].
  • Repeat until desired.
  • Pick the param set and model which scores highest on the validation set
  • Compute final accuracy on the test set

[1]:
The sampling can be done in two ways, and both needs to be explored. This entirely depends on the size of the sample space.

  1. Iterate through the entire space: Possible only if it is small, and if modeling is fast
  2. Perform Random Sampling of the space: Might not give us the best model

[2]:
As of today, the accuracy metric is still not finalized. Refer #39

Fix issues in Rollup script

Right now the script takes up an enormous amount of memory and time.
The script shouldn't generate data for each complaint type x date x ward, which is what it is doing.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.