connectedplacescatapult / sharingcitiesdashboard Goto Github PK
View Code? Open in Web Editor NEWPersonalisable dashboard for Sharing Cities IoT data
License: Apache License 2.0
Personalisable dashboard for Sharing Cities IoT data
License: Apache License 2.0
Once the request is verified launch a new thread to run the analytics request, the new thread should have access to task_id, user_id and api address to which it needs to send the data. The thread should also have access to database to save the computed data
So that it fits into the setup @arya-hemanshu is creating.
Ensure that tests are in place to ensure that we are getting the data we expect from each of the incoming APIs.
allow users to pass date range and train the model only using the data specified in that range
Based on existing test code and new API client infrastructure.
Based on user input update the null or empty cells
Example: http://datos.santander.es/api/rest/datasets/sensores_smart_parking.json
This API returns data whenever you call it. However, the timestamp gets updated only when the status is changed. How do we determine how often to ping it?
With the passed col name and table names, retrieve the data from database and convert them into dataframe so that it can be passed for type checking
Users may want to plot temperature vs bikes available at a particular site. For this to happen, time has to be normalised between the two. Approaches to this have been discussed and investigated (e.g. forward filling and back filling).
Recommendation for first implementation: only allow for the combination of two fields from two different data sources - no more.
create a tool to migrate mysql to postgresql database
Pipeline to build automated tests on branch creation and merging branch into master - Backend
For the UI and the analytics engine.
Pipeline to build automated tests on branch creation and merging branch into master - Frontend
Build Docker image
Upload it to docker hub
Create a wercker file
Provide an overview of the project.
Validate the data type of retrieved data
Based on existing test code and new API client infrastructure.
Build deployment pipeline for production environment
i) Message about the status of the request
ii) ID of the request
iii) ID of the requestor
iv) Timestamp of the response
Based on existing test code and new API client infrastructure.
Enabling continuous integration and testing.
ignore_object_tags only considers tags in first level of json heirarchy
Documentation has incorrect date range in the request section, update the documentation to reflect the current method
Pipeline to build automated tests on branch creation and merging branch into master - Analytics
We want the users / cities to have their own API credentials that are used to access the data from each API. We're currently using our development credentials and shouldn't expose these. Ideally we need to:
Create readme for analytics folder
Depending on the selected operation, defaults should be supplied:
E.g. Default
missingValues
= forward fill
.
Referring to below:
{
"columnsX": {
"table1": ["col1", "col2", "col3"],
"table2": ["col4", "col5", "col6"]
},
"columnY": "ycola",
"tableY": "table3",
"operation": "classification",
"requestor_id": 456,
"timeseries": "True",
"missingValues": "mean",
"dataRange": "12-03-2018 00:00:00.12-03-2018 00:00:00"
}
Just testing this out
For every request first check if there is already an existing model, if there is
return the prediction with request id
if no, check if data is under default limit
if yes, create model and send the predictions, if no start processing and send the request id
TFL's TIMS feed will be a valuable addition.
http://content.tfl.gov.uk/tims-feed-technical-specification.pdf
Give user an option in the api to send values on which data can be filtered, e.g if user just want to predict NO2, then user should be able to send these in the post request
It would be good if the users can view data with different temporal frequency in a common temporal extent. This would require resampling of one versus the other
eg. variable x temporal frequency is in minutes
variable y temporal frequency is in days
Feature scale the data so that it can be passed to ML libraries for training purposes
In date range check if users have passed the right values, if from date is smaller than to date then reject the request
To be used by the front end globally and including:
Setup Repo with disabling users to directly merge into master or without code review
Testing different machine learning libraries for identify which one is best suited for the project
The application / DB will grow through time. Need to decide the following:
if a col or row in a dataframe contains NaN or is empty, take mean or median
Set up tests for basic functionality and health of the API client.
Create an api endpoint that accepts:
i) Column name for dependent variables
ii) Table names for dependent variables
iii) Column name for Independent Variable
iv) Table name for Independent Variable
v) Operation to be performed on dataset
vi) Requestor Id
vii) A flag to indicate whether the data is timeseries data or not.
viii) If null flag to indicate what to do with the null rows
ix) Range of data
i) Error message defining what is actually missing and what caused the error while accepting the request.
ii) Request id (need to give more thought on it, is it even required)
i) Prediction array
ii) Timestamp for the predictions
iii) Request id
iv) Timestamp of the response
There;s no guarantee that the incoming data are errorless. Need to find a way to filter out erroneous (but numerical) records
Include a summary of the data that are included in the app.
For completed analytics runs and when new data is gathered though the API client. Rely on push notifications (EventSource or similar).
Look at all the code created to gather data from various APIs, find a common ground and create an infrastructure to add them to, and any new ones.
Should explain the structure of the code, how to extend it (add new APIs) and how to run tests where appropriate.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.