api-server's People
api-server's Issues
api-server crashed
sensor value subscription endpoint
In order to finish waziup integration, we would need to subscribe to Orion.
We need to let IVIS instance be able to subscribe to Orion
We can implement the same endpoint as Orion api/v1/orion/subscription/....
SensingDevice, QuantityKinds and Units
Changing the definition of a Measurement slightly to match Fiesta IoT ontology.
Before:
Measurement: {
id : ID of the measurement
name : name of the measurement
last_value : Timetime of the measurement
timestamp : time of the last measurement
dimension : physical dimension of the measurement
unit : unit of the measurement
sensor_kind :kind of sensor providing the measurement
}
After:
Measurement: {
id : ID of the measurement
name : name of the measurement
last_value : Timetime of the measurement
timestamp : time of the last measurement
sensing_device : sensing platform used for the measurement, from https://github.com/Waziup/waziup-js/blob/master/src/model/SensingDevices.js
quantity_kind : quantity measured, from https://github.com/Waziup/waziup-js/blob/master/src/model/QuantityKinds.js
unit : unit of the measurement, from https://github.com/Waziup/waziup-js/blob/master/src/model/Units.js
}
Quotas
Currently, a user can push as much data as he wants. How to limit that?
subscription endpoint
Currently, a user can only retrieve sensor data through GET (pooling).
Let's create a new endpoint to subscribe to sensor data.
lastN is not correct with Cloud timestamp
lastN returns the last N sensors, ordered by device timestamp:
$ curl -X GET "http://api.waziup.io/api/v1/sensors/LUMS_NODE_1/measurements/SM2/values?lastN=100" -H "accept: application/json"
[
{
"timestamp": "2016-06-08T18:23:57.873Z",
"value": 92,
"date_received": "2019-02-14T06:36:26.000Z"
},
{
"timestamp": "2016-06-08T18:23:57.873Z",
"value": 91,
"date_received": "2019-02-14T06:47:24.000Z"
},
{
"timestamp": "2016-06-08T18:23:57.873Z",
"value": 56,
"date_received": "2019-02-14T07:17:45.000Z"
}
]
However, if the device doesn't send this time (or a wrong one), this wouldn't work.
Migration Swagger 2.0 to OpenAPI 3.0
Environment variable for admin password
add delay_inactive to sensor definition
Currently there is no way to know if a sensor is inactive.
The idea is to add a field delay_inactive
to evaluate if the sensor is inactive or not.
Should we put it at sensor level or measurement level?
redirect api.waziup.io to api.waziup.io/docs
Permissions endpoint
Currently, the api-server check for permissions when the user performs a request on any endpoint.
For instance, permissions are checked when the user performs a DELETE /sensors/XXX
.
However, sometimes it is necessary to know the permission in advance, before performing any request.
For example, the UI could check that the user has permission to delete a resource, before displaying a "delete" icon. The current behavior is that the "delete" icon is displayed by default. If the user clicks on it, a message "unauthorized" might appear.
sensor measurements: last value
Currently, the sensor API returns an array of historical values. They can be filtered using #28 .
It would be nice to have also the last datapoint (as stored by Orion) in a separate field.
Error management
Currently, the api-server returns the errors from the underlying component (keycloak, orion, mongo, socials).
So the error format is not standardized, that makes it difficult to analyse.
Orion format:
{
"error": "NotFound",
"description": "The requested entity has not been found. Check type and id"
}
Mongo format:
Proxy error: backend service unavailable
Keycloak format:
{
"error": "invalid_resource",
"error_description": "Resource with name [Sensor2-ea0541de1ab7132a1d45b85f9b2139f5] does not exist."
}
nan value in sensor measurement
It seems that pushing "nan" as a value for a measurement crashes the server.
However, at the moment I cannot reproduce the error. With API call was used?
wrong date in datapoint CSV
Date is returned as "Thu Sep 20 2018 08:17:34 GMT+0000 (Coordinated Universal Time)" instead of UTC.
Secure social networks
Currently any user can send messages on the socials (including SMS).
Social channels should be protected with per-user and per-channel authorization.
Get sensors: number of sensors returned is not correct
When retrieving a number of sensors, less might be returned. For example, retrieving 20 sensors might yield 16. It's because we retrieve 20 sensors from Orion database, and then remove the one you don't have access to. So the remainder is 16.
Retrieving a non existing measurement
Retrieving a non existing measurement gives a positive result:
$ curl -X GET "http://localhost:80/api/v1/domains/waziup/sensors/MySensor/measurements/ttty/values?lastN=20&limit=50&offset=0&dateFrom=2016-01-01T00%3A00%3A00.000Z&dateTo=2019-01-31T23%3A59%3A59.999Z" -H "accept: application/json"
200
[]
It should return a 404.
/users/search endpoint not working
Users endpoint: missing POST
Currently on the users endpoint, there is no way to create a new user.
Clients token
Currently, only normal users can retrieve a token in order to make requests on the API.
However, for some applications it's not practical to declare all users on Keycloak/Waziup.
We could use a "client token" instead.
Specific access policies can be created for Clients in Keycloak.
Query strings
Insert query strings in GET methods to narrow the results:
GET /domains/{domain}/sensors
- filter by max numbers
- filter by attribute values
- sort?
- maximum number of values
- date of values
"The entity does not have such an attribute"
When trying to add a name to a sensor without a name:
$ curl -X PUT "http://localhost:80/api/v1/domains/waziup/sensors/MySensor/name" -H "accept: application/json" -H "Authorization: Bearer $TOKEN" -H "Content-Type: text/plain" -d "Sensor1"
{
"error": "NotFound",
"description": "The entity does not have such an attribute"
}
Include entities endpoint
Filter notifications
Currently the api-server returns all subscriptions from Orion when querying notifications:
$ curl -X GET "http://localhost:80/api/v1/domains/waziup/notifications" -H "accept: application/json"
[
{
"id": "5a71f14491ebc9a029760bc2",
"description": "Orion-Elasticsearch Feeder instance waziup tb2",
"subject": {
"condition": {
"attrs": []
},
"entityNames": [
null
]
},
"status": "active"
}
]
We should filter out subscriptions that are not related to Waziup notifications.
Additional types support for sensor values
Currently, only numerical values included in strings are supporter, i.e. "25.5".
Idea is to support all JSON types:
- string.
- Numeric types.
- JSON object.
- array.
- boolean.
simplify domains
As part of the documentation, the "domain" part of the URL embodies several concepts.
It's structure is <service>-<subservice>
, such as "waziup-farm1".
- The service is the FIWARE-Service in Orion
- The subservice is the FIWARE-ServicePath in Orion
- The service is also the realm in Keycloak
- The full domain is used as index name in Keycloak (for data analytics)
- The full domain is used as collection name in Mongo (for short historic)
This has several problems:
- It's not possible to move easily an entity from one ServicePath to another in Orion
- The api-server component is connected to one client realm in Keycloak, it's difficult to change realm based on the URL
- It's difficult for the user to understand the concept of domain, as it combines many things. For example a user can create a domain "waziup-farm1" but not "farm1" as this implicates the creation of another realm in KC.
Implement delete User
notif: internal server error
Posting a wrong/empty notification results in 500 error:
curl -X POST "http://localhost:80/api/v1/domains/waziup/notifications" -H "accept: application/json" -H "Content-Type: application/json" -d "{}"
It should be 422 unprocessable.
domains management
Currently, you can use domains freely in the API:
GET /domains/waziup-farm1/sensors
[
{
"id": "Sensor2-ea0541de1ab7132a1d45b85f9b2139f5",
"gateway_id": "ea0541de1ab7132a1d45b85f9b2139f5",
"name": "My weather station",
"owner": "cdupont",
"subservice": "farm1"
(...)
}
]
But those domains are not represented as separate entities. You cannot create and list domains for instance.
The idea is to create more endpoints to manage domains:
GET /domains
POST /domains
Include guest account
Guest account should be included in the waziup-realm.json
User endpoint data format
Currently the users endpoint data format is:
{ "id": "eb78eed6-d01a-4f69-b562-1f30002c08d9",
"createdTimestamp": 1497370576385,
"username": "waziup",
"enabled": true,
"totp": false,
"emailVerified": true,
"firstName": "test",
"lastName": "test",
"email": "[email protected]",
"attributes": { "permissions": [ "admin;advisor:/Calci;farmer:/Calci;" ] }, "disableableCredentialTypes": [ "password" ], "requiredActions": []
}
But it should be:
{
“id”: “eb78eed6”,
“createdTime": "2017-09-01T20:00:00.00Z",
“username": "test_user",
“firstName": "test",
“lastName": “test”,
“subservice”: “farm1”,
“email": “[email protected]”,
“cell": “+390000000”,
“address": “25 test street”,
“facebook”: “test1”,
“twitter”: “@Test1”,
“permissions”: “admin”
}
Include the data type in a measurement value
Currently all measurement are numbers as Strings (such as "25.6").
"TC1": {
"type": "Measurement",
"value": "25.6",
"metadata": {
"dimension": {
"type": "String",
"value": "temperature"
},
"name": {
"type": "String",
"value": "My garden temperature"
},
"sensor_kind": {
"type": "String",
"value": "Air temperature sensor"
},
"timestamp": {
"type": "DateTime",
"value": "2016-06-08T18:20:27.00Z"
},
"unit": {
"type": "String",
"value": "Degree C"
}
}
}
But it could also be a date or a location.
Suggestion is to embed also the type of the value:
"TC1": {
"type": "Measurement",
"value": {
"type": "Number",
"value": "25.6",
"metadata": {
"dimension": {
"type": "String",
"value": "temperature"
...
}
}
value types are the Orion basic types: Number, Date, Location...
Support measurement values as numbers
Trying to input a value as number yields:
$ curl -s -X POST http://dev.waziup.io/api/v1/domains/waziup-UPPA-TESTS/sensors/UPPA_Sensor6_4b13a223f24d3dba5403c2727fa92e62/measurements/TC/values -H 'accept:application/json' -H 'Content-Type:application/json' -d '{"value":22.5,"timestamp":"2017-12-10T11:39:48+01:00"}' -vvv
* Trying 18.195.197.182...
* Connected to dev.waziup.io (18.195.197.182) port 80 (#0)
> POST /api/v1/domains/waziup-UPPA-TESTS/sensors/UPPA_Sensor6_4b13a223f24d3dba5403c2727fa92e62/measurements/TC/values HTTP/1.1
> Host: dev.waziup.io
> User-Agent: curl/7.47.0
> accept:application/json
> Content-Type:application/json
> Content-Length: 54
>
* upload completely sent off: 54 out of 54 bytes
< HTTP/1.1 500 Internal Server Error
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< Content-Type: application/json; charset=utf-8
< Content-Length: 432
< ETag: W/"1b0-7XmdV2RhjV7IUehKM638MPzlfbg"
< Date: Sun, 10 Dec 2017 11:02:25 GMT
< Connection: keep-alive
<
* Connection #0 to host dev.waziup.io left intact
{"config":{"transformRequest":{},"transformResponse":{},"timeout":0,"xsrfCookieName":"XSRF-TOKEN","xsrfHeaderName":"X-XSRF-TOKEN","maxContentLength":-1,"headers":{"Accept":"application/json, text/plain, */*","Content-Type":"text/plain","Fiware-Service":"waziup","User-Agent":"axios/0.16.2"},"method":"put","url":"http://orion:1026/v2/entities/UPPA_Sensor6_4b13a223f24d3dba5403c2727fa92e62/attrs/TC/value","data":22.5,"params":null}}
Only number in a String (e.g. "25.6") are supported.
/auth/permissions returns 403 when no permissions
When querying permissions, and there is no permissions to return (user has access to nothing), the endpoint returns 403.
It might be better to return the empty list?
add times_sent and last_notification to notifications API.
Currently they are not available, but provides a valuable insight.
Retrieving users list
For a non-privileged user, retrieving the user list results in a "error: forbidden" message.
Should it return only one user (himself)?
History endpoint security
Currently there is some security implemented with Kibana-backend and api-server:
- Kibana-backend is a proxy, able to serve Kibana UI
- Kibana-backend re-route the user to keycloak for authentication
- Kibana is connected to the api-server to collect data, instead of elasticsearch
- api-server is implementing basic authorization.
Currently the authorization is: DELETE forbidden on all indexes.
However this is a bit restrictive: you cannot delete an index pattern for example.
Index patterns are stored on .kibana.
Proposed solution:
Implement authorization based on index names. e.g. .kibana index can be modified only by administrators.
Normal users can access the index that corresponds to a domain they own.
Socials service names
Currently, the socials endpoints look like:
POST /domains/{domain}/socials
{
"username": "cdupont",
"channel": "facebook",
"message": "Test message"
}
It would be easier to secure with explicit endpoint names for each service:
POST /domains/{domain}/socials/sms
{
"username": "cdupont",
"message": "Test message"
}
POST /domains/{domain}/socials/facebook
{
"username": "cdupont",
"message": "Test message"
}
POST /domains/{domain}/socials/twitter
{
"username": "cdupont",
"message": "Test message"
}
Domains
Specification for the domains usage and management.
Domains creation
Registered users can create domains:
POST /api/v1/domains {"id":"farm1"}
Domains deletion
Domains can be deleted by its owner or an admin.
DELETE /api/v1/domains/farm1
Deleting a domain will also delete all resources that belongs to that domain.
Resource creation
A resource is created under a domain. For example:
POST /api/v1/domains/farm1/sensors -d '{"id": "Sensor1"}'
If the domain doesn't exist, the creation is rejected with 404 "Domain doesn't exists".
If the domain exists, the resource is created. The domain is recorded as an attribute in the entity.
Resource view
You can view a resource using its domain.
GET /api/v1/domains/farm1/sensors
Will return only the sensors belonging to the domain "farm1".
GET /api/v1/domains/*/sensors
Will return all the sensors.
Resource update
You can update a specific resource using the domain:
PUT /api/v1/domains/farm1/sensors/Sensor1
If the domain doesn't exists, a 404 will be returned.
If the domain exists but the resource doesn't belong to it, an error code is returned.
For specific resources, a wildcard can be used for the domain:
PUT /api/v1/domains/*/sensors/Sensor1
Using the wildcard allows to ignore the restrictions on the domain and to update directly the resource.
Move from one domain to another
The user can move his resource form one domain to another:
PUT /api/v1/domains/farm1/sensors/Sensor1 "farm2"
An error is raised is the destination domain doesn't exists.
Gateways
Gateways can create a sensor in its own domain:
POST /api/v1/domains/farm1/sensors -d '{"id": "Sensor1"}'
However, the user should make sure that the domain exists beforehand.
That's why it might be better to create the sensor in a "standard" domain:
POST /api/v1/domains/waziup/sensors -d '{"id": "Sensor1"}'
The user will then be able to move the sensor to another domain.
To push the data regardless of domain, the gateway should use the wildcard:
POST /api/v1/domains/*/sensors/Sensor1/measurements/TC/values {"value": "25.6", "timestamp": "2016-06-08T18:20:27.873Z"}
Pagination of datapoints
To help retrieving big amounts of sensor datapoints, we need to use a pagination mechanism.
We can use those query strings:
- vLastN: Only the requested last datapoints will be returned. Datapoints are sorted with the most recent datapoint first. It is on by default, with a value of 20.
- vLimit: In case of pagination, the number of datapoints per page.
- vOffset: In case of pagination, the offset to apply to the requested search of datapoints.
- vDateFrom: The starting date and time from which the datapoints are desired. It is an optional parameter.
- vDateTo: The final date and time until which the datapoints are desired. It is an optional parameter.
Example of query:
curl -X GET "http://dev.waziup.io:80/api/v1/domains/waziup/sensors/mySensor/measurements/TC/values?vLimit=50&vOffset=0&vDateFrom=2016-01-01T00:00:00.000Z&vDateTo=2016-01-31T23:59:59.999Z"
This query will retrieve the first 50 datapoints from January 2016.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.