waziup / api-server Goto Github PK

The api-server seem to be crashed. It outputs single characters on the log.

In order to finish waziup integration, we would need to subscribe to Orion.
We need to let IVIS instance be able to subscribe to Orion
We can implement the same endpoint as Orion api/v1/orion/subscription/....

Changing the definition of a Measurement slightly to match Fiesta IoT ontology.
Before:

Measurement: {
  id : ID of the measurement
  name : name of the measurement
  last_value : Timetime of the measurement
  timestamp : time of the last measurement
  dimension : physical dimension of the measurement
  unit : unit of the measurement
  sensor_kind :kind of sensor providing the measurement
}

After:

Measurement: {
  id : ID of the measurement
  name : name of the measurement
  last_value : Timetime of the measurement
  timestamp : time of the last measurement
  sensing_device : sensing platform used for the measurement, from https://github.com/Waziup/waziup-js/blob/master/src/model/SensingDevices.js
  quantity_kind : quantity measured, from https://github.com/Waziup/waziup-js/blob/master/src/model/QuantityKinds.js
  unit : unit of the measurement, from https://github.com/Waziup/waziup-js/blob/master/src/model/Units.js
}

Currently, a user can push as much data as he wants. How to limit that?

Currently, a user can only retrieve sensor data through GET (pooling).
Let's create a new endpoint to subscribe to sensor data.

lastN returns the last N sensors, ordered by device timestamp:

$ curl -X GET "http://api.waziup.io/api/v1/sensors/LUMS_NODE_1/measurements/SM2/values?lastN=100" -H  "accept: application/json"
[
  {
    "timestamp": "2016-06-08T18:23:57.873Z",
    "value": 92,
    "date_received": "2019-02-14T06:36:26.000Z"
  },
  {
    "timestamp": "2016-06-08T18:23:57.873Z",
    "value": 91,
    "date_received": "2019-02-14T06:47:24.000Z"
  },
  {
    "timestamp": "2016-06-08T18:23:57.873Z",
    "value": 56,
    "date_received": "2019-02-14T07:17:45.000Z"
  }
]

However, if the device doesn't send this time (or a wrong one), this wouldn't work.

Currently there is no way to know if a sensor is inactive.
The idea is to add a field delay_inactive to evaluate if the sensor is inactive or not.
Should we put it at sensor level or measurement level?

Currently, the api-server check for permissions when the user performs a request on any endpoint.
For instance, permissions are checked when the user performs a DELETE /sensors/XXX.
However, sometimes it is necessary to know the permission in advance, before performing any request.

For example, the UI could check that the user has permission to delete a resource, before displaying a "delete" icon. The current behavior is that the "delete" icon is displayed by default. If the user clicks on it, a message "unauthorized" might appear.

Currently, the sensor API returns an array of historical values. They can be filtered using #28 .
It would be nice to have also the last datapoint (as stored by Orion) in a separate field.

Currently, the api-server returns the errors from the underlying component (keycloak, orion, mongo, socials).
So the error format is not standardized, that makes it difficult to analyse.
Orion format:

{
  "error": "NotFound",
  "description": "The requested entity has not been found. Check type and id"
}

Mongo format:

Proxy error: backend service unavailable

Keycloak format:

{
  "error": "invalid_resource",
  "error_description": "Resource with name [Sensor2-ea0541de1ab7132a1d45b85f9b2139f5] does not exist."
}

It seems that pushing "nan" as a value for a measurement crashes the server.
However, at the moment I cannot reproduce the error. With API call was used?

Date is returned as "Thu Sep 20 2018 08:17:34 GMT+0000 (Coordinated Universal Time)" instead of UTC.

Currently any user can send messages on the socials (including SMS).
Social channels should be protected with per-user and per-channel authorization.

When retrieving a number of sensors, less might be returned. For example, retrieving 20 sensors might yield 16. It's because we retrieve 20 sensors from Orion database, and then remove the one you don't have access to. So the remainder is 16.

Retrieving a non existing measurement gives a positive result:

$ curl -X GET "http://localhost:80/api/v1/domains/waziup/sensors/MySensor/measurements/ttty/values?lastN=20&limit=50&offset=0&dateFrom=2016-01-01T00%3A00%3A00.000Z&dateTo=2019-01-31T23%3A59%3A59.999Z" -H  "accept: application/json"
200
[]

It should return a 404.

Currently on the users endpoint, there is no way to create a new user.

Currently, only normal users can retrieve a token in order to make requests on the API.
However, for some applications it's not practical to declare all users on Keycloak/Waziup.
We could use a "client token" instead.
Specific access policies can be created for Clients in Keycloak.

Insert query strings in GET methods to narrow the results:

GET /domains/{domain}/sensors

• filter by max numbers
• filter by attribute values
• sort?
• maximum number of values
• date of values

When trying to add a name to a sensor without a name:

$ curl -X PUT "http://localhost:80/api/v1/domains/waziup/sensors/MySensor/name" -H  "accept: application/json" -H  "Authorization: Bearer $TOKEN" -H  "Content-Type: text/plain" -d "Sensor1"
{
  "error": "NotFound",
  "description": "The entity does not have such an attribute"
}

Currently the api-server returns all subscriptions from Orion when querying notifications:

$ curl -X GET "http://localhost:80/api/v1/domains/waziup/notifications" -H  "accept: application/json"
[
  {
    "id": "5a71f14491ebc9a029760bc2",
    "description": "Orion-Elasticsearch Feeder instance waziup tb2",
    "subject": {
      "condition": {
        "attrs": []
      },
      "entityNames": [
        null
      ]
    },
    "status": "active"
  }
]

We should filter out subscriptions that are not related to Waziup notifications.

Currently, only numerical values included in strings are supporter, i.e. "25.5".
Idea is to support all JSON types:

• string.
• Numeric types.
• JSON object.
• array.
• boolean.

As part of the documentation, the "domain" part of the URL embodies several concepts.
It's structure is <service>-<subservice>, such as "waziup-farm1".

• The service is the FIWARE-Service in Orion
• The subservice is the FIWARE-ServicePath in Orion
• The service is also the realm in Keycloak
• The full domain is used as index name in Keycloak (for data analytics)
• The full domain is used as collection name in Mongo (for short historic)

This has several problems:

• It's not possible to move easily an entity from one ServicePath to another in Orion
• The api-server component is connected to one client realm in Keycloak, it's difficult to change realm based on the URL
• It's difficult for the user to understand the concept of domain, as it combines many things. For example a user can create a domain "waziup-farm1" but not "farm1" as this implicates the creation of another realm in KC.

Posting a wrong/empty notification results in 500 error:

curl -X POST "http://localhost:80/api/v1/domains/waziup/notifications" -H  "accept: application/json" -H  "Content-Type: application/json" -d "{}"

It should be 422 unprocessable.

Currently, you can use domains freely in the API:

GET /domains/waziup-farm1/sensors
[
  {
    "id": "Sensor2-ea0541de1ab7132a1d45b85f9b2139f5",
    "gateway_id": "ea0541de1ab7132a1d45b85f9b2139f5",
    "name": "My weather station",
    "owner": "cdupont",
    "subservice": "farm1"
   (...)
  }
]

But those domains are not represented as separate entities. You cannot create and list domains for instance.
The idea is to create more endpoints to manage domains:

GET /domains
POST /domains

Guest account should be included in the waziup-realm.json

Currently the users endpoint data format is:

{    "id": "eb78eed6-d01a-4f69-b562-1f30002c08d9",
    "createdTimestamp": 1497370576385,
    "username": "waziup",
    "enabled": true,
    "totp": false,
    "emailVerified": true,
    "firstName": "test",
    "lastName": "test",
    "email": "[email protected]",
    "attributes": {      "permissions": [        "admin;advisor:/Calci;farmer:/Calci;"      ]    },    "disableableCredentialTypes": [      "password"    ],    "requiredActions": []  
}

But it should be:

{
   “id”: “eb78eed6”,
   “createdTime": "2017-09-01T20:00:00.00Z",
   “username": "test_user",
   “firstName": "test",
   “lastName": “test”,
   “subservice”: “farm1”,
   “email": “[email protected]”,
   “cell": “+390000000”,
   “address": “25 test street”,
   “facebook”: “test1”,
   “twitter”: “@Test1”,
   “permissions”: “admin”
}

Currently all measurement are numbers as Strings (such as "25.6").

    "TC1": {
      "type": "Measurement",
      "value": "25.6",
      "metadata": {
        "dimension": {
          "type": "String",
          "value": "temperature"
        },
        "name": {
          "type": "String",
          "value": "My garden temperature"
        },
        "sensor_kind": {
          "type": "String",
          "value": "Air temperature sensor"
        },
        "timestamp": {
          "type": "DateTime",
          "value": "2016-06-08T18:20:27.00Z"
        },
        "unit": {
          "type": "String",
          "value": "Degree C"
        }
      }
    }

But it could also be a date or a location.
Suggestion is to embed also the type of the value:

    "TC1": {
      "type": "Measurement",
      "value": {
          "type": "Number",
          "value": "25.6",
      "metadata": {
        "dimension": {
          "type": "String",
          "value": "temperature"
        ...
      }
    }

value types are the Orion basic types: Number, Date, Location...

Trying to input a value as number yields:

$ curl -s -X POST http://dev.waziup.io/api/v1/domains/waziup-UPPA-TESTS/sensors/UPPA_Sensor6_4b13a223f24d3dba5403c2727fa92e62/measurements/TC/values -H 'accept:application/json' -H 'Content-Type:application/json' -d '{"value":22.5,"timestamp":"2017-12-10T11:39:48+01:00"}' -vvv
*   Trying 18.195.197.182...
* Connected to dev.waziup.io (18.195.197.182) port 80 (#0)
> POST /api/v1/domains/waziup-UPPA-TESTS/sensors/UPPA_Sensor6_4b13a223f24d3dba5403c2727fa92e62/measurements/TC/values HTTP/1.1
> Host: dev.waziup.io
> User-Agent: curl/7.47.0
> accept:application/json
> Content-Type:application/json
> Content-Length: 54
> 
* upload completely sent off: 54 out of 54 bytes
< HTTP/1.1 500 Internal Server Error
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< Content-Type: application/json; charset=utf-8
< Content-Length: 432
< ETag: W/"1b0-7XmdV2RhjV7IUehKM638MPzlfbg"
< Date: Sun, 10 Dec 2017 11:02:25 GMT
< Connection: keep-alive
< 
* Connection #0 to host dev.waziup.io left intact
{"config":{"transformRequest":{},"transformResponse":{},"timeout":0,"xsrfCookieName":"XSRF-TOKEN","xsrfHeaderName":"X-XSRF-TOKEN","maxContentLength":-1,"headers":{"Accept":"application/json, text/plain, */*","Content-Type":"text/plain","Fiware-Service":"waziup","User-Agent":"axios/0.16.2"},"method":"put","url":"http://orion:1026/v2/entities/UPPA_Sensor6_4b13a223f24d3dba5403c2727fa92e62/attrs/TC/value","data":22.5,"params":null}}

Only number in a String (e.g. "25.6") are supported.

When querying permissions, and there is no permissions to return (user has access to nothing), the endpoint returns 403.
It might be better to return the empty list?

Currently they are not available, but provides a valuable insight.

For a non-privileged user, retrieving the user list results in a "error: forbidden" message.
Should it return only one user (himself)?

Currently there is some security implemented with Kibana-backend and api-server:

• Kibana-backend is a proxy, able to serve Kibana UI
• Kibana-backend re-route the user to keycloak for authentication
• Kibana is connected to the api-server to collect data, instead of elasticsearch
• api-server is implementing basic authorization.

Currently the authorization is: DELETE forbidden on all indexes.
However this is a bit restrictive: you cannot delete an index pattern for example.
Index patterns are stored on .kibana.

Proposed solution:
Implement authorization based on index names. e.g. .kibana index can be modified only by administrators.
Normal users can access the index that corresponds to a domain they own.

Currently, the socials endpoints look like:

POST /domains/{domain}/socials
{
  "username": "cdupont",
  "channel": "facebook",
  "message": "Test message"
}

It would be easier to secure with explicit endpoint names for each service:

POST /domains/{domain}/socials/sms
{
  "username": "cdupont",
  "message": "Test message"
}
POST /domains/{domain}/socials/facebook
{
  "username": "cdupont",
  "message": "Test message"
}
POST /domains/{domain}/socials/twitter
{
  "username": "cdupont",
  "message": "Test message"
}

Specification for the domains usage and management.

Domains creation

Registered users can create domains:

POST /api/v1/domains {"id":"farm1"}

Domains deletion

Domains can be deleted by its owner or an admin.

DELETE /api/v1/domains/farm1

Deleting a domain will also delete all resources that belongs to that domain.

Resource creation

A resource is created under a domain. For example:

POST /api/v1/domains/farm1/sensors -d '{"id": "Sensor1"}'

If the domain doesn't exist, the creation is rejected with 404 "Domain doesn't exists".
If the domain exists, the resource is created. The domain is recorded as an attribute in the entity.

Resource view

You can view a resource using its domain.

GET  /api/v1/domains/farm1/sensors

Will return only the sensors belonging to the domain "farm1".

GET  /api/v1/domains/*/sensors

Will return all the sensors.

Resource update

You can update a specific resource using the domain:

PUT  /api/v1/domains/farm1/sensors/Sensor1

If the domain doesn't exists, a 404 will be returned.
If the domain exists but the resource doesn't belong to it, an error code is returned.
For specific resources, a wildcard can be used for the domain:

PUT  /api/v1/domains/*/sensors/Sensor1

Using the wildcard allows to ignore the restrictions on the domain and to update directly the resource.

Move from one domain to another

The user can move his resource form one domain to another:

PUT  /api/v1/domains/farm1/sensors/Sensor1 "farm2"

An error is raised is the destination domain doesn't exists.

Gateways

Gateways can create a sensor in its own domain:

POST /api/v1/domains/farm1/sensors -d '{"id": "Sensor1"}'

However, the user should make sure that the domain exists beforehand.
That's why it might be better to create the sensor in a "standard" domain:

POST /api/v1/domains/waziup/sensors -d '{"id": "Sensor1"}'

The user will then be able to move the sensor to another domain.
To push the data regardless of domain, the gateway should use the wildcard:

POST /api/v1/domains/*/sensors/Sensor1/measurements/TC/values {"value": "25.6", "timestamp": "2016-06-08T18:20:27.873Z"}

To help retrieving big amounts of sensor datapoints, we need to use a pagination mechanism.
We can use those query strings:

• vLastN: Only the requested last datapoints will be returned. Datapoints are sorted with the most recent datapoint first. It is on by default, with a value of 20.
• vLimit: In case of pagination, the number of datapoints per page.
• vOffset: In case of pagination, the offset to apply to the requested search of datapoints.
• vDateFrom: The starting date and time from which the datapoints are desired. It is an optional parameter.
• vDateTo: The final date and time until which the datapoints are desired. It is an optional parameter.

Example of query:

curl -X GET "http://dev.waziup.io:80/api/v1/domains/waziup/sensors/mySensor/measurements/TC/values?vLimit=50&vOffset=0&vDateFrom=2016-01-01T00:00:00.000Z&vDateTo=2016-01-31T23:59:59.999Z"

This query will retrieve the first 50 datapoints from January 2016.