waziup / els-feeder Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 4.0 67 KB

From Orion to ElasticSearch

JavaScript 99.36% Shell 0.64%

els-feeder's People

Contributors

Watchers

Forkers

fuinha muskanmahajan486 ronak877

els-feeder's Issues

Error: No Living connections

When started with the platform, feeder complains with:

els-feeder_1      | info: index: "waziup"
els-feeder_1      | info: Creating/updating an index for waziup
els-feeder_1      | info: Listening on port  9000
els-feeder_1      | Elasticsearch ERROR: 2017-12-01T22:10:20Z
els-feeder_1      |   Error: Request error, retrying
els-feeder_1      |   HEAD http://elasticsearch:9200/waziup => connect ECONNREFUSED 172.19.0.4:9200
els-feeder_1      |       at Log.error (/opt/feeder/node_modules/elasticsearch/src/lib/log.js:225:56)
els-feeder_1      |       at checkRespForFailure (/opt/feeder/node_modules/elasticsearch/src/lib/transport.js:258:18)
els-feeder_1      |       at HttpConnector.<anonymous> (/opt/feeder/node_modules/elasticsearch/src/lib/connectors/http.js:157:7)
els-feeder_1      |       at ClientRequest.bound (/opt/feeder/node_modules/lodash/dist/lodash.js:729:21)
els-feeder_1      |       at emitOne (events.js:96:13)
els-feeder_1      |       at ClientRequest.emit (events.js:191:7)
els-feeder_1      |       at Socket.socketErrorListener (_http_client.js:358:9)
els-feeder_1      |       at emitOne (events.js:96:13)
els-feeder_1      |       at Socket.emit (events.js:191:7)
els-feeder_1      |       at emitErrorNT (net.js:1279:8)
els-feeder_1      |       at _combinedTickCallback (internal/process/next_tick.js:80:11)
els-feeder_1      |       at process._tickCallback (internal/process/next_tick.js:104:9)
els-feeder_1      | 
els-feeder_1      | Elasticsearch WARNING: 2017-12-01T22:10:20Z
els-feeder_1      |   Unable to revive connection: http://elasticsearch:9200/
els-feeder_1      | 
els-feeder_1      | Elasticsearch WARNING: 2017-12-01T22:10:20Z
els-feeder_1      |   No living connections
els-feeder_1      | 
els-feeder_1      | error: ERROR in checking index Error: No Living connections
els-feeder_1      |     at sendReqWithConnection (/opt/feeder/node_modules/elasticsearch/src/lib/transport.js:225:15)
els-feeder_1      |     at next (/opt/feeder/node_modules/elasticsearch/src/lib/connection_pool.js:213:7)
els-feeder_1      |     at _combinedTickCallback (internal/process/next_tick.js:73:7)
els-feeder_1      |     at process._tickCallback (internal/process/next_tick.js:104:9)
els-feeder_1      | info: Creating an index for  waziup
els-feeder_1      | Elasticsearch WARNING: 2017-12-01T22:10:20Z
els-feeder_1      |   Unable to revive connection: http://elasticsearch:9200/
els-feeder_1      | 
els-feeder_1      | Elasticsearch WARNING: 2017-12-01T22:10:20Z
els-feeder_1      |   No living connections
els-feeder_1      | 
els-feeder_1      | error: ERROR in creating index Error: No Living connections
els-feeder_1      |     at sendReqWithConnection (/opt/feeder/node_modules/elasticsearch/src/lib/transport.js:225:15)
els-feeder_1      |     at next (/opt/feeder/node_modules/elasticsearch/src/lib/connection_pool.js:213:7)
els-feeder_1      |     at _combinedTickCallback (internal/process/next_tick.js:73:7)
els-feeder_1      |     at process._tickCallback (internal/process/next_tick.js:104:9)

It's probably due to the fact that elasticsearch is not started yet.
In a docker-compose environment, it's not possible to guaranty that one component should complete starting before another...
So feeder should probably wait in a loop that elasticsearch and also orion are responding.

Make feeder persistent to Orion and ElasticSearch downs

Display startup message

It would be nice to have some trace at startup, with the configuration (orion and elasticsearch endpoints, for instance).

Config variables

At least 4 environment variables are necessary (values given just for example):

ELASTIC_SEARCH_API="http://elasticsearch"
ORION_API="http://orion:1026"
SERVER_PORT=9000
SERVER_URL=http://localhost:9000

Logging levels

Currently there is some logging but there is no date and level.
The idea is to provide the date and logging levels ("Error", "Warn", "Info", "Debug").
You can use a library such as Winston: https://github.com/Waziup/api-server/blob/master/src/log.js

Config for the platform

The feeder configuration need to be changed for the current platform.
The data format in Orion is:

Service: waziup
Entity id: xxx
Entity type: SensingDevice
The "domain" of an entity is encoded as an attribute.
The measurements are stored as attributes with the type "Measurement".

This is an example of SensingDevice in Orion:

 {
    "id": "Sensor2-ea0541de1ab7132a1d45b85f9b2139f5",
    "type": "SensingDevice",
    "gateway_id": {
      "type": "String",
      "value": "ea0541de1ab7132a1d45b85f9b2139f5",
    },
    "location": {
      "type": "geo:json",
      "value": {
        "type": "Point",
        "coordinates": [
          4.0083,
          5.36
        ]
      },
    },
    "name": {
      "type": "String",
      "value": "My weather station",
    },
    "owner": {
      "type": "String",
      "value": "cdupont",
    },
    "domain": {
      "type": "String",
      "value": "farm1",
    },
    "TC1": {
      "type": "Measurement",
      "value": {
         type: "Number",
         value: "25.6",
         }
      "metadata": {
        "dimension": {
          "type": "String",
          "value": "temperature"
        },
        "name": {
          "type": "String",
          "value": "My garden temperature"
        },
        "sensor_kind": {
          "type": "String",
          "value": "Air temperature sensor"
        },
        "timestamp": {
          "type": "DateTime",
          "value": "2016-06-08T18:20:27.00Z"
        },
        "unit": {
          "type": "String",
          "value": "Degree C"
        }
      }
    }
  }

The data in ELS should be stored in indexes with the name <service>-<domain>.

Provide README file

With some explainations, usage commands, configurations etc.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.