Giter Club home page Giter Club logo

anomaly-detection-template's Introduction

ipm package: anomaly-detection

Overview

Run a self-training anomaly detection algorithm against a given dataset. The selected algorithm is Moving Median Decomposition as seen in this whitepaper for Anomaly Detection for Predictive Maintenance. This Library is for identification of items, events or observations which do not conform to an expected pattern or other items in a dataset.

This is an ipm package, which contains one or more reusable assets within the ipm Community. The 'package.json' in this repo is a ipm spec's package.json, here, which is a superset of npm's package.json spec, here.

Browse ipm Packages

Setup

Basic Usage (No Alerts)

Ready out-of-the-box

Advanced Usage (With Alerts)

Requires API keys from Email and SMS Alerts via SendGrid and Twilio.

See AnomalyDetectionConstants Library.

Usage

Portals

AnomalyDetection - Portal for viewing and configuring live datafeeds, rules, and alerts

Code Services

ExampleDetectAnomaly - Runs anomaly detection against an example dataset, returns an array of Anomalies with metadata.

Code Libraries

AnomalyDetection - Core library for detecting anomalies in datasets AnomalyDetectionConstants - Optional credentials for Alerting (SMS via Twilio, Email via Sendgrid) jStat - Dependency on jstat-statistics-toolkit TwilioSMS - Dependency on twilio-sms-library SendGridEmail - Dependency on(https://github.com/rreinold/clearblade-sendgrid-email-integration)

Collections

Anomalies - Tracks all detected anomalies AnomalyConfiguration - Stores user-defined rules and alerts for Anomaly Detection

API

Members

Functions

AnomalyDetection()

Detects Anomalies with Moving Median Decomposition See attached whitepaper for Anomaly Detection for Predictive Maintenance https://files.knime.com/sites/default/files/inline-images/Anomaly_Detection_Time_Series_final.pdf

Run a self-training anomaly detection algorithm against a given dataset

Twilio(user, pass, sourceNumber)

Sends a text message using Twilio's REST API.

DetectAnomaly()AnomalyVisual

Detects an anomaly in the last X MQTT Messages Uses 'AnomalyConfiguration' config to check for a key in the JSON of each message. ex. 'sensor_key' will be set to 'temperature', so we know to pull the temperature key/value pair from a mqtt message like this: {"temperature":40,"humidity":31}

ExampleDetectAnomaly()Array.<Anomaly>

Example logic for detecting anomalies in a dataset of sensor values

SaveAnomalyConfiguration()

Ingests updated anomaly configuration, and saves to AnomalyConfiguration Collection

SendAlert()

This is triggered by an Anomaly being recorded, and inserted into the Anomalies Collection

  • Fetches the anomaly row
  • Checks for alerting configuration
  • Sends email, text alert
SimulateSensorFeed()

Simulates a sensor feed from a sensor network of temperature sensors Publishes NUM_MESSAGES messages to TOPIC

Typedefs

Calibration
Anomaly

SEND_GRID_API_URI

Creator: Robert Reinold Updated: 2017-10-02T00:00:00Z Version: v2.0 Tags: email, sendgrid, marketing, mail, api, rest, http

Usage:

  1. Create a free SendGrid Account.
  2. Log into your SendGrid account, and view the Settings > API Keys tab. Create an API Key with full access to "Mail Send" rights.
  3. Replace <SEND_GRID_API_KEY> with SendGrid API key
  4. Replace <ORIGIN_EMAIL_ADDRESS> with your desired email address. This will be the 'sender' of the email.
  5. Add 'SendGridEmail' as a dependency to your code services (Settings > Requires > Add)

Kind: global variable

AnomalyDetection()

Detects Anomalies with Moving Median Decomposition See attached whitepaper for Anomaly Detection for Predictive Maintenance https://files.knime.com/sites/default/files/inline-images/Anomaly_Detection_Time_Series_final.pdf

Run a self-training anomaly detection algorithm against a given dataset

Kind: global function
Parameter: number[] dataset array of numbers

AnomalyDetection~calibrate()

Use precomputed anomaly detection calibration profile This can be used to speed up performance for real-time anomaly detection

Kind: inner method of AnomalyDetection

AnomalyDetection~detect() ⇒ Array.<Anomaly>

Run the algorithm against the provided dataset.

Note: This is the most computation-heavy method in this library

Kind: inner method of AnomalyDetection
Returns: Array.<Anomaly> - anomalies - the list of anomalies found in dataset
Parameter: number strictnessOverride (optional) Set the strictness of the anomaly detection

AnomalyDetection~getCalibration() ⇒ Calibration

Fetch the computed anomaly detection calibration parameters

Kind: inner method of AnomalyDetection
Returns: Calibration - calibration

DetectAnomaly() ⇒ AnomalyVisual

Detects an anomaly in the last X MQTT Messages Uses 'AnomalyConfiguration' config to check for a key in the JSON of each message. ex. 'sensor_key' will be set to 'temperature', so we know to pull the temperature key/value pair from a mqtt message like this: {"temperature":40,"humidity":31}

Kind: global function
Returns: AnomalyVisual - data to visualize this anomaly detection

ExampleDetectAnomaly() ⇒ Array.<Anomaly>

Example logic for detecting anomalies in a dataset of sensor values

Kind: global function
Returns: Array.<Anomaly> - anomalies

SaveAnomalyConfiguration()

Ingests updated anomaly configuration, and saves to AnomalyConfiguration Collection

Kind: global function
Parameter: AnomalyConfiguration anomalyConfiguration - map of settings to update

SendAlert()

This is triggered by an Anomaly being recorded, and inserted into the Anomalies Collection

  • Fetches the anomaly row
  • Checks for alerting configuration
  • Sends email, text alert

Kind: global function

SimulateSensorFeed()

Simulates a sensor feed from a sensor network of temperature sensors Publishes NUM_MESSAGES messages to TOPIC

Kind: global function

Calibration

Kind: global typedef
Properties

Name Type Description
min number min value within threshold
max number max value within threshold
strictness number how strict the threshold is against the dataset
medians number intermediary dataset representing the moving medians
pointsPerWindow number calibration, number of points per processing window. may be increased for larger datasets
numWindows number calibration metric derived from pointsPerWindow

Anomaly

Kind: global typedef
Properties

Name Type Description
index number index in the provided dataset
value number value of the data point that is detected as an anomaly

anomaly-detection-template's People

Contributors

rreinold avatar

Forkers

clearblade

anomaly-detection-template's Issues

DetectAnomaly service bugs

Right now the graph in the portal shows max and min values determined by the anomaly detection algorithm, but it does not display the values of the data. This is because the data key in the graph is labeled sensorValue, but the variable name used to create the toViz array sent to the portal is temperature. Changing the variable name from temperature to sensorValue in lines 76 and 77 of DetectAnomaly fixes this.

Also in the DetectAnomaly service, the fetchMessages function currently is set to get data from "sensor" topic regardless of what is in the portal text-boxes. Changing the string to config.topic fixes the problem, allowing users to enter the topic name in the portal.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.