y-ohr-n / kenchi Goto Github PK

A scikit-learn compatible library for anomaly detection

License: BSD 3-Clause "New" or "Revised" License

Python 99.72% Makefile 0.22% Batchfile 0.04% Shell 0.02%

data-science statistics machine-learning data-mining anomaly-detection outlier-detection novelty-detection one-class-classification scikit-learn

kenchi's Introduction

kenchi

This is a scikit-learn compatible library for anomaly detection.

Dependencies

Required dependencies
1. numpy>=1.13.3 (BSD 3-Clause License)
2. scikit-learn>=0.20.0 (BSD 3-Clause License)
3. scipy>=0.19.1 (BSD 3-Clause License)
Optional dependencies
1. matplotlib>=2.1.2 (PSF-based License)
2. networkx>=2.2 (BSD 3-Clause License)

Installation

You can install via pip

pip install kenchi

or conda.

conda install -c y_ohr_n kenchi

Algorithms

Outlier detection
1. FastABOD¹
2. LOF² (scikit-learn wrapper)
3. KNN³,⁴
4. OneTimeSampling⁵
5. HBOS⁶
Novelty detection
1. OCSVM⁷ (scikit-learn wrapper)
2. MiniBatchKMeans
3. IForest⁸ (scikit-learn wrapper)
4. PCA
5. GMM (scikit-learn wrapper)
6. KDE⁹ (scikit-learn wrapper)
7. SparseStructureLearning¹⁰

Examples

import matplotlib.pyplot as plt
import numpy as np
from kenchi.datasets import load_pima
from kenchi.outlier_detection import *
from kenchi.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

np.random.seed(0)

scaler = StandardScaler()

detectors = [
    FastABOD(novelty=True, n_jobs=-1), OCSVM(),
    MiniBatchKMeans(), LOF(novelty=True, n_jobs=-1),
    KNN(novelty=True, n_jobs=-1), IForest(n_jobs=-1),
    PCA(), KDE()
]

# Load the Pima Indians diabetes dataset.
X, y = load_pima(return_X_y=True)
X_train, X_test, _, y_test = train_test_split(X, y)

# Get the current Axes instance
ax = plt.gca()

for det in detectors:
    # Fit the model according to the given training data
    pipeline = make_pipeline(scaler, det).fit(X_train)

    # Plot the Receiver Operating Characteristic (ROC) curve
    pipeline.plot_roc_curve(X_test, y_test, ax=ax)

# Display the figure
plt.show()

References

Kriegel, H.-P., Schubert, M., and Zimek, A., "Angle-based outlier detection in high-dimensional data," In Proceedings of SIGKDD, pp. 444-452, 2008.↩
Breunig, M. M., Kriegel, H.-P., Ng, R. T., and Sander, J., "LOF: identifying density-based local outliers," In Proceedings of SIGMOD, pp. 93-104, 2000.↩
Angiulli, F., and Pizzuti, C., "Fast outlier detection in high dimensional spaces," In Proceedings of PKDD, pp. 15-27, 2002.↩
Ramaswamy, S., Rastogi, R., and Shim, K., "Efficient algorithms for mining outliers from large data sets," In Proceedings of SIGMOD, pp. 427-438, 2000.↩
Sugiyama, M., and Borgwardt, K., "Rapid distance-based outlier detection via sampling," Advances in NIPS, pp. 467-475, 2013.↩
Goldstein, M., and Dengel, A., "Histogram-based outlier score (HBOS): A fast unsupervised anomaly detection algorithm," KI: Poster and Demo Track, pp. 59-63, 2012.↩
Scholkopf, B., Platt, J. C., Shawe-Taylor, J. C., Smola, A. J., and Williamson, R. C., "Estimating the Support of a High-Dimensional Distribution," Neural Computation, 13(7), pp. 1443-1471, 2001.↩
Liu, F. T., Ting, K. M., and Zhou, Z.-H., "Isolation forest," In Proceedings of ICDM, pp. 413-422, 2008.↩
Parzen, E., "On estimation of a probability density function and mode," Ann. Math. Statist., 33(3), pp. 1065-1076, 1962.↩
Ide, T., Lozano, C., Abe, N., and Liu, Y., "Proximity-based anomaly detection using sparse structure learning," In Proceedings of SDM, pp. 97-108, 2009.↩

kenchi's People

Contributors

Stargazers

Watchers

Forkers

adripurkayastha tungk deltat99 zuomatthew abhimalhotra00 evans-m ahoyosid mikeshihyaolin exp-time-series-tools adityakumarsinha

kenchi's Issues

Return a different value each time a score method is called

Consider whether if_delegate_has_method is needed

Add a fetch_aloi function

References

Kriegel, H.-P., Kroger, P., Schubert E., and Zimek, A., "Interpreting and unifying outlier scores," In Proceedings of SDM, pp. 13-24, 2011.

Create a Code Climate configuration file

Add notebook examples

plot the ROC curves
compare various methods using 2-dimensional data
measure execution time of various methods
tune hyperparameters using the Lee-Liu metric
use UMAP and HDBSCAN
use explanation models

Add a predict_proba method to a BaseDetector class

Measure test coverage with Code Climate

Add kenchi to related projects of scikit-learn

Implement Iterative sampling

References

Wu, M., and Jermaine, C., "Outlier detection by sampling with accuracy guarantees," In Proceedings of SIGKDD, pp. 767-772, 2006.

Add a load_wilt function

References

Goix, N., "How to evaluate the quality of unsupervised anomaly detection algorithms?" In ICML Anomaly Detection Workshop, 2016.

Separate novelty_detection module from outlier_detection module

Add sample usages presented as doctests

Implement iForest

References

Liu, F. T., Ting K. M., and Zhou, Z.-H., "Isolation forest," In Proceedings of ICDM, pp. 413-422, 2008.

Launch binder with a Dockerfile

Description

Insert the following command in a Dockerfile

pip install --upgrade pip setuptools

Measure the time for fitting the outlier detector

Compute anomaly scores in parallel

Implement the function that compute the area under the MV curve

References

Thomas, A., Clémençon, S., Feuillard V., and Gramfort, A., "Learning hyperparameters for unsupervised anomaly detection," ICML Anomaly Detection Workshop, 2016.

Add a plot_featurewise_anomaly_score function

Test that the predict method in kenchi returns the same value as the predict method in sklearn

Create Issue and Pull Request templates

Investigate SVDD

Tax, D. M. J., and Duin, R. P. W., "Support Vector Data Description," Machine Learning, 54, pp. 45-66, 2004.

Write metadata and options in setup.cfg

Add a logging decorator

NegativeMVAUCScorer returns 0 in most cases

Implement LoOP

References

Kriegel, H.-P., Kroger, P., Schubert E., and Zimek A., "LoOP: local outlier probabilities," In Proceedings of CIKM, pp. 1649-1652, 2009.

Add a fetch_adult function

References

Goix, N., "How to evaluate the quality of unsupervised anomaly detection algorithms?" In ICML Anomaly Detection Workshop, 2016.

data_volume_ should be computed outside a cross validation loop

FastABOD computes the anomaly score for each training data twice

Add an offset_ attribute to a BaseDetector class

Add a plot_mv_curve function

Implement the function that compute the Lee-Liu metric

References

Lee, W. S, and Liu, B., "Learning with positive and unlabeled examples using weighted Logistic Regression," In Proceedings of ICML, pp. 448-455, 2003.

Add a load_annthyroid function

References

Goix, N., "How to evaluate the quality of unsupervised anomaly detection algorithms?" In ICML Anomaly Detection Workshop, 2016.

Implement FBOD

References

Lazarevic, A., and Kumar, V., "Feature bagging for outlier detection," In Proceedings of SIGKDD, pp. 157–166, 2005.

Create JSON files that have the default parameter grid to explore

See below

TPOT
auto-sklearn
hyperopt-sklearn
auto_ml

Use Pipenv

Description

matplotlib and networkx are optional.

Remove a score method in a BaseDetector class

Add a to_pickle method

Output a pandas dataframe

Raise AttributeError when the negative_mv_auc_score receives a Pipeline

Implement AKNN

References

Angiulli, F., and Pizzuti, C., "Fast outlier detection in high dimensional spaces," In Proceedings of PKDD, pp. 15-27, 2002.

Upload to conda-forge

Implement Regularizer classes that regularize the anomaly score

Description

BaselineRegularizer
LinearInvRegularizer
LogInvRegularizer

References

Kriegel, H.-P., Kroger, P., Schubert E., and Zimek, A., "Interpreting and unifying outlier scores," In Proceedings of SDM, pp. 13-24, 2011.

Output a Bunch that has a feature_names attribute

Implement OCSVM

Add a load_pima function

References

Goix, N., "How to evaluate the quality of unsupervised anomaly detection algorithms?" In ICML Anomaly Detection Workshop, 2016.

Add common tests for outlier detection

Description

test_check_estimator
test_fit_predict
test_fit
test_predict
test_decision_function
test_score_samples
test_anomaly_score
test_plot_anomaly_score
test_plot_roc_curve
test_predict_notfitted
test_decision_function_notfitted
test_score_samples_notfitted
test_anomaly_score_notfitted
test_plot_anomaly_score_notfitted
test_plot_roc_curve_notfitted

Implement HBOS

References

Goldstein, M., and Dengel, A., "Histogram-based outlier score (HBOS): A fast unsupervised anomaly detection algorithm," KI: Poster and Demo Track, pp. 59-63, 2012.

Add a fetch_kddcup99 function

References

Kriegel, H.-P., Kroger, P., Schubert E., and Zimek, A., "Interpreting and unifying outlier scores," In Proceedings of SDM, pp. 13-24, 2011.

Add a score_samples method to a BaseDetector class

Implement Normalizer classes that normalize the anomaly score

Description

LinearNormalizer
GaussianNormalizer
GammaNormalizer

References

Kriegel, H.-P., Kroger, P., Schubert E., and Zimek, A., "Interpreting and unifying outlier scores," In Proceedings of SDM, pp. 13-24, 2011.

y-ohr-n / kenchi Goto Github PK

kenchi's Introduction

kenchi

Dependencies

Installation

Algorithms

Examples

References

kenchi's People

Contributors

Stargazers

Watchers

Forkers

kenchi's Issues

References

References

References

References

Description

References

References

References

References

References

References

Description

References

Description

References

References

Description

References

References

Description

References

Recommend Projects

Recommend Topics

Recommend Org