SHAP in unsupervised anomaly detection about shap HOT 7 CLOSED

Cool application. What you want is an XGBoost model that would do a good job identifying new anomalies in future data. If your model doesn't, then it is not capturing real structure in the data.

So since SHAP values are only as meaningful as the model they explain, I would for sure do a train test split to determine the XGBoost parameters. However, once you have those params I would also then retrain the XGBoost model on the whole dataset for a bit of extra accuracy before explaining it.

from shap.

Thank you for you comments. I absolutely agree that the XGBoost should be able to capture the real structure of the data.

So, are you suggesting doing cross-validation to get the optimal hyper-parameters and then re-train the model on the whole dataset (and make predictions on the same one)?

from shap.

from shap.

Is it possible to output shap values out of an Isolation Forest by its own, without the need to use xgboost?

I have tried to compute the shap values for Anomaly Detection with Local Outlier Factor, SVM and Isolation Forest. They all output -1 or 1 with respect to if the observation is an outlier or not, respectively.

My question is whether we need xgboosting to calculate probabilities to find outliers and then compute the shap values, or if there is another way of computing the shap values of these methods by its own?

Every time that I have run the experiment with the outcome labelled -1 or 1, I couldn't match the results because they all yielded 1 as if the shap values don't go well with these models...

from shap.

I am using Anomaly Detection with Local Outlier Forest, One Class Svm ,Isolation Forest .They have Output as an outlier 1 or -1.

I am trying to use Shap For finding interpretation of these models but It is saying These Explainer doesnot go well with these models except for isolation forest.

Is there any explainer that works with Local Outlier Forest and One Class Svm

from shap.

@Harshit200107 did you get a solution? if not what did you use as an alternative? I have the same problem

from shap.

the LOF in sklearn provides a method called decision_function which outputs the raw scores of the prediction. Using this will give better explanation of the features. But the novelty parameter should be set to True for this

from shap.