Reference: https://arxiv.org/abs/1802.03888.

Fitting the same model here, we can construct PDPs, ICE curves, and VIPs with minimal code using the pred.fun argument to pdp::partial():

# Centered ICE curves
library(pdp)
pfun <- function(object, newdata) {
  predict_proba(object, x = as.matrix(newdata)) %>%
    as.vector()  # %>% mean()  # uncomment for PDP instead of ICE curves
}
pd <- partial(model_keras, pred.var = "TotalCharges", progress = "text",
              pred.fun = pfun, train = as.data.frame(x_train_tbl))
autoplot(pd, alpha = 0.01, center = TRUE)

# Variable importance plot (VIP)
library(vip)
vip(model_keras, method = "ice", feature_names = names(x_train_tbl), 
    pred.fun = pfun, train = as.data.frame(x_train_tbl), num_features = 35)

Similar to the TODO list on RBitmojis README page

Some support for regression models is there, but it needs to be tested! Also, h2o.varimp() output is model specific!

Hi. I am using vip to measure variable importance scores from my super learner model.
And I get the following error message:

install.packages("vip")
library(vip)
set.seed(150) # for reproducibility
sl <- SuperLearner(Y = label, X = train,
SL.library=c("SL.randomForest", "SL.glmnet", "SL.svm"),
method = "method.NNLS", verbose=TRUE)
words <- colnames(train)
p1 <- vi(sl, method = "perm", obs = label, feature_names = words)
Error: Permutation-based variable importance scores not yet implemented.

What does this error message mean?
Also, does vip support super learner models?
I would appreciate your help. Thanks!

Added vint function.

Thanks to @mpane21 for the suggestion!

For example, vi() should always check for is.null(feature_names), etc.

So far, the max difference seems the most reasonable; diff(range(yhat)).

Should have methods for:
rpart
party
partykit

Columns should be, for example, "Variable", "Importance", and "Sign/direction".

The nsim argument (which will default to 1) enables multiple replications of shuffling each feature to reduce Monte Carlo error and help stabilize results from run to run (and maybe even produce standard deviations?).

imp <- vi_permute(fit, nsim = 10, ...)

Maybe something like...

vip(model, type = "permute", train = trn, metric = "rmse")

Extracts the relative rank of a particular feature. For example,

feature_rank(rfo, feature_names = c("rm", "lstat"))
# [1] 1 2

Are we interested in incorporating our own measures for local vip (i.e. Shapley, LIME)?

E.g.,

vi(fit, method = "permute", metric = "rmse", sample_size = 50)

• Unnecessary check on feature_names argument;
• Passes feature_names to pdp::partial() using the wrong argument name.

Argument FUN could be a list which, for example, defaults to

FUN = list(
  "con" = stats::sd,  # continuous
  "cat" = function(x) diff(x) / 4  # categorical
)

However, the denominator in "cat" should depend on the number of categories, say K.

This would support, for starters, vi_model.nnet(). keras would be nice too!

Should not be hard to reach >90% test coverage.

Using method = "ice" it is possible to compute VI scores for each individual case. For example,

vi(fit, method = "ice", case = TRUE)

What's the best way to plot this?

For example, bar = FALSE might give a dotchart instead.

Add options to control bar width (when bar = TRUE) and orientation (e.g., horizontal = TRUE).

Should check object$forest$independent.variable.names first since write.forest = TRUE by default in ranger. Otherwise, should throw an error.

See, for example, the code here: http://mevik.net/work/software/VIP.R

Update the range-based statistic (currently, range/4) to adapt to the number of categories K.

What are your thoughts on using ggplot2 >= 3.0.0, moving dplyr >= 0.7.0 to Imports, and getting rid of plyr (no commits in over two years) and magrittr/tibble (dplyr exports the pipe and tibble()) from Imports? I could do a pull request if you're intersted.

Helper functions:

vi_model()  # a generic with various methods: vi_model.randomForest()
vi_partial()
vi_permute()
vi_shapely()  # <- this would def be nice!
...

which would then be called by vi() as in

vi(fit, type = "model", ...)

Note: will have to figure out how to deal with naming conflicts! For example, if fit is a "randomForest" object, then issues would arise with

vi(fit, type = "model", type = 2)

One work around is to prefix argument names with a . like in plyr:

vi(fit, .type = "model",  .feature_names = c("x1", "x2"), 
   # Arguments to be passed on through ...
    type = 2)

Computed using the squared (Pearson) correlation between the transformed training data found using acepack::ace().

Source: https://www.quora.com/Correlation-can-measure-only-the-linear-relationship-between-variables-What-are-the-methods-for-measuring-non-linear-relationships-between-two-variables

See https://www.tidyverse.org/lifecycle/#experimental

Using ICE-curves (or c-ICE curves), we can compute case-specific VIs similar to the reason codes provided by DataRobot!

vi(fit, relative = TRUE) # scaled so largest is 100

For example, when partial = TRUE, all PDPs get constructed (which can be expensive) even when num_features is set.