majkamichal / naivebayes Goto Github PK

I was wondering, if further additional parameters of the stats::density() function can be used when executing naive_bayes().

Actually, I am applying the naive_bayes() classifier on a mixed variables data set where most of the numeric data is non-negative. For this reason a log-normal prior-distribution or a KDE, which ensures no probabilities for values < 0 are estimated, seem to be a good choice for my case.
The stats::density() function, which you used for the KDE in naive_bayes() has the argument 'to', which could ensure probablities for values < 0 are zero.

Is it possible to make use of this argument when executing the naive_bayes() with kernel= TRUE?

Many thanks in advance for any reply and best regards
André

Naive Bayes is vulnerable to numerical underflow in the prediction step if the dimensionality of the predictors is much larger than the number of observations. For example, consider the following:

library(naivebayes)
n<-100; k<-2000
X<-matrix(rnorm(n*k),nrow=n)
b<-rnorm(k)
eta<-drop(X%*%b)
y<-rbinom(n,1,plogis(eta))
tr_idx<-1:floor(.8*n)
Xtrn<-X[tr_idx,]
ytrn<-y[tr_idx]
Xtst<-X[-tr_idx,]
ytst<-y[-tr_idx]
fit<-naive_bayes(Xtrn,ytrn,usekernel=TRUE)
preds<-predict(fit,Xtst,type="prob")
head(preds) #they will mostly all be NaN

I believe this is due to the implementation of the log-sum-exp operation. If these lines are replaced with the equivalent, more stable functions from package matrixStats such as logSumExp and/or rowLogSumExps, the underflow issue will probably go away.

In predict.naive_bayes, around line 46 and 50, p might be =0
p[p==0]=threshold should be added to get -Inf for log(p).

This works as expected

library(naivebayes)
m <- naive_bayes(Species ~ Sepal.Width, data=iris)
plot(m)

This crashes

iris$Sepal.Width[1] <- NA
m <- naive_bayes(Species ~ Sepal.Width, data=iris)
plot(m)

Error in seq.default(r[1], r[2], length.out = 512) : 
  'from' must be a finite number

Great package btw. I love how the naive_bayes interface is modeled after base R!

> sessionInfo()
R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.5 LTS

Matrix products: default
BLAS: /usr/lib/openblas-base/libblas.so.3
LAPACK: /usr/lib/libopenblasp-r0.2.18.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=nl_NL.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=nl_NL.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=nl_NL.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=nl_NL.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] naivebayes_0.9.3

loaded via a namespace (and not attached):
[1] compiler_3.5.2 tools_3.5.2    yaml_2.2.0    

Excellent package! The most accessible approach to NB classification that I've found.

I'm wondering if there is a way to extract feature weight/importance from the model? I didn't see any relevant accessors nor any obvious slots in the naive_bayes object.

>     i<-2
>     nbmodel<-naive_bayes(data=trainset, y=trainset$label,x=trainset[2:i],usekernel= TRUE)
>     nbmodel_predict<-predict(nbmodel,as.vector(x_test))
Warning message:
In t(log_sum) + log(prior) :
  Recycling array of length 1 in array-vector arithmetic is deprecated.
  Use c() or as.vector() instead.

I suppose the package does not expect to handle a data-set with just 1 feature? Or i am misunderstanding some fundamental concept here.