Comments (30)
The bug was in fact in the selection of the threshold. When two predictor values are too close, their mean could not be represented in the ieee 754 arithmetic and would be equal to one or the other value, pretty much arbitrarily depending on the implementation.
> a <- c(0.65354946699793847742, 0.65354946699793858844)
> print(mean(a), digits = 20)
[1] 0.65354946699793847742
> mean(a) == a
[1] TRUE FALSE
Commit 87fceab should fix the issue by selecting the appropriate threshold, only in the case a threshold is identical to some of the predictor values in the first place.
This fix is modifying a core algorithm of pROC (the threshold selection) and I need to do more tests before I push this to CRAN in order to make sure it doesn't break something elsewhere in pROC.
For now the build seems to be passing the automated tests, with tests for both direction
arguments. Some cases may not be handle properly still and will result in a new error message. I can't think of a way this would be triggered, but who knows. As we are talking about near ties, close to the precision limit of the ieee 754 arithmetic, they are effectively ties and I will probably convert those errors to warnings later on.
from proc.
Thank you for your report. Can you please provide a reproducible example that produces this error?
Also please report the output of sessionInfo()
.
from proc.
Just to provide a bit more details about the error:
pROC calculates the AUC with the trapezoidal method. This is the AUC you get when you call auc(roc)
. With your data the AUC value was 0.50032663726931247972.
When using the DeLong method (for roc.test, variance etc), the AUC is also calculated with an other method (similar to the Wilcoxon/Mann-Whitney statistic). With your data the result is 0.50057161522129678399.
These two values should be identical, at least down to something close to the floating point precision of your machine, typically below 10^-8. To be on the safe side, pROC checks this assumption after calculating the DeLong AUC.
For some reason with your data the error was around 0.5%, in the 10^4 range. This is far too high to be caused by numerical approximations of floating point arithmetic, and highly suggestive of a bug, either in the trapezoid or in the DeLong calculation of the AUC.
from proc.
R version 3.4.4 (2018-03-15)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)
Matrix products: default
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] nricens_1.5 survival_2.41-3 lazyeval_0.2.1 ipred_0.9-6 pROC_1.11.0 caret_6.0-78 bindrcpp_0.2 dplyr_0.7.4 plyr_1.8.4 reshape2_1.4.3 ggplot2_2.2.1
[12] lattice_0.20-35
loaded via a namespace (and not attached):
[1] tidyselect_0.2.4 purrr_0.2.4 kernlab_0.9-25 splines_3.4.4 colorspace_1.3-2 stats4_3.4.4 prodlim_1.6.1 rlang_0.2.0 ModelMetrics_1.1.0
[10] pillar_1.2.1 withr_2.1.2 foreign_0.8-69 glue_1.2.0 foreach_1.4.4 bindr_0.1.1 dimRed_0.1.0 lava_1.6 robustbase_0.92-8
[19] stringr_1.3.0 timeDate_3043.102 munsell_0.4.3 gtable_0.2.0 recipes_0.1.2 codetools_0.2-15 psych_1.7.8 parallel_3.4.4 class_7.3-14
[28] DEoptimR_1.0-8 broom_0.4.3 Rcpp_0.12.16 scales_0.5.0 CVST_0.2-1 mnormt_1.5-5 stringi_1.1.7 RcppRoll_0.2.2 ddalpha_1.3.1.1
[37] grid_3.4.4 tools_3.4.4 magrittr_1.5 tibble_1.4.2 tidyr_0.8.0 DRR_0.0.3 pkgconfig_2.0.1 MASS_7.3-49 Matrix_1.2-12
[46] lubridate_1.7.2 gower_0.1.2 assertthat_0.2.0 iterators_1.0.9 R6_2.2.2 rpart_4.1-13 sfsmisc_1.1-2 nnet_7.3-12 nlme_3.1-131.1
[55] compiler_3.4.4
from proc.
The code builds different size models with two sets of covariates (clinical features and biomarker values) in several loops. I use glm from caret, metric = "ROC". This is for a clinical test so I have to limit the specificity and get the highest sensitivity for that specificity. I use pROC to reconstruct the ROC curve. Then I calculate the NRI to see if the clinical features made any improvement in the reclassification of the patients we are interested in.
I know the indices of the loops that caused the code to stop. What do you exactly need?
from proc.
Just to chime in. I just got the same issue today.
I had some code that ran on the very same data, unchanged. Today I upgraded my R version from 3.2.* to 3.4.4, this seems to have caused the problem (I also redownloaded pROC, but the previous download was also within the last few weeks.)
The error that I got was :
A problem occured while calculating DeLong's theta: got 0.79243483788938329226 instead of 0.79307056579783852257. This is a bug in pROC, please report it to the maintainer
I hope this information helps!
from proc.
This seems very concerning and I will give it the highest priority to fix this issue.
This bit of pROC code is thoroughly tested by automated unit tests, which are currently all successfully passing on the CRAN.
This error is obviously triggered by a specific set of data and arguments, so in order to move on I need a reproducible example that triggers this bug. Please provide the exact code and data that triggers this error.
from proc.
@ogroendal and @mirshahin have you managed to resolve the issue? I need a reproducible example to move forward with this issue.
from proc.
Hello,
I had the same issue. I included some files that reproduce the error on a small dataset I was playing with. Please let me know if you need more information. The function is under development.
_R version 3.2.3 (2015-12-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.4 LTS
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=de_DE.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=de_DE.UTF-8
[6] LC_MESSAGES=en_US.UTF-8 LC_PAPER=de_DE.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=de_DE.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] pROC_1.11.0 gdata_2.18.0
loaded via a namespace (and not attached):
[1] colorspace_1.3-2 scales_0.5.0 lazyeval_0.2.1 plyr_1.8.4 tools_3.2.3 pillar_1.1.0 gtable_0.2.0 tibble_1.4.2
[9] yaml_2.1.18 Rcpp_0.12.16 ggplot2_2.2.1 grid_3.2.3 rlang_0.2.0 munsell_0.4.3 gtools_3.5.0_
pROC_debug.zip
from proc.
Thanks a lot eudesbarbosa, this is extremely useful.
The following seems to be a pretty minimal reproducible example:
library(pROC)
roc(
c("A", "A", "A", "A", "A", "A", "A", "A", "B", "B", "B", "B",
"A", "A", "A", "A", "A", "A", "A", "A", "B", "B", "B", "B", "A",
"A", "A", "A", "A", "A", "A", "A", "B", "B", "B", "B", "A", "A",
"A", "A", "A", "A", "A", "A", "B", "B", "B", "B", "A", "A", "A",
"A", "A", "A", "A", "A", "B", "B", "B", "B"),
c(1, 0.98, 1, 0.98, 1, 1, 0.98, 1, 0.14, 0.14, 0.2, 0.2, 1, 1,
1, 0.98, 1, 0.98, 1, 1, 0.16, 0.16, 0.04, 0.12, 1, 1, 1, 1, 1,
1, 1, 1, 0.14, 0.14, 0.14, 0.14, 0.98, 1, 1, 0.98, 1, 1, 1, 1,
0.14, 0.2, 0.16, 0.24, 0.98, 1, 1, 1, 1, 1, 1, 1, 0.18, 0.12,
0.16, 0.22), ci=TRUE, percent=TRUE)
I will take it from there.
from proc.
Until I find out what is going on exactly, and so that you can carry on, setting percent = FALSE appears to be a workaround.
from proc.
This seems to be a minimal example:
r <- roc(c("A", "B", "A", "B"), c(1, 0, 1, 0), percent = TRUE)
ci(r)
It seems to be important to have direction=">".
The following curve also seems to produce erroneous NaN results, and is probably related:
r <- roc(c("A", "B"), c(0, 1))
ci(r)
from proc.
It turns out this specific combination wasn't tested. Fixed in commit 41869d8 (automated tests should fail from now on)
from proc.
The bug is now be fixed on the master branch. I will aim to push it to CRAN during the weekend or early next week.
The reason for the bug was a broken conversion of the roc curve from percent to fractions for internal calculations, followed by a broken check that the DeLong code produced the correct AUC, which in combination caused an error in the specific case where percent=TRUE and direction=">". The check was introduced in pROC version 1.9.1. The bug in the conversion from percent to fraction was present in earlier versions, however it never affected calculations, which is why it was left unnoticed.
@eudesbarbosa thanks a lot for helping pinpointing the bug with a reproducible example.
I tried to run your code and I get unrelated errors about missing errormult
. If you want you can installing the development version of pROC with:
install.packages("devtools")
devtools::install_github("xrobin/pROC")
from proc.
Hey @xrobin
I just installed the newest version from the github repo, but it seems to give the same error as before.
Error in delongPlacements(roc) :
A problem occured while calculating DeLong's theta: got 0.74011194029850746468 instead of 0.74048507462686563585. This is a bug in pROC, please report it to the maintainer.
This is my sessionInfo :
> sessionInfo()
R version 3.4.4 (2018-03-15)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8
[8] LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices datasets utils methods base
other attached packages:
[1] bindrcpp_0.2.2 doBy_4.6-1 gridExtra_2.3 survminer_0.4.2 ggpubr_0.1.6 survival_2.42-3 ggplot2_2.2.1 pROC_1.11.9 purrr_0.2.4 lubridate_1.7.4 dplyr_0.7.4 magrittr_1.5
[13] nvimcom_0.9-71 colorout_1.2-0
loaded via a namespace (and not attached):
[1] zoo_1.8-1 reshape2_1.4.3 splines_3.4.4 lattice_0.20-35 colorspace_1.3-2 survMisc_0.5.4 rlang_0.2.0 pillar_1.2.2 foreign_0.8-69 glue_1.2.0
[11] withr_2.1.2 bindr_0.1.1 plyr_1.8.4 stringr_1.3.0 munsell_0.4.3 gtable_0.2.0 devtools_1.13.5 psych_1.8.3.3 memoise_1.1.0 labeling_0.3
[21] knitr_1.20 curl_3.2 parallel_3.4.4 broom_0.4.4 Rcpp_0.12.16 xtable_1.8-2 scales_0.5.0 cmprsk_2.2-7 km.ci_0.5-2 mnormt_1.5-5
[31] digest_0.6.15 stringi_1.1.7 KMsurv_0.1-5 grid_3.4.4 tools_3.4.4 lazyeval_0.2.1 tibble_1.4.2 tidyr_0.8.0 pkgconfig_2.0.1 MASS_7.3-49
[41] Matrix_1.2-12 data.table_1.10.4-3 assertthat_0.2.0 httr_1.3.1 R6_2.2.2 nlme_3.1-131 git2r_0.21.0 compiler_3.4.4
from proc.
@ogroendal can you please also send the code and data that trigger this error? Your sessionInfo doesn't point to anything obvious.
from proc.
@xrobin , thanks for the quick feedback and I'm glad I could help a little =)
The errormult
error is because I didn't included that particular function in the code I sent you. Nothing to worry about, it runs fine with the "percent = FALSE".
from proc.
@eudesbarbosa can you please confirm if the fix working for you (ie can you use "percent=TRUE" now)?
from proc.
After installing the development version of pROC, I no longer have the error. It seems to be working fine with "percent = TRUE".
from proc.
@eudesbarbosa thanks for the confirmation, and I'm glad to hear it works for you now. So we're really looking at two different issues here.
@mirshahin @ogroendal please provide a reproducible example. I've been asking for it for a month now, this bug is not going to get fixed until I can reproduce it. Please post the command you're running with the exact data that you feed into it (use dump
to dump it as a command that can be used to re-generate it).
from proc.
With some googling and a bit of luck, I found a cached page of the automated checks of an old version of a CRAN package triggering that error and could isolate a reproducible example from it:
response <- c(2, 1, 1, 2, 2, 1, 2, 2, 1, 1, 1, 2, 1, 2, 2, 2, 2, 2)
predictor <- c(0.960602681556147, 0.0794407386056549, 0.144842404246611,
0.931816485855784, 0.931816485855784, 0.97764041048215, 0.653549466997938699464,
0.796401132206396, 0.427720540184519, 0.811278021288732, 0.0188323116581187,
0.653549466997938588442, 0.653549466997938477419, 0.959111701445925, 0.931816485855784,
0.663663279418747, 0.800100838413179, 0.780456095511079)
r <- roc(response, predictor)
ci(r)
var(r)
There are 3 nearly but not exactly identical predictor values (which dump doesn't preserve).
Due to the values being so close to each other, they are effectively identical to the threshold chosen by pROC:
> r$thresholds[6] == predictor
[1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[13] TRUE FALSE FALSE FALSE FALSE FALSE
> r$thresholds[7] == predictor
[1] FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE
[13] FALSE FALSE FALSE FALSE FALSE FALSE
Also different algorithms handle this case differently:
> auc(r)
Area under the curve: 0.7727
> auc(roc(response, predictor, algorithm=2))
Area under the curve: 0.7792
from proc.
Hello,
Also received the error today:
Error in delongPlacements(roc) :
A problem occured while calculating DeLong's theta: got 0.74135039370078736898 instead of 0.74135826771653545730. This is a bug in pROC, please report it to the maintainer.
pROC worked fine with this exact data until I needed to uninstall and reinstall R 5 days ago (working now with version 3.4.0)
I'm running several roc() and have this problem only with one of them.
This is my code (data attached):
roc2<-roc(proc_problem$var1, proc_problem$var_works) plot=T,add=T,lty=2,ci=T,levels=c(0, 1))
roc3<-roc(proc_problem$var1, proc_problem$var_problem, plot=T, add=T,lty=3,ci=T,levels=c(0, 1))
The "percent = FALSE" fix didn't work for me.
proc_problem.txt
Thanks!
from proc.
@ealeph could you please test with the latest version on github to confirm if the fix works for you?
if (! requireNamespace("devtools")) install.packages("devtools")
devtools::install_github("xrobin/pROC")
I must say your data works fine for me. It could be because it wasn't saved with enough precision in the CSV file, or something else.
> proc_problem <- read.csv("proc_problem.txt")
> roc3<-roc(proc_problem$var1, proc_problem$var_problem, plot=T, lty=3,ci=T,levels=c(0, 1))
> roc3
Call:
roc.default(response = proc_problem$var1, predictor = proc_problem$var_problem, levels = c(0, 1), ci = T, plot = T, lty = 3)
Data: proc_problem$var_problem in 1016 controls (proc_problem$var1 0) < 125 cases (proc_problem$var1 1).
Area under the curve: 0.7414
95% CI: 0.6915-0.7914 (DeLong)
If my analysis is correct, the bug shouldn't be dependent on a specific R version. pROC has been checking for this inconsistency since pROC 1.9.1 released more than a year ago (Feb. 2017). Unless you were using a very outdated version I don't see why you didn't get the same error. So I would really appreciate if you could check that the fix indeed solves your problem. As you can read I'm pretty much shooting in the dark here.
from proc.
from proc.
@ealeph you can clone the repository locally with git clone https://github.com/xrobin/pROC.git
or download the code as a zip file, and then use devtool's install_local, pointing to the folder where the source code is:
devtools::install_local("pROC")
#or
devtools::install_local("pROC-master")
If you have RStudio you can also do File > Open Project and point to the default.Rproj file within the git repository that you just cloned/downloaded. Once it is open you can use menu Build > Install and Restart.
The RData file you sent wasn't copied to the GitHub issue so I can't test unfortunately.
from proc.
I am preparing a CRAN release. I tweaked the error message a bit, and made it dump some relevant data into a file in order to make it easier to debug if the problem persists.
from proc.
In addition, I ran the full test suite (including testthat tests with RUN_SLOW_TESTS and man page examples with \dontrun
) with a stop()
in if (any(o <- outer(thresholds, predictor,
==))) {
to make sure this is not triggered in normal use. All tests ran fine so I am happy it shouldn't impact the calculations in normal cases.
from proc.
pROC 1.12.0 is on CRAN now. Please update:
install.packages("pROC")
It may take a few hours for Mac/Windows binaries to be available. If you still get the error, please open a new issue with the output of sessionInfo()
and either steps to reproduce or the contents of the pROC_bug.RData dump file (make sure it doesn't contain sensitive data first before posting it as it will be available publicly).
from proc.
from proc.
Sure! By the way please make sure you get 1.12.1 because 1.12.0 has a major performance regression.
from proc.
Related Issues (20)
- ggroc.list parameter legacy.axes break HOT 2
- One-sided CIs for AUCs HOT 2
- Averaging 10 ROC curves HOT 4
- How to print the threshold without specificity and sensitivity HOT 2
- Cannot create a roc curve with a formula and a with clause HOT 2
- CRAN submission failed with new message Apparent methods for exported generics not registered
- Fix warning: `aes_string()` was deprecated in ggplot2 3.0.0 HOT 1
- Move aes_string() to aes() HOT 1
- Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0. HOT 1
- The `path` argument of `expect_doppelganger()` is deprecated as of vdiffr 1.0.0. HOT 1
- Uncaught warnings in tests HOT 1
- Support for spaces in column names with formula
- A non-monotonic ROC is being produced by ggroc HOT 2
- Obuchowski and McClish (1997) sample size calculation incorrect HOT 6
- Mean ROC curve in ggroc() HOT 5
- pROC, detectable AUC HOT 2
- What does "direction" mean in roc function HOT 3
- Default method parameter in ci.auc function is different from documentation HOT 1
- Example for AUPRC with confidence interval HOT 1
- Incorrect AUC value and CI [bug] HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from proc.