sevvandi / dobin Goto Github PK
View Code? Open in Web Editor NEWDimension reduction for outlier detection
Home Page: https://sevvandi.github.io/dobin/
License: Other
Dimension reduction for outlier detection
Home Page: https://sevvandi.github.io/dobin/
License: Other
The main dobin
procedure actually turns out to be unusable for me because it's simply too slow. My current data aren't huge, 100,000s of rows, around 100 columns, but clearly way too big for dobin
. The main "bottleneck" is the calculation of RANN::nn2
, which has to be re-calculated on every iteratively reduced matrix:
Line 11 in 04453c1
The following code illustrates just one of several available alternatives that is a lot more efficient:
nrow <- 10000
ncol <- 50
x <- array (runif (nrow * ncol), dim = c (nrow, ncol))
n <- floor (10 ^ (8:16 / 4))
k <- 20
res <- vapply (n, function (i) {
xtest <- x [seq (i), ]
bench::mark (
nn_obj <- RANN::nn2 (xtest, xtest, k = k),
nn_obj <- dbscan::kNN (xtest, k = k),
check = FALSE,
time_unit = "s")$median },
numeric (2))
#> Warning: Some expressions had a GC in every iteration; so filtering is disabled.
#> Warning: Some expressions had a GC in every iteration; so filtering is disabled.
res <- data.frame (n = n,
RANN = res [1, ],
dbscan = res [2, ])
res <- tidyr::gather (res,
key = "method",
value = "duration",
RANN, dbscan)
library (ggplot2)
ggplot (res, aes (x = n, y = duration, colour = method)) +
geom_line () +
geom_point ()
Created on 2021-09-02 by the reprex package (v2.0.0.9000)
dbscan:kNN
is at least twice as fast as RANN::nn2
, and scales much better.
That is nevertheless unlikely to make dobin
useable at scale. I suspect it may be necessary to reconsider the brute-force knn
calls, and hand-code some sort of transformation of former neighbour relationships into your new B
-basis. Updated neighbour relationships change very little, especially in the early (high-dimensional) stages, so there's a lot of unnecessary processing going on recalculating those from scratch each time. Happy to discuss approaches if and when things get that far, but at least dropping RANN
will help us along the way. Thanks!
I've started playing around with this package for my own work, for which it's looking really promising. Also opens up a very interesting "can-o-worms" regarding justification of which ordination method one chooses ๐ ... but that discussion can be held some other time and place. In the meantime, as a first observation:
It would be useful to have a couple of pre-processing steps to make the package more "user friendly." One would be to do a simple check for any non-numeric columns and reduce the input object, xx
, down to numeric columns only. This is done in most stats
routines including cmdscale
, prcomp
, and the like, which just issue warnings yet still process the data. dobin
currently errors with very uninformative message, Error in max.z - min.z : non-numeric argument
.
Thanks for package.
For univariate time series, I used the timesteps of measured values as a dummy second dimension, and got mixed results, in terms of what I'd visually identify as outliers.
Is dobin applicable to univariate time series to start? If so, is using timesteps the correct approach?
Extending from #2, another useful pre-processing step would be to enable user-control over handling of NA values. Current routines simply assume there are none, and error if any are encountered. Many stats
routines have useNA
or na.rm
-type arguments, and there are also the stand-alone functions like na.omit
. Exposing at least some minimal options which default to something like na.omit
would greatly improve usability, through enabling NAs to be processed rather than current uninformative error. Even better would be to enable domain-specific options along the lines of zoo::na.approx()
. Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.