Speed up dedup() about scrubr HOT 3 CLOSED

got some speed ups with a combination of using data.table and fastmatch

microbenchmark(
  `old` = dedup_old(x),
  `new` = dedup(x),
  times = 10
)
#> Unit: milliseconds
#>  expr      min       lq     mean   median       uq      max neval
#>   old 617.9352 674.4674 709.8393 687.4735 766.6044 832.4115    10
#>   new 365.2439 369.0146 395.8602 375.9335 398.7851 524.3056    10

will look for more spots that can be sped up

from scrubr.

This is already an impressive speedup, nice job!

For a data.frame as big as the waxwings one, I'm still afraid that my computer might fail (not because of speed but because of memory?) but I'll try it. 😸

from scrubr.

yeah, it still should be much faster, may try to do some c++ soon (unless all the bottleneck parts are already in it)

from scrubr.