mlampros / nmslibr Goto Github PK

I have made several tests with sparse matrix based on the Vignette and /tests folder.
Results are obviously wrong, even more strange is that it seems that there is a transpose of the matrix (I have 700K rows and 14K columns, and when I perform knn_Query_Batch on the full matrix, the max index returned is 14K.

Is there something I need to do to make it work correctly?

dtm_not_prefix_tfidf_scipy$shape
# (76212, 14498)
init_nms = NMSlib$new(input_data = dtm_not_prefix_tfidf_scipy,
                      Index_Params = list('indexThreadQty' = config$cpu),
                      space = 'cosinesimil_sparse',
                      method = 'hnsw',
                      data_type = 'SPARSE_VECTOR',
                      dtype = 'FLOAT',
                      print_progress = TRUE)

 b = init_nms$knn_Query_Batch(query_data = dtm_not_prefix_tfidf_scipy$getcol(1L),
                            k = 5,
                            num_threads = config$cpu)

If I change the space to l1 (same code as above) I get the following error at the init part (there is an error per thread):

Check failed: obj1->datalength() == obj2->datalength()
Check failed: obj1->datalength() == obj2->datalength()
Check failed: obj1->datalength() == obj2->datalength()
Check failed: obj1->datalength() == obj2->datalength()
Check failed: obj1->datalength() == obj2->datalength()
Check failed: obj1->datalength() == obj2->datalength()
Check failed: obj1->datalength() == obj2->datalength()
Check failed: obj1->datalength() == obj2->datalength()
Check failed: obj1->datalength() == obj2->datalength()
Check failed: obj1->datalength() == obj2->datalength()
Check failed: obj1->datalength() == obj2->datalength()
Error in py_call_impl(callable, dots$args, dots$keywords) : 
  RuntimeError: Check failed: it's either a bug or inconsistent data!

Original API separate different steps of the creation of an index (setup, data loading, index loading OR index creation). There is also function to change parameter of search (setQueryTimeParams). Would it be possible to reproduce this API? It would be easier to reproduce the code of the original project and leverage the existing documentation.

Using nmslibR_1.0.3, I'm getting an off by one error in the returned index of the similar row from $KnnQuery().

> x <- rbind( c(1,1,1,1), c(2,2,2,2), c(1,1,50,50), c(2,2,51,51), c(1,2,1,2), c(1,2,51,50) )
> x
     [,1] [,2] [,3] [,4]
[1,]    1    1    1    1
[2,]    2    2    2    2
[3,]    1    1   50   50
[4,]    2    2   51   51
[5,]    1    2    1    2
[6,]    1    2   51   50

> dataBase <- NMSlib$new( x, method= "hnsw", space= "l2" )
> dataBase$Knn_Query( x[1,], k= 1 )
> result
[[1]]
[1] 4

[[2]]
[1] 1.414214

> x[4,]
[1]  2  2 51 51

library(nmslibR)
x = matrix(runif(1000), nrow = 100, ncol = 10)
y = runif(100)
out = KernelKnnCV_nmslib(x, y, k = 5, folds = 5)

cross-validation starts ..
| | 0%
Error in .subset2(public_bind_env, "initialize")(...) :
attempt to apply non-function

version
_
platform x86_64-pc-linux-gnu
arch x86_64
os linux-gnu
system x86_64, linux-gnu
status
major 3
minor 3.3
year 2017
month 03
day 06
svn rev 72310
language R
version.string R version 3.3.3 (2017-03-06)
nickname Another Canoe

I am attempting to convert a dgCMatrix in R to a sci py matrix using the nmslibR package in order to analyze it using the other functions available in nmslibR. When I do this I receive the error in the title. Below is the code to generate a sample matrix and attempt to convert it. I asked this question on stack overflow and was referred here, below is the link to the original question.
https://stackoverflow.com/questions/51137267/converting-from-dgcmatrix-dgrmatrix-to-scipy-sparse-matrix

library(Matrix)
library(nmslibR)
sparse<-Matrix(sample(c(1,2,3,4,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0),10000,
               replace=T),
               ncol=50,
               byrow=T)
dfm3 <- TO_scipy_sparse(sparse)
Error in TO_scipy_sparse(sparse) : attempt to apply non-function

To answer a question about whether sparse is a dgCMatrix:

str(sparse)
Formal class 'dgCMatrix' [package "Matrix"] with 6 slots
  ..@ i       : int [1:2414] 0 6 9 10 13 20 22 23 25 49 ...
  ..@ p       : int [1:51] 0 45 92 146 185 227 277 330 383 435 ...
  ..@ Dim     : int [1:2] 200 50
  ..@ Dimnames:List of 2
  .. ..$ : NULL
  .. ..$ : NULL
  ..@ x       : num [1:2414] 4 1 1 2 5 3 2 5 3 5 ...
  ..@ factors : list()