Is this method works with other PQ methods? (OPQ, LOPQ) about pqkmeans HOT 5 CLOSED

Hi @insikk

OPQ is a straightforward extension of PQ, where a rotation matrix is applied for each vector in advance. Implementing OPQ is on my TODO list, but PR is welcome :)

LOPQ is a different technology, and cannot be combined to the current PQk-means.

The SOTA search system consists of (1) coarse quantization and (2) PQ compression for each vector. LOPQ is a strategy to train the coarse-quantizer in the per-cell manner, whereas PQk-means does not include coarse-quantization step (because maintaining such data structure is an additional cost in our setting).

from pqkmeans.

Thank you for your kind explanation. Now I understand what is the difference.

I am really looking forward OPQ implementation.
Because I don't know much about OPQ, I don't think I can send PR on this feature.

Meanwhile, I might able to plug and play with external OPQ library. Can you recommend any C++, C++ + Python binding, or Pure Python implementation of OPQ?

from pqkmeans.

OPQ is an encoding step, so it should be pure Python (pq_encoder.py). Actually I have my own C++ implementation of OPQ. I'll port it to Python (maybe next month)

from pqkmeans.

May I get your C++ implementation of OPQ encoding (just for educational purpose)? This is my email: [email protected]. I'll really appreciate it if you could send me a copy.

from pqkmeans.

Oh I had completely forgotten about this issue 😂 Python implemention of OPQ is available here https://github.com/matsui528/nanopq

from pqkmeans.