Support PCA in ANNlite about annlite HOT 8 CLOSED

pca will be integrated as a feature provided by annlite. It provides the function to reduce the embedding dimension so as the memory footprint. At this moment, we make a strong assumption that the training data (for pca) comes from the same distribution of indexed data, and the incremental data distribution would not have a large shift.

from annlite.

How will this be integrated? is it a way to do dimensionality reduction? How will u mantain ANNLite being incremental?

from annlite.

But why is this better than PQ? We have PQ already for "dim reduction" right?

from annlite.

Generally, we want to take advantage of the fact that PCA dimensionality reduction preserves more information in the first components. PCA theoretically can achieve better ANN results with the same compress rate. In practice, pca and pq are usually used together (pca first, and then pq) to achieves good accuracy–space trade-offs. And we also find research work that demonstrates the combination idea works https://cs.uwaterloo.ca/~jimmylin/publications/Ma_etal_EMNLP2021.pdf

from annlite.

Hello,

would the PCA work with multiple shards of ANNLITE ? And how ? Will there will be a different projections for each shard. Would it be ok ?

Thanks

from annlite.

A good practice is to prepare a PCA ahead (offline) which can be shared across shards.

from annlite.

Hi @numb3r3, I briefly check the Projector code and it doesn't seems to me that there is some prepare loading of pretrained PCA. Or am I missing something ?

from annlite.

@jemmyshin do we have such document or examples?

from annlite.