[Request] Allow for using OpenAI or other 3P to generate embeddings about khoj HOT 2 OPEN

@rvjosh I've just worked on adding in support to use 3P embeddings API hosted on huggingface (if you configure it to be run as a Sentence Embedding task). See PRs #609 , #616 . This should be fairly extensible with using the OpenAI service directly (eventually), but it would also work for any model via HuggingFace now. Will be available in the next release.

from khoj.

Are you on a machine with GPU? It's pretty fast when running on GPU enabled machines.

• You can choose any sentence transformer compatible model for indexing
• Using the standard sentence transformer models is generally faster for search as they're smaller in size and of decently good quality. Ada's larger embedding size slows down the speed of search
• For large datasets, the first run will be slower but subsequent runs should be fine as only the updated embeddings are regenerated.

We haven't seen a strong enough need to support generating embedding via API yet. But open to considering this if there's a strong enough reason to do so in the future

from khoj.