Model weights on VRAM and Embeddings on RAM about h2ogpt HOT 1 CLOSED

Use --pre_load_embedding_model=True --embedding_gpu_id=cpu --cut_distance=10000 --hf_embedding_model=BAAI/bge-base-en-v1.5 --score_model=None on new main.

The first 2 are required, the other 2 are optional but recommended so the model is smaller. That's because it takes about 0.3seconds per chunk on my i9 using instructor-large. That's why you probably want to use a smaller bge model of much smaller size like above. E.g. 90 seconds for 270 chunks. But with bge base above it only takes 20 seconds, so about 4x faster.

See https://huggingface.co/spaces/mteb/leaderboard for other options for smaller size that are still quite accurate, where smaller should be faster on CPU.

Also review the low memory documentation in FAQ for other low memory options.

from h2ogpt.