[Question]: embedding model support multi-gpu about llama_index HOT 2 OPEN

Hello, @RuiqingGuo! I'm here to help you with any bugs, questions, or contributions you may have. Let's tackle this issue together!

Yes, there is support for embedding model parallelism across multiple GPUs when using the semantic splitter from LlamaIndex for document segmentation. The provided code includes configurations for tensor parallelism (tp_size) and pipeline parallelism (pp_size), which are used to distribute the model across multiple GPUs.

tp_size = config["builder_config"]["tensor_parallel"]
pp_size = config["builder_config"]["pipeline_parallel"]
world_size = tp_size * pp_size
assert (
    world_size == tensorrt_llm.mpi_world_size()
), f"Engine world size ({world_size}) != Runtime world size ({tensorrt_llm.mpi_world_size()})"

However, it is noted that the Python runtime does not support pipeline parallelism:

assert (
    pp_size == 1
), "Python runtime does not support pipeline parallelism"

This means that while tensor parallelism is supported, pipeline parallelism is not supported in the Python runtime ^[1].

Additionally, the HuggingFaceEmbedding model does not include explicit handling for multi-GPU setups. The _device attribute is set to a single device, and the model is moved to this single device:

self._device = device or infer_torch_device()
self._model = model.to(self._device)

This indicates that the current implementation does not support distributing the model across multiple GPUs ^[2].

For better multi-GPU support, consider using alternative embedding models such as OptimumEmbedding, OpenVINOEmbedding, TogetherEmbedding, IpexLLMEmbedding, OctoAIEmbedding, VoyageEmbedding, or GradientEmbedding ^[3][4].

To continue talking to Dosu, mention @dosu.

from llama_index.

i meet the same problem with a large pdf(260 pages).

when I use SimpleDirectoryReader, it returns documents for every page. And then SemanticSplitterNodeParser can embed the text. However, when I use my own OCR and put all the content in one string, the SemanticSplitterNodeParser get OOM

from llama_index.