[Bug]: Can't use offline inference embedding about vllm HOT 6 CLOSED

Embedding models are going to be supported in v0.4.3

So, install from source to use now

from vllm.

Thank you very much. I will try to install new version.

from vllm.

Hello, I would like to ask why this error:
"---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
in <cell line: 1>()
----> 1 outputs = model.encode(prompts)

10 frames
/content/vllm/vllm/model_executor/sampling_metadata.py in _prepare_seq_groups(seq_group_metadata_list, seq_lens, query_lens, device)
206
207 if seq_group_metadata.is_prompt:
--> 208 if sampling_params.seed is not None:
209 seq_group_metadata.state.generator = torch.Generator(
210 device=device).manual_seed(sampling_params.seed)
AttributeError: 'NoneType' object has no attribute 'seed'"occurs during use?

The model I'm using is llama3-8b, thanks!

from vllm.

Hello, I would like to ask why this error: "--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) in <cell line: 1>() ----> 1 outputs = model.encode(prompts)

10 frames /content/vllm/vllm/model_executor/sampling_metadata.py in _prepare_seq_groups(seq_group_metadata_list, seq_lens, query_lens, device) 206 207 if seq_group_metadata.is_prompt: --> 208 if sampling_params.seed is not None: 209 seq_group_metadata.state.generator = torch.Generator( 210 device=device).manual_seed(sampling_params.seed) AttributeError: 'NoneType' object has no attribute 'seed'"occurs during use?

The model I'm using is llama3-8b, thanks!

If you're running the generic Llama-3-8b model, you are running with a generation model rather than an embedding model. I should update the encode API to fail more gracefully if called in this configuration.

If you want to use an embedding model, try:

• https://huggingface.co/intfloat/e5-mistral-7b-instruct

from vllm.

Hello, I would like to ask why this error: "--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) in <cell line: 1>() ----> 1 outputs = model.encode(prompts)
10 frames /content/vllm/vllm/model_executor/sampling_metadata.py in _prepare_seq_groups(seq_group_metadata_list, seq_lens, query_lens, device) 206 207 if seq_group_metadata.is_prompt: --> 208 if sampling_params.seed is not None: 209 seq_group_metadata.state.generator = torch.Generator( 210 device=device).manual_seed(sampling_params.seed) AttributeError: 'NoneType' object has no attribute 'seed'"occurs during use?
The model I'm using is llama3-8b, thanks!

If you're running the generic Llama-3-8b model, you are running with a generation model rather than an embedding model. I should update the encode API to fail more gracefully if called in this configuration.

If you want to use an embedding model, try:

• https://huggingface.co/intfloat/e5-mistral-7b-instruct

Thank you very much for your quick reply! May I ask if there is an API for vllm if I want to extract the hidden state of the generation model?

from vllm.

Hello, I would like to ask why this error: "--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) in <cell line: 1>() ----> 1 outputs = model.encode(prompts)
10 frames /content/vllm/vllm/model_executor/sampling_metadata.py in _prepare_seq_groups(seq_group_metadata_list, seq_lens, query_lens, device) 206 207 if seq_group_metadata.is_prompt: --> 208 if sampling_params.seed is not None: 209 seq_group_metadata.state.generator = torch.Generator( 210 device=device).manual_seed(sampling_params.seed) AttributeError: 'NoneType' object has no attribute 'seed'"occurs during use?
The model I'm using is llama3-8b, thanks!

If you're running the generic Llama-3-8b model, you are running with a generation model rather than an embedding model. I should update the encode API to fail more gracefully if called in this configuration.
If you want to use an embedding model, try:

• https://huggingface.co/intfloat/e5-mistral-7b-instruct

Thank you very much for your quick reply! May I ask if there is an API for vllm if I want to extract the hidden state of the generation model?

Getting the hidden state of the generation model is exactly what I am struggling with. Have you found any good solutions? All the tools I found are targeted for embedding model, not supporting llama3.

from vllm.