Not able to install locally about text-generation-inference HOT 8 OPEN

have resolved the issues using the following set of install-scripts;
https://github.com/nyunAI/Faster-LLM-Survey/tree/A100TGIv2.0.1/scripts

Usually, if u have required version of cmake, libkineto, protobuff & rust installed you can directly run

1. scripts/install-tgi.sh , then
2. scripts/parallel-install-extensions.sh (this parallely installs all extensions - flash-attn, flash-attn-v2-cuda, vllm-cuda, exllamav2_kernels, etc.)

use other scripts in the directory as required.

for other system and driver details see - https://github.com/nyunAI/Faster-LLM-Survey/blob/A100TGIv2.0.1/experiment_details.txt

ps. maintainer can close this. leaving open for anyone facing a similar issue.

from text-generation-inference.

Build and install rotary and layer_norm from https://github.com/Dao-AILab/flash-attention/tree/main/csrc.
This work for me

from text-generation-inference.

You need to re-install vllm and flash-attention-v2
`cd text-generation-inference/server
rm -rf vllm
make install-vllm-cuda

rm -rf flash-attention-v2
make install-flash-attention-v2-cuda`

They forgot to add this to the release notes about local installs.
#1738
I tried this and solved my problem.

from text-generation-inference.

I have been installing all of the extensions via those commands for 2 days now;
I also tried using the release v2.0.1 code zip
let me try this once more with a clean installation

from text-generation-inference.

I have been installing all of the extensions via those commands for 2 days now; I also tried using the release v2.0.1 code zip let me try this once more with a clean installation

I feel you, did exactly the same. install/delete about 4 times

from text-generation-inference.

I have been installing all of the extensions via those commands for 2 days now; I also tried using the release v2.0.1 code zip let me try this once more with a clean installation

You can follow the steps in the Dockerfile, after compile flash-attn with cmd 'make install-flash..‘, the script moves the compiled file to python's site-package folder, just like
cp -r /text-generation-inference/server/flash-attention-v2/build/lib.linux-x86_64-cpython-39/* /usr/local/lib/python3.10/site-packages/

from text-generation-inference.

have resolved the issues using the following set of install-scripts; https://github.com/nyunAI/Faster-LLM-Survey/tree/A100TGIv2.0.1/scripts

Usually, if u have required version of cmake, libkineto, protobuff & rust installed you can directly run

1. [scripts/install-tgi.sh](https://github.com/nyunAI/Faster-LLM-Survey/blob/A100TGIv2.0.1/scripts/install-tgi.sh) , then

2. [scripts/parallel-install-extensions.sh](https://github.com/nyunAI/Faster-LLM-Survey/blob/A100TGIv2.0.1/scripts/parallel-install-extensions.sh) (this parallely installs all extensions - flash-attn, flash-attn-v2-cuda, vllm-cuda, exllamav2_kernels, etc.)

use other scripts in the directory as required.

for other system and driver details see - https://github.com/nyunAI/Faster-LLM-Survey/blob/A100TGIv2.0.1/experiment_details.txt

ps. maintainer can close this. leaving open for anyone facing a similar issue.

When install vllm for TGI-2.0.1, I came across :

error: triton 2.3.0 is installed but triton==2.1.0 is required by {'torch'}
make: *** [Makefile-vllm:12: install-vllm-cuda] Error 1

Is this because I use wrong vllm version. I don't modify anything in the Makefile-* scriot

from text-generation-inference.

Your PyTorch version might be different. I faced this issue for the same reason that my PyTorch version was higher than torch==2.1.0 and hence the default triton that was installed was 2.2.0 (afair).
Nonetheless, use a fresh virtual env (maybe conda)

install torch==2.1.0 or use install-tgi.sh

from text-generation-inference.