[Installation]: vllm on NVIDIA jetson AGX orin about vllm HOT 4 OPEN

I suggest you contact your support team. You are using a custom built pytorch, which we can answer very limited questions about.

from vllm.

Thanks, will do. But does anyone have rough idea what might have caused this?

from vllm.

Same issue here
Appears right after fresh Jetson AGX flashing

here some sw info

jetson_release 

Model: NVIDIA Jetson AGX Orin Developer Kit - Jetpack 6.0 [L4T 36.3.0]
NV Power Mode[0]: MAXN
Hardware:
 - Module: NVIDIA Jetson AGX Orin
Platform:
 - Distribution: Ubuntu 22.04 Jammy Jellyfish
 - Release: 5.15.136-tegra
jtop:
 - Version: 4.2.8
 - Service: Active
Libraries:
 - CUDA: 12.2.140
 - cuDNN: 8.9.4.25
 - TensorRT: 8.6.2.3
 - VPI: 3.1.5
 - Vulkan: 1.3.204
 - OpenCV: 4.10.0-dev - with CUDA: YES

Python 3.10.12 [GCC 11.4.0] on linux
>>> import torch
>>> print(torch.version.cuda)
12.2

from vllm.

Same problem here. Using print(torch.__version__), I found that current vllm tries to use PyTorch version 2.3.0 in setup.py . However, my currently installed version is 2.1.0a0+41361538.nv23.06, which is the latest supported by my Jetpack 5.1.2. According to the NVIDIA documentation, PyTorch 2.3.0 is not supported on my setup.

To work around this, I attempted to install vllm-0.2.4, which requires PyTorch only 2.1.0. However, I encountered an OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root. error, even though I have already added the correct CUDA_HOME to my .bashrc

from vllm.