Comments (5)
I suggest you contact your support team. You are using a custom built pytorch, which we can answer very limited questions about.
from vllm.
Thanks, will do. But does anyone have rough idea what might have caused this?
from vllm.
Same issue here
Appears right after fresh Jetson AGX flashing
here some sw info
jetson_release
Model: NVIDIA Jetson AGX Orin Developer Kit - Jetpack 6.0 [L4T 36.3.0]
NV Power Mode[0]: MAXN
Hardware:
- Module: NVIDIA Jetson AGX Orin
Platform:
- Distribution: Ubuntu 22.04 Jammy Jellyfish
- Release: 5.15.136-tegra
jtop:
- Version: 4.2.8
- Service: Active
Libraries:
- CUDA: 12.2.140
- cuDNN: 8.9.4.25
- TensorRT: 8.6.2.3
- VPI: 3.1.5
- Vulkan: 1.3.204
- OpenCV: 4.10.0-dev - with CUDA: YES
Python 3.10.12 [GCC 11.4.0] on linux
>>> import torch
>>> print(torch.version.cuda)
12.2
from vllm.
Same problem here. Using print(torch.__version__)
, I found that current vllm tries to use PyTorch version 2.3.0 in setup.py . However, my currently installed version is 2.1.0a0+41361538.nv23.06, which is the latest supported by my Jetpack 5.1.2. According to the NVIDIA documentation, PyTorch 2.3.0 is not supported on my setup.
To work around this, I attempted to install vllm-0.2.4, which requires PyTorch only 2.1.0. However, I encountered an OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
error, even though I have already added the correct CUDA_HOME to my .bashrc
from vllm.
怎么解决
from vllm.
Related Issues (20)
- [Performance]: vllm 0.5.4 with enable_chunked_prefill =True, throughput is slightly lower than 0.5.3~0.5.0. HOT 6
- [Bug]: Gemma 2 9b errors HOT 5
- [Bug]: Unusual Memory Usage on H100 with Meta llama 8-B 72 GB it should not be around 8x2x1.2 in bfloat16
- [Usage]: Seeing perf regression using chunked_prefill on VLLM 0.5.4 HOT 2
- [Feature]: Enable Prefix caching kernel on Pallas for TPU backend
- [Bug]: ModuleNotFoundError: No module named 'openai.types' HOT 3
- [Installation]: container images - too big and need to publish also cpu versions HOT 3
- [Bug]: CUDA error: an illegal memory access was encountered when running autofp8 HOT 1
- [Performance]: Block manager v2 has low throughput with prefix caching warmup HOT 3
- [Bug]: vllm server 部署base和lora模型后,请求lora模型失败 HOT 3
- [Doc]: Has the offline chat inference function been updated? HOT 1
- [Bug]: AttributeError: Model BitsAndBytesModelLoader does not support BitsAndBytes quantization yet HOT 1
- [Bug]: The error is caused by: RuntimeError: out must have shape (total_q, num_heads, head_size_og), leading to the following error: vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop has errored already. HOT 1
- [Bug]: Speculative sampling does not excatly maintain the distribution HOT 3
- [Bug]: OpenGVLab/InternVL-Chat-V1-5 never stops properly HOT 8
- [Bug]: assert num_new_tokens > 0 crashes entire worker instead of just failing single API call HOT 1
- [Feature]: Exit on failures HOT 2
- [Misc]: TTFT profiling with respect to prompt length HOT 3
- [Usage]: The results of the model depend on the number of GPUs. HOT 2
- [Feature]: support chunked_prefill for llava
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vllm.