Comments (6)
Also related: #5404
(Note: I'm not involved with this part of vLLM so I can't help much apart from triaging the issues)
from vllm.
@gjgjos Do you see the same behaviour if you use float32
out-of-interest?
from vllm.
also: how different are the responses you are seeing? are they total garbage? or do they start off as expected and then diverge to something reasonable but different to the single request case.
from vllm.
This phenomenon is caused by the different operations corresponding to float16 and float32.
from vllm.
also: how different are the responses you are seeing? are they total garbage? or do they start off as expected and then diverge to something reasonable but different to the single request case.
The responses are very similar. Only one or two words are different. This issue is only for float 16. Also, not only vllm but also huggingface or fastertransformer makes this issue.
What is exact reason and is there any solution for FP16?
from vllm.
makes
Hi! I tried fp32
when generating MATH500 results. And I found about 10/500 pieces are different during dual testing. So it seems that fp32
cannot fix this bug.
from vllm.
Related Issues (20)
- [Usage]: openai.APIStatusError: Error code: 405 - {'detail': 'Method Not Allowed'} HOT 2
- [Bug]: Gemma-2-2b-it load model hangs by vLLM==0.5.1 on Tesla T4 GPU HOT 5
- [Feature]: Use .env file to deploy vllm HOT 3
- [Usage]: Got nccl error when deploy vllm in k8s with multiple GPUs HOT 5
- [Feature]: ROCm 6.2 support & FP8 Support
- [Bug]: BlockManagerV2 allocate with Sliding Window Attention HOT 2
- [Bug]: FP8 Quantization support for AMD GPUs HOT 1
- [Bug]: a bug in CUDA capabilities test with different GPUs available HOT 11
- [Bug]: Model serving failed with these arguments --tensor-parallel-size 2 --pipeline-parallel-size 2 HOT 1
- [Bug]: Support Falcon Mamba HOT 3
- [Bug]: aqlm test failing on H100
- [RFC]: Add a FastChat like routing server to provide a central endpoint for multiple models HOT 6
- Release v0.5.5 HOT 10
- [TPU] Make sure worker index aligns with node boundary
- [Doc]: massive amount of docs are missing - just empty pages HOT 1
- [Feature]: need to be able to load tiny models with vllm for testing - PagedAttention forces large models HOT 2
- [Bug]: Unable to generate output when add_eos_token = True for Mistral 7b instruct v0.1
- [Bug]: Unable to generate text tokens when add_eos_token = true for Mistral 7B instruct v0.1 HOT 1
- [Bug]: DeepSeek-Coder-V2-Instruct-AWQ assert self.quant_method is not None HOT 5
- [Usage]: Is there an option to reduce GPU memory usage? HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vllm.