Comments (2)
Observed similar results on my experiments. It seems like TPOT is calculated with the final "[Done]" latency included, whereas ITL does not include the final latency, as shown here. Would like some more explanation on the difference between these metrics.
from vllm.
Can confirm on my exps, especially for marlin24 model, ITL is much lower than TPOT, while TPOT for marlin24 is much higher than norm GPTQ model with marlin kernel.
from vllm.
Related Issues (20)
- [Bug]: Triton assert/crash in tests/samplers/test_logprobs.py
- [Usage]: How can I use InfiniBand with 2 hosts through vllm docker? Execution failed.
- [Bug]: Does not work on MacOS HOT 2
- [Usage]: GPTQ quantization behavior HOT 2
- [Feature]: CI - Split up "Models Test" and "Vision Language Models Test"
- [Bug]: base_model.model.score.weight is unsupported LoRA weight HOT 1
- [Feature]: sliding window attention for odd layers
- [Feature][Encoder-Decoder]: Add support for cuda graph during decoding in encoder-decoder models
- [Feature]: Integrate `flash-infer` FP8 KV Cache Chunked-Prefill (Append Attention) HOT 3
- [Bug]: (VllmWorkerProcess pid=3253) WARNING 08-13 11:31:37 shm_broadcast.py:386] No available block found in 60 second HOT 1
- [Misc]: I want to run Llama 3.1 405B using speculative. Can you give me a guide? HOT 1
- [Usage]: release notes,best practice, abort-actively HOT 1
- [Feature]: early-stop in mooncake
- [Usage]: openai.APIStatusError: Error code: 405 - {'detail': 'Method Not Allowed'} HOT 2
- [Bug]: Gemma-2-2b-it load model hangs by vLLM==0.5.1 on Tesla T4 GPU HOT 5
- [Feature]: Use .env file to deploy vllm HOT 3
- [Usage]: Got nccl error when deploy vllm in k8s with multiple GPUs HOT 5
- [Feature]: ROCm 6.2 support & FP8 Support
- [Bug]: BlockManagerV2 allocate with Sliding Window Attention HOT 2
- [Bug]: FP8 Quantization support for AMD GPUs HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vllm.