Comments (8)
What is the performance impact ?
from ctranslate2.
The goal would be to maintain the same level of performance.
I'm not sure how the TopK from TensorRT compares to other implementation, such as the one from PyTorch. I used it because it was the easiest to integrate.
from ctranslate2.
Possible reference implementations:
- PyTorch: https://github.com/pytorch/pytorch/blob/4ced22c5de4d5500aac45a1ecb0e7751e1e4a092/aten/src/THC/THCTensorTopK.cuh#L26
- ONNX runtime: https://github.com/microsoft/onnxruntime/blob/86eaa71ec69fe8961dd58a364052d61831dae5ed/onnxruntime/core/providers/cuda/math/topk_impl.cu
from ctranslate2.
Looks like the 2 candidates mentioned above are significantly slower than TensorRT's TopK. We might not be able to easily get rid of TensorRT.
from ctranslate2.
do you have numbers? good to know though.
from ctranslate2.
Yes.
Here are some execution times of the TopK layer with K = 4 and an input with shape [32, 4 * 32000] (a typical input shape during batched beam search):
TopK implementation | Execution time (ms) |
---|---|
TensorRT | 0.233461 |
ONNX Runtime | 1.55496 |
PyTorch | 2.78814 |
Here are some tokens/s when running a CTranslate2 translation with batch = 32 and beam = 4:
TopK implementation | Translation throughput (tokens/s) |
---|---|
TensorRT | 1969.13 |
ONNX Runtime | 1848.19 |
PyTorch | 1790.37 |
Given the poor result of PyTorch's TopK, we could think I did something wrong during the integration. That's possible. However, there are some open issues on the PyTorch repository regarding the poor TopK performance.
from ctranslate2.
I don't get how we can have an order of magnitude in the first table vs a few % differences in the second.
from ctranslate2.
The TopK layer only accounts for a few % of the total runtime. That's why the impact appears smaller when running a full translation.
from ctranslate2.
Related Issues (20)
- Gemma model - help needed HOT 4
- Download ready to use model HOT 3
- [feature request] Mixed quantizations. HOT 1
- Segmentation fault when using `return_alternatives` flag on NLLB model on cuda HOT 2
- [Profiling] Help needed with Profiling CT2 on both CUDA and CPU HOT 1
- Add gemma2 support
- [Feature Request] Expose Profiler to Python HOT 1
- Falcon-11B support
- Other differences in the beam search implementation? HOT 5
- Introduce better format for whisper models HOT 1
- The version of CUB in your include path is not compatible with this release of Thrust. HOT 1
- Trouble converting Mistral-Nemo despite architecture indicating MistralForCausalLM HOT 6
- Converting to Llama architecture to expand models supported by Ctranslate2
- Llama 3.1 support please? HOT 2
- Flash Attention regurgitates repeated tokens - seq2seq HOT 1
- CI failing in the several recent PRs HOT 2
- Docker images not published HOT 2
- Support for ARM64 on Windows HOT 3
- set_random_seed does not make temperature based decoding deterministic HOT 2
- When I run online for a long time, the gpu memory will get bigger and bigger
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ctranslate2.