Comments (2)
It is not able to run cuda graph on variable input and output shape. Cuda graph requires that inputs are fixed buffers.
However, you can try capture multiple cuda graphs on different input shapes using an run option called gpu_graph_id
.
See https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#using-cuda-graphs-preview
from onnxruntime.
Thank you for your kindly support.
As you say, if you understand how CUDA Graph works, it is clear that it does not support variable tensors.
That said, it would be great for everyone if this was clearly stated in the documentation.
from onnxruntime.
Related Issues (20)
- [Web] Using ceil() in shape computation is not yet supported for MaxPool HOT 3
- Stateful/Memory models HOT 1
- Incompatible libs between libonnxruntime_providers_cuda.so version 1.18 and CUDA 12.4 HOT 2
- [Performance] Is my script set to get optimal performance of onnxruntime?
- [Performance] How to used pinned memory in onnxruntime.
- Mac m1 build android.The compiler doesn't support BFLOAT16!!! HOT 2
- Graph optimization results in broken model HOT 6
- [Build] Unable to build ONNX Runtime against CUDA 12.5 HOT 7
- How to run test file HOT 2
- [JAVA] Ability to construct a Tensor from a GPU memory pointer HOT 4
- Request for Hidden States Access in Phi-3 with ONNX Runtime HOT 3
- [Web] LinkError when using custom built WASM artifacts HOT 1
- [Build] Unable to build onnxruntime from source (with oneDNN EP) HOT 1
- How QLinearConv layer absorb the Relu function HOT 4
- [Web] Latest version does nonstandard imports HOT 6
- [Feature Request] Extend quantization tool to support blocked quantization
- [Performance] Severe performance penalty with transformer model and DirectML HOT 4
- After adding preprocessing steps to the model, on inference an error throws: Non-zero status code returned while running Add node. HOT 2
- [Build] moduleNotfoundError: no module named 'onnxruntime.training' & 'No matching distribution found for onnxruntime-training' HOT 7
- Mismatch in results for TensorRT session and cuda Session HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from onnxruntime.