Comments (5)
Same here on Windows, versions 1.16.0 to 1.17.3 work fine over multiple threads, however 1.18.0 gives Windows fatal exception: access violation
with the following stack trace produced by my own Windows SEH handler:
-----------
Caught unhandled exception...
-----------
Terminating from thread id 10152
Non-C++ exception:
Error: EXCEPTION_ACCESS_VIOLATION
Type: Read
Addr: 0x0
Trace:
40: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
39: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
38: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
37: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
36: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
35: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
34: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
33: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
32: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
31: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
30: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
29: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
28: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
27: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
26: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
25: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
24: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
23: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
22: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
21: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
20: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
19: ?: PyInit_onnxruntime_pybind11_state (onnxruntime_pybind11_state.pyd)
18: ?: pybind11::error_already_set::discard_as_unraisable (onnxruntime_pybind11_state.pyd)
17: ?: PyObject_MakeTpCall (python311.dll)
16: ?: PyObject_Vectorcall (python311.dll)
15: ?: PyEval_EvalFrameDefault (python311.dll)
14: ?: PyFunction_Vectorcall (python311.dll)
13: ?: PyFunction_Vectorcall (python311.dll)
12: ?: PyObject_CallObject (python311.dll)
11: ?: PyEval_EvalFrameDefault (python311.dll)
10: ?: PyFunction_Vectorcall (python311.dll)
9: ?: PyObject_CallObject (python311.dll)
8: ?: PyEval_EvalFrameDefault (python311.dll)
7: ?: PyFunction_Vectorcall (python311.dll)
6: ?: PyFunction_Vectorcall (python311.dll)
5: ?: PyObject_Call (python311.dll)
4: ?: PyInterpreterState_Delete (python311.dll)
3: ?: PyInterpreterState_Delete (python311.dll)
2: ?: recalloc (ucrtbase.dll)
1: ?: BaseThreadInitThunk (KERNEL32.DLL)
0: ?: RtlUserThreadStart (ntdll.dll)
from onnxruntime.
@PatriceVignola @smk2007 @fdwr
Hi, sorry to bother, there are some news for this problem? Actually testing 1.18.1 and the problem is still present :(
Thank you
from onnxruntime.
Tagging @PatriceVignola @smk2007 @fdwr for visibility.
from onnxruntime.
We’ve noted the issue with GPU resource contention due to multiple threads. This usage pattern is not recommended as it makes multiple threads request all of the GPU resources, and can cause contention. Also, the allocator in python API (both CUDA and DML) is explicitly not thread safe because it initializes the allocator as a global singleton due it living outside of the session.
We’re investigating the recent failure and will address it. Meanwhile, please avoid this pattern to prevent GPU contention.
from onnxruntime.
Hi @liuyunms
Sorry to bother, I'm currently using an InferenceSession per tread, but you say it shouldn't be used this way.
4 threds -> 4 inference session with same gpu
Do you mean to use the same InferenceSession in multiple threads? Is it possible?
4 threds -> 1 inference session with same gpu
from onnxruntime.
Related Issues (20)
- Missing onnxruntime_perf_test.exe in Release Assets (or what actually is "Build Drop"?) HOT 2
- [Build]: cmake', '--build', '/temp/liz/onnxruntime/build/Linux/RelWithDebInfo', '--config', 'RelWithDebInfo', '--', '-j64'] HOT 1
- [Feature Request] Request grid_sample 5D support 🌟 HOT 1
- [Build][Bug] The compiler doesn't support BFLOAT16!!! HOT 2
- [WebGPU] `Error: [WebGPU] Kernel "[MaxPool] /sincnet/pool1d.0/MaxPool" failed. Error: length of specified kernel shapes should be 2 less than length of input dimensions` HOT 2
- Error Instantiating EmbeddingModel with ONNX Model intfloat/multilingual-e5-large HOT 1
- [Documentation] Community blog post contribution HOT 1
- [ARM][CPU] Unit test and onnx_runtime_perf test gives cpuinfo error for new Windows ARM chips HOT 2
- [Feature Request] Mark as negative tests for minimal CUDA build
- New restricted asymmetric quantization mode in QDQ mode with zero_point restricted to either 128 or 0
- Trilu op still not work with INT32 input HOT 3
- [WebNN EP] Support int64 output data type for CoreML backend HOT 1
- [Web] where is the demo of object detection on web HOT 2
- LNK2019: unresolved external symbol OrtGetApiBase HOT 1
- Multi-threaded GPU inferencing failing with whisper-small: Non-zero status code returned while running DecoderMaskedMultiHeadAttention node HOT 4
- TensorRT EP failed to create engine from network. HOT 4
- Custom Op Library does not work for CUDA
- onnxruntime.InferenceSession.run sometimes get stuck, sometimes not HOT 3
- How to do multithreaded infer with onnxruntime HOT 1
- CUDA provider fallback to CPU is not working when CUDA_PATH environment variable exists
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from onnxruntime.