Comments (3)
This problem has troubled me for a long time. I have tried many versions of OpenVINO, including building from source code, installing with pip, and installing with apt, and all of them report the same error.
I would like to know how to use OpenVINO's benchmark_app tool to perform performance tests on current mainstream large models such as qwen, chatglm, etc.
Is it that benchmark_app testing of large models is not yet supported?
Please let me know, thank you very much!
from openvino.
Hi @HPUedCSLearner!
We recommend to use dedicated llm_bench for these purposes. It is the most convinient way to benchmark LLM workloads with OpenVINO runtime.
Benchmark_app is mostly dedicated to profile "traditional" models and might lack some LLM related functionality.
from openvino.
Thank you for your suggestion, I will try your suggestion next.
from openvino.
Related Issues (20)
- [Good First Issue][TF FE]: Support HSVToRGB operation for TensorFlow HOT 3
- [Good First Issue][TF FE]: Support RGBToHSV operation for TensorFlow HOT 7
- Unable to get working OpenVINO/ONNX GPU accelleration on Ubuntu 24.04, Intel i7 1355U HOT 4
- Unable to detect GPU as inference device HOT 6
- [Bug]: Unable to set the number of threads for inference
- [Good First Issue][TF FE]: Support BatchMatrixInverse operation for TensorFlow HOT 5
- Dose OpenVINO support INT8 Matmul?
- [Bug]:using openvino complining to npu usng llama 3 8b instruct int 4 version is not happening HOT 2
- [Bug]: unwanted calling thread's cpu affinity mask change on a NUMA machine HOT 4
- What should I do if openvino2019R1 does not support operators slice, resize, and avgpool during model conversion? HOT 5
- [Bug]: Linking openvino with static library results in error HOT 2
- [Bug]: [get_profiling_info]:Why is the output of my openvino.runtime.ProfilingInfo all Zeros? HOT 4
- [Performance]: how to release compiled model or infer request's memory under python HOT 1
- [Good First Issue]: Add python bindings for TensorDescriptor HOT 15
- Question - Add extension to core in C-API?
- [Good First Issue]: Set/Get Properties in C-API HOT 4
- [Bug]: Device with "gpu" name is not registered in the OpenVINO Runtime HOT 2
- [Bug]: Fail to read YOLOv8 Pytorch model HOT 1
- [Bug]: ONNX model inferencing does not work on Nvidia GPU, only CPU HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openvino.