Need to get support to NPU on Meteor Lake laptop about openvino_notebooks HOT 12 CLOSED

How to run LLM with NPU device in Windows11 ？

from openvino_notebooks.

Hi @olegmikul, in addition to the drivers installed, have you ran inference on the NPU plugin with OpenVINO Runtime outside of the OpenVINO Notebooks? If not, you may need to install the NPU plugin which you can access through the OpenVINO Archives. You can find the install instructions from Archive File here.

After this you should be able to use the NPU device with the notebooks where available, for details/limitations on NPU plugin please refer to NPU plugin documentation here. Hope this helps, let us know if you have any questions or face any issues.

from openvino_notebooks.

Hi, @avitial,

Thank you for your prompt response.

Yes, I ran simple tasks (e.g. C++ samples, hello_query_device) outside the OpenVino Notebooks, and got the same results - CPU and GNA only.

I have install OpenVino Archives from the exact same instructions you have mentioned. I have checked installed runtime OpenVino libs - they have npu and gpu plugins: in /opt/intel/openvino_2023/runtime/lib/intel64:
libopenvino_intel_gpu_plugin.so and
libopenvino_intel_npu_plugin.so

But OpenVino cannot recognize neither GPU nor NPU device.

From NPU plugin documentation one item is really scary:
Supported Platforms:
Host: Intel® Core™ Ultra (former Meteor Lake)
NPU device: NPU 3720
OS: Ubuntu 20*, MS Windows* 11 (both 64-bit)

Does it mean NPU can not be recognized on even Ubuntu 22.0.3 ? Only on Ubuntu 20? New laptop features (wifi 7, meteor lake sound) requires new OS/kernels. I have completed CPU benchmarking and results heavily impacted by OS and kernel versions, e.g. Ubuntu 22 with kernel 6.2 is about 3x less performing for single thread vs Ubuntu 22 with kernel 6.6. Same HW with Arch Linux OS provides even better scores. You can check results I have uploaded to Geekbenche6 for "Micro-Star International Co., Ltd. Prestige 16 AI Evo B1MG" search prompt.

Using old OS/kernels for new laptops is kind of mutually inconsistent.

Can you, please, help me dig more? May be somebody from NPU driver/plugin development team can help? If only Ubuntu 20 supports that, I can install that instead of Ubuntu 22, just I need to know that for sure.

Thank you,
Oleg

from openvino_notebooks.

@olegmikul very valid questions! We are checking with the development team for further clarification on this information, we will get back to you promptly.

Ref. 128821

from openvino_notebooks.

@avitial , thanks!

Couple updates from my side:

On my archlinux distro (same HW, kernel 6.6.9, may have some back-ported features from 6.7 pre-release) I can see GPU (Intel Arc) but not NPU (supported on Debian/Ubuntu only). I can use GPU for OpenCL offload with some good results. Also it can see Wifi7 card, while not stable. But no Meteor Lake Sound, no NPU. I wish NPU driver can work outside of Debian based OS, just a wish.

On Ubuntu 23.10 with 6.7 pre-release kernel I can see Meteor Lake sound, but no GPU for OpenVino/OpenCL, no NPU (I see vpu by dmesg, but not by OpenVino), no wifi7 card.

With lower kernel versions, obviously, we have much less enabled.

I guess with release of 6.7 kernel we should have much better Linux support for Meteor Lake. Not sure if OpenVino and NPU plugin will follow new kernel release cycle, I afraid we need to wait for Ubuntu 24.04 LTS update. Please check with other teams.

Thank you,
Oleg

from openvino_notebooks.

Hi @olegmikul we got confirmation from the development team that Ubuntu 22.04 LTS is supported by NPU plugin, so it should work. We have been able to get the NPU functional with Ubuntu 22.04 LTS and OpenVINO 2023.3 LTS (see env/config details below). There was a step missing to add read/write access to group, which was causing inference to fail due to device access permissions. This was addressed in the documentation with the following PR intel/linux-npu-driver#19.

OpenVINO: 2023.3 LTS (from Archives)
CPU: Intel(R) Core(TM) Ultra 7 155H
OS: Ubuntu 22.04.3 LTS
Kernel: 6.6.13-060613-generic
NPU driver: v1.1.0, commit 853fb4ac15a2e1b58be963328479dfed3ecf6e24 (built as described in https://github.com/intel/linux-npu-driver).

If the issue persists on your side, please share any error messages or detailed logs of the issue you see when trying to inference with NPU device.

from openvino_notebooks.

How to run LLM with NPU device in Windows11 ？

May I ask which specific LLM network are you interested in using with NPU?

from openvino_notebooks.

如何在 Windows11 中使用 NPU 设备运行 LLM？

请问您有兴趣将哪个特定的 LLM 网络与 NPU 一起使用？

LLAMA

from openvino_notebooks.

@etchosts no LLMs are currently supported on NPU.

from openvino_notebooks.

On windows 11, a similar issue is seen for example, when trying to run the 108-gpu-device.ipynb notebook, the NPU doesn't appear when calling core.available_devices.
Tried with latest NPU driver x.2016, tried with suggested nightly openvino build also.

from openvino_notebooks.

@ep150de the NPU plugin is currently available only with the Archive distribution of OpenVINO™, have you installed OpenVINO from the Archive files? You need to source the environment variables (i.e. source /opt/intel/openvino_2023.3.0/setupvars.sh for Linux or "C:\Program Files (x86)\Intel\openvino_2023.3.0\setupvars.bat" on Windows ) prior to starting/executing the notebook. I have driver x.x.x.1688 version and its working, so for latest x.x.x.2016 version I'd expect to work.

108 Notebook queried devices on Linux:

108 Notebook queried devices on Windows:

from openvino_notebooks.

Closing this, I hope previous responses were sufficient to help you proceed. Feel free to reopen and provide additional information if the issue persists, or to ask any questions related to this topic.

from openvino_notebooks.