Comments (16)
andy-neuma
commented on May 12, 2024
1
@vvolhejn , sorry to see it not working. let me do a bit more investigation on my end.
from deepsparse.
tlrmchlsmth
commented on May 12, 2024
Hi @vvolhejn, thank you for your bug report. Could you share the results of lscpu to help us debug this issue?
from deepsparse.
vvolhejn
commented on May 12, 2024
@tlrmchlsmth Sure!
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 71
Model name: Intel(R) Xeon(R) CPU E3-1284L v4 @ 2.90GHz
Stepping: 1
CPU MHz: 2900.354
CPU max MHz: 3800.0000
CPU min MHz: 800.0000
BogoMIPS: 5799.77
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 6144K
L4 cache: 131072K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt dtherm ida arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
from deepsparse.
andy-neuma
commented on May 12, 2024
hello @vvolhejn , sorry for the inconvenience. Could you check your system files by listing the contents of "/sys/devices/system/cpu", i.e. ls /sys/devices/system/cpu?
from deepsparse.
vvolhejn
commented on May 12, 2024
@andy-neuma here it is:
[vvolhejn@eu-login-18 ~]$ ls /sys/devices/system/cpu
cpu0 cpu3 isolated modalias online present vulnerabilities
cpu1 cpuidle kernel_max nohz_full possible smt
cpu2 intel_pstate microcode offline power uevent
I'm on a different node of the cluster than before, but the CPU model and everything else is the same.
from deepsparse.
andy-neuma
commented on May 12, 2024
thanks @vvolhejn
While we work on our end to resolve the issue, you should be able to run using a specified arch topology.
Please save this arch topology json, topo-four-core-json.txt to your system. Once it is in place please set the environment variable NM_ARCH_FILE to the location you saved it. The environment variable needs to be set before importing or running.
For example,
>> export NM_ARCH_FILE=<location of arch topology json>
Once the environment variable is set you should be able to run. Thanks.
from deepsparse.
vvolhejn
commented on May 12, 2024
@andy-neuma thank you for looking into this. Unfortunately, this did not help. See below - I first demonstrate that the environment variable is set to the file you sent me, and then I try to import deepsparse again:
>>> os.environ["NM_ARCH_FILE"]
'/cluster/home/vvolhejn/topo-four-core-json.txt'
>>> with open(os.environ["NM_ARCH_FILE"], "r") as f:
... f.readlines()
...
['{\n', ' "architecture": "x86_64",\n', ' "hardware_threads": [\n', ' {\n', ' "available": true,\n', ' "caches": [\n', ' {\n', ' "cache_id": 0,\n', ' "coherency_line_size": 64,\n', ' "index": 0,\n', ' "level": 1,\n', ' "size": 32768,\n', ' "type": "data"\n', ' },\n', ' {\n', ' "cache_id": 0,\n', ' "coherency_line_size": 64,\n', ' "index": 1,\n', ' "level": 1,\n', ' "size": 32768,\n', ' "type": "instruction"\n', ' },\n', ' {\n', ' "cache_id": 0,\n', ' "coherency_line_size": 64,\n', ' "index": 2,\n', ' "level": 2,\n', ' "size": 262144,\n', ' "type": "data"\n', ' },\n', ' {\n', ' "cache_id": 0,\n', ' "coherency_line_size": 64,\n', ' "index": 3,\n', ' "level": 3,\n', ' "size": 6291456,\n', ' "type": "data"\n', ' }\n', ' ],\n', ' "core_id": 0,\n', ' "index": 0,\n', ' "l3_index": 0,\n', ' "numa_node": 0,\n', ' "socket_id": 0,\n', ' "thread_id": 0\n', ' },\n', ' {\n', ' "available": true,\n', ' "caches": [\n', ' {\n', ' "cache_id": 1,\n', ' "coherency_line_size": 64,\n', ' "index": 0,\n', ' "level": 1,\n', ' "size": 32768,\n', ' "type": "data"\n', ' },\n', ' {\n', ' "cache_id": 1,\n', ' "coherency_line_size": 64,\n', ' "index": 1,\n', ' "level": 1,\n', ' "size": 32768,\n', ' "type": "instruction"\n', ' },\n', ' {\n', ' "cache_id": 1,\n', ' "coherency_line_size": 64,\n', ' "index": 2,\n', ' "level": 2,\n', ' "size": 262144,\n', ' "type": "data"\n', ' },\n', ' {\n', ' "cache_id": 0,\n', ' "coherency_line_size": 64,\n', ' "index": 3,\n', ' "level": 3,\n', ' "size": 6291456,\n', ' "type": "data"\n', ' }\n', ' ],\n', ' "core_id": 1,\n', ' "index": 1,\n', ' "l3_index": 0,\n', ' "numa_node": 0,\n', ' "socket_id": 0,\n', ' "thread_id": 1\n', ' },\n', ' {\n', ' "available": true,\n', ' "caches": [\n', ' {\n', ' "cache_id": 2,\n', ' "coherency_line_size": 64,\n', ' "index": 0,\n', ' "level": 1,\n', ' "size": 32768,\n', ' "type": "data"\n', ' },\n', ' {\n', ' "cache_id": 2,\n', ' "coherency_line_size": 64,\n', ' "index": 1,\n', ' "level": 1,\n', ' "size": 32768,\n', ' "type": "instruction"\n', ' },\n', ' {\n', ' "cache_id": 2,\n', ' "coherency_line_size": 64,\n', ' "index": 2,\n', ' "level": 2,\n', ' "size": 262144,\n', ' "type": "data"\n', ' },\n', ' {\n', ' "cache_id": 0,\n', ' "coherency_line_size": 64,\n', ' "index": 3,\n', ' "level": 3,\n', ' "size": 6291456,\n', ' "type": "data"\n', ' }\n', ' ],\n', ' "core_id": 2,\n', ' "index": 2,\n', ' "l3_index": 0,\n', ' "numa_node": 0,\n', ' "socket_id": 0,\n', ' "thread_id": 2\n', ' },\n', ' {\n', ' "available": true,\n', ' "caches": [\n', ' {\n', ' "cache_id": 3,\n', ' "coherency_line_size": 64,\n', ' "index": 0,\n', ' "level": 1,\n', ' "size": 32768,\n', ' "type": "data"\n', ' },\n', ' {\n', ' "cache_id": 3,\n', ' "coherency_line_size": 64,\n', ' "index": 1,\n', ' "level": 1,\n', ' "size": 32768,\n', ' "type": "instruction"\n', ' },\n', ' {\n', ' "cache_id": 3,\n', ' "coherency_line_size": 64,\n', ' "index": 2,\n', ' "level": 2,\n', ' "size": 262144,\n', ' "type": "data"\n', ' },\n', ' {\n', ' "cache_id": 0,\n', ' "coherency_line_size": 64,\n', ' "index": 3,\n', ' "level": 3,\n', ' "size": 6291456,\n', ' "type": "data"\n', ' }\n', ' ],\n', ' "core_id": 3,\n', ' "index": 3,\n', ' "l3_index": 0,\n', ' "numa_node": 0,\n', ' "socket_id": 0,\n', ' "thread_id": 3\n', ' }\n', ' ],\n', ' "isa": "avx2",\n', ' "vendor": "GenuineIntel",\n', ' "vendor_id": "Intel",\n', ' "vendor_model": "Intel(R) Core(TM) i9-7980XE CPU @ 2.60GHz",\n', ' "vnni": false\n', '}\n']
>>> import deepsparse
DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.12.1 (18c5ee67) (release) (optimized)
Date: 05-25-2022 @ 13:32:47 CEST
OS: Linux eu-login-19 3.10.0-1160.62.1.el7.x86_64 #1 SMP Tue Apr 5 16:57:59 UTC 2022
Arch: x86_64
CPU:
Vendor:
Cores/sockets/threads: [0, 0, 0]
Available cores/sockets/threads: [0, 0, 0]
L1 cache size data/instruction: 0k/0k
L2 cache size: 0Mb
L3 cache size: 0Mb
Total memory: 47.349G
Free memory: 20.53G
Assertion at src/lib/core/cpu.cpp:273
Backtrace:
0# wand::detail::abort_prefix(std::ostream&, char const*, char const*, int, bool, bool, unsigned long) in /cluster/home/vvolhejn/venv/lib64/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
1# wand::detail::assert_fail(char const*, char const*, int) in /cluster/home/vvolhejn/venv/lib64/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
2# 0x00002B0354304E9B in /cluster/home/vvolhejn/venv/lib64/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
3# 0x00002B035430513C in /cluster/home/vvolhejn/venv/lib64/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
4# 0x00002B03379699C3 in /lib64/ld-linux-x86-64.so.2
5# 0x00002B033796E59E in /lib64/ld-linux-x86-64.so.2
6# 0x00002B03379697D4 in /lib64/ld-linux-x86-64.so.2
7# 0x00002B033796DB8B in /lib64/ld-linux-x86-64.so.2
8# 0x00002B0338549FAB in /lib64/libdl.so.2
9# 0x00002B03379697D4 in /lib64/ld-linux-x86-64.so.2
10# 0x00002B033854A5AD in /lib64/libdl.so.2
11# dlopen in /lib64/libdl.so.2
12# _PyImport_FindSharedFuncptr in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
13# _PyImport_LoadDynamicModuleWithSpec in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
14# 0x00002B0337D21449 in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
15# 0x00002B0337C5DE03 in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
16# PyVectorcall_Call in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
17# _PyEval_EvalFrameDefault in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
18# _PyEval_EvalCodeWithName in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
19# _PyFunction_Vectorcall in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
20# _PyEval_EvalFrameDefault in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
21# 0x00002B0337BE9209 in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
22# _PyEval_EvalFrameDefault in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
23# 0x00002B0337BE9209 in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
Please email a copy of this stack trace and any additional information to: [email protected]
fish: Job 1, 'env NM_ARCH_FILE=/cluster/home/…' terminated by signal SIGABRT (Abort)
(venv) vvolhejn@eu-login-19 ~ [SIGABRT]>
from deepsparse.
andy-neuma
commented on May 12, 2024
@vvolhejn ... thanks for your patience.
there is a patch in our nightly that i'm hoping will unblock you. the bug should be addressed in our next release, but until then would you mind trying the nightly?
To try the nightly, please follow these steps.
- clone deepsparse repo, i.e.
git clone https://github.com/neuralmagic/deepsparse.git - cd into deepsparse,
cd deepsparse - pip install from deepsparse directory, e.g.
pip install -e .
Here is what I see when following these steps.
ubuntu@ip-10-10-50-142:~/lab/deepsparse$ pwd
/home/ubuntu/lab/deepsparse
ubuntu@ip-10-10-50-142:~/lab/deepsparse$ export NM_ARCH_FILE=topo-four-core-json.txt
ubuntu@ip-10-10-50-142:~/lab/deepsparse$ python3
Python 3.6.9 (default, Mar 15 2022, 13:55:28)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import deepsparse
Using json arch topology file: topo-four-core-json.txt
>>> exit()
Please let us know if this unblocks you.
from deepsparse.
vvolhejn
commented on May 12, 2024
@andy-neuma Progress - it's a different error now!
Python 3.8.5 (default, Sep 27 2021, 10:10:37)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import deepsparse
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/cluster/home/vvolhejn/deepsparse/src/deepsparse/__init__.py", line 33, in <module>
from .engine import *
File "/cluster/home/vvolhejn/deepsparse/src/deepsparse/engine.py", line 27, in <module>
from deepsparse.benchmark import BenchmarkResults
File "/cluster/home/vvolhejn/deepsparse/src/deepsparse/benchmark/__init__.py", line 17, in <module>
from .ort_engine import ORTEngine
File "/cluster/home/vvolhejn/deepsparse/src/deepsparse/benchmark/ort_engine.py", line 50, in <module>
ARCH = cpu_architecture()
File "/cluster/home/vvolhejn/deepsparse/src/deepsparse/cpu.py", line 167, in cpu_architecture
arch = _parse_arch_bin()
File "/cluster/home/vvolhejn/deepsparse/src/deepsparse/cpu.py", line 49, in __call__
self.memo[args] = self.f(*args)
File "/cluster/home/vvolhejn/deepsparse/src/deepsparse/cpu.py", line 132, in _parse_arch_bin
error = json.loads(ex.stdout)
File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
>>>This one has a stack trace, which is nice. I tried printing out the error by editing src/deepsparse/cpu.py and inserting print(ex) on line 132. This added the following line to the output:
Command '/cluster/home/vvolhejn/deepsparse/src/deepsparse/arch.bin' died with <Signals.SIGFPE: 8>.
Hope this helps.
from deepsparse.
andy-neuma
commented on May 12, 2024
@vvolhejn - thanks for giving this a try!
there seems to be an issue decoding the json. would you mind double checking that you have set NM_ARCH_FILE to the json file?
from deepsparse.
vvolhejn
commented on May 12, 2024
@andy-neuma I have checked that - I'm fairly confident that's not the issue because the error is coming from the line error = json.loads(ex.stdout) where the program is actually trying to parse an exception that has already occurred. I tried printing this exception ex and got
Command '/cluster/home/vvolhejn/deepsparse/src/deepsparse/arch.bin' died with <Signals.SIGFPE: 8>.
Also, ex.stdout is the empty string and ex.stderr is None if that helps.
from deepsparse.
andy-neuma
commented on May 12, 2024
@vvolhejn - thanks for taking the extra step and yes it helps. I'll take a look on my end and get back to you.
from deepsparse.
NicolasGehring
commented on May 12, 2024
Hello, I'm currently experiencing the same importing issues with deepsparse 12.2. Using the nightly build, exporting NM_ARCH_FILE and using the json provided by @andy-neuma resolved that issues. However the node I want to install deepsparse on has more than 4 cores and multiple CPUs available. Therefore I was looking for resources how to adapt the Arch file to my needs. Also are there any updates on the original import issue?
from deepsparse.
jeanniefinks
commented on May 12, 2024
Hi @NicolasGehring
Circling back here as we know it's been a while: we've updated DeepSparse to several versions now. If you want to try that and see if you're still having import issues, we can see if the newer version has resolved the issues. Thank you so much.
- Jeannie / Neural Magic
from deepsparse.
andy-neuma
commented on May 12, 2024
hello, @NicolasGehring - what version are you on currently? we have done a fair amount of work in this area since 12.2.
thanks, Andy
from deepsparse.
jeanniefinks
commented on May 12, 2024
Hello @NicolasGehring!
As we have not heard an update on this, I will close this issue for now. If are able to retry the import task with the new version of DeepSparse to see if it's still a problem, we definitely want to know. Please re-open this issue.
Regards, Jeannie / Neural Magic
from deepsparse.
Related Issues (20)
- Paper: New algorithm for pruning HOT 3
- Does he support stable diffusion or fooocus, or other image generation acceleration? HOT 1
- Segmentation fault (core dumped) HOT 6
- https://www.therapyinsightspractice.com HOT 1
- Adding Contributors Section to readme.md HOT 1
- Research: 4-bit quantization HOT 5
- Assertion `!cache_sizes.empty()' failed HOT 2
- transformers_embedding-extraction for text-generation tasks HOT 3
- Question on quantization size HOT 1
- NM: error: Node (/model/Add_1) Op (Add) [ShapeInferenceError] Incompatible dimension HOT 5
- Using output_value as "token_embeddings" is broken for Sentence Transformer HOT 2
- docker access denied error HOT 8
- Unsupported ONNX type 10 for FP16 HOT 5
- Assertion at src/lib/core/topology.cpp:627 HOT 1
- yolo-v8 in onnx-runtime outperforms deepsparse on iMX8 HOT 3
- Python3.12? HOT 3
- How to use for the fintuned roberta model for text classification HOT 3
- Purpose of exporter.export_onnx(sample_batch=torch.randn(1, 1, 28, 28)) HOT 4
- deepsparse.TextGeneration doesn't accept `trust_remote_code` as an arg anymore HOT 1
- YOLOv8 - Display bounding boxes and classes names in image using python. HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepsparse.