iree-org / iree-samples Goto Github PK

View Code? Open in Web Editor NEW

43.0 43.0 47.0 1.56 MB

License: Apache License 2.0

Python 33.36% Shell 7.47% CMake 3.15% MLIR 41.51% C++ 10.58% C 2.48% Starlark 0.56% Makefile 0.58% Dockerfile 0.32%

iree-samples's People

Contributors

Stargazers

Watchers

iree-samples's Issues

Triage failure cases from when CI was down

c5f57ec

TFLITEHUB :: bird_classifier_test.py
TFLITEHUB :: lightning_i8_test.py
TFLITEHUB :: magenta_test.py
TFLITEHUB :: mobilenet_ssd_quant_test.py
TFLITEHUB :: mobilenet_v2_uint8_test.py
TFLITEHUB :: ssd_mobilenet_v1_uint8_test.py
TFLITEHUB :: visual_wake_words_i8_test.py

[runtime-library] CMake build of runtime-library failed

Ignored this CMake configuration warning, I kept moving on building runtime-library with the command cmake --build build, which printed the following error:

[173/185] Building C object CMakeFiles/ireert_test.dir/test_api.c.o
FAILED: CMakeFiles/ireert_test.dir/test_api.c.o
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -DCPUINFO_SUPPORTED_PLATFORM=1 -I/Users/y/w/iree-samples/runtime-library/build/include -I/Users/y/w/iree/third_party/cpuinfo/include -isystem /Users/y/w/iree -isystem /Users/y/w/iree-samples/runtime-library/build/iree_core -isystem /Users/y/w/iree/third_party/flatcc/include -O3 -DNDEBUG -flto=thin -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -fvisibility=hidden -Werror -Wall -Wno-ambiguous-member-template -Wno-char-subscripts -Wno-extern-c-compat -Wno-gnu-alignof-expression -Wno-gnu-variable-sized-type-not-at-end -Wno-ignored-optimization-argument -Wno-invalid-offsetof -Wno-invalid-source-encoding -Wno-mismatched-tags -Wno-pointer-sign -Wno-reserved-user-defined-literal -Wno-return-type-c-linkage -Wno-self-assign-overloaded -Wno-sign-compare -Wno-signed-unsigned-wchar -Wno-strict-overflow -Wno-trigraphs -Wno-unknown-pragmas -Wno-unknown-warning-option -Wno-unused-command-line-argument -Wno-unused-const-variable -Wno-unused-function -Wno-unused-local-typedef -Wno-unused-private-field -Wno-user-defined-warnings -Wctad-maybe-unsupported -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wfor-loop-analysis -Wformat-security -Wgnu-redeclared-enum -Wimplicit-fallthrough -Winfinite-recursion -Wliteral-conversion -Wnon-virtual-dtor -Woverloaded-virtual -Wpointer-arith -Wself-assign -Wstring-conversion -Wtautological-overlap-compare -Wthread-safety -Wthread-safety-beta -Wunused-comparison -Wvla -fno-lax-vector-conversions -fmacro-prefix-map=/Users/y/w/iree=iree -MD -MT CMakeFiles/ireert_test.dir/test_api.c.o -MF CMakeFiles/ireert_test.dir/test_api.c.o.d -o CMakeFiles/ireert_test.dir/test_api.c.o -c /Users/y/w/iree-samples/runtime-library/test_api.c
In file included from /Users/y/w/iree-samples/runtime-library/test_api.c:10:
In file included from /Users/y/w/iree-samples/runtime-library/build/include/iree/hal/api.h:12:
In file included from /Users/y/w/iree-samples/runtime-library/build/include/iree/hal/allocator.h:13:
In file included from /Users/y/w/iree-samples/runtime-library/build/include/iree/base/api.h:82:
In file included from /Users/y/w/iree-samples/runtime-library/build/include/iree/base/allocator.h:17:
In file included from /Users/y/w/iree-samples/runtime-library/build/include/iree/base/status.h:22:
/Users/y/w/iree-samples/runtime-library/build/include/iree/base/string_view.h:67:27: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
  iree_string_pair_t v = {iree_string_view_empty(), iree_string_view_empty()};
                          ^~~~~~~~~~~~~~~~~~~~~~~~
                          {                       }
/Users/y/w/iree-samples/runtime-library/build/include/iree/base/string_view.h:67:53: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
  iree_string_pair_t v = {iree_string_view_empty(), iree_string_view_empty()};
                                                    ^~~~~~~~~~~~~~~~~~~~~~~~
                                                    {                       }
/Users/y/w/iree-samples/runtime-library/build/include/iree/base/string_view.h:74:27: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
  iree_string_pair_t v = {first, second};
                          ^~~~~
                          {    }
/Users/y/w/iree-samples/runtime-library/build/include/iree/base/string_view.h:74:34: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
  iree_string_pair_t v = {first, second};
                                 ^~~~~~
                                 {     }
4 errors generated.
[183/185] Building C object iree_core/runtime/src/iree/vm/CMakeFiles/iree_vm_bytecode_module.objects.dir/bytecode_disasm.c.o
ninja: build stopped: subcommand failed.

iree.importer not found in failing regression tests

ModuleNotFoundError: No module named 'iree.importer' is causing failures in multiple regression tests
https://github.com/iree-org/iree-samples/runs/7261304178?check_suite_focus=true

mobilenet_ssd_quant_test.py failing in regresssion tests

https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "/opt/hostedtoolcache/Python/3.9.13/x64/bin/python" "/home/runner/work/iree-samples/iree-samples/tflitehub/mobilenet_ssd_quant_test.py"
# command output:
TMP_DIR = /home/runner/work/iree-samples/iree-samples/tflitehub/tmp/mobilenet_ssd_quant_test.py
# command stderr:
2022-06-03 15:18:24.518417: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.13/x64/lib
2022-06-03 15:18:24.526413: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Running tests under Python 3.9.13: /opt/hostedtoolcache/Python/3.9.13/x64/bin/python
2022-06-03 15:18:27.614514: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.13/x64/lib
2022-06-03 15:18:27.614629: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-06-03 15:18:27.614671: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (fv-az58-885): /proc/driver/nvidia/version does not exist
[ RUN      ] MobilenetSsdQuantTest.test_compile_tflite
I0603 15:18:27.842373 140275194853184 test_util.py:119] Setting up for IREE
I0603 15:18:27.843448 140275194853184 binaries.py:218] Invoke IREE Pipeline:
  /opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/site-packages/iree/tools/tflite/iree-import-tflite /home/runner/work/iree-samples/iree-samples/tflitehub/tmp/mobilenet_ssd_quant_test.py/model.tflite --mlir-print-debuginfo --save-temp-tfl-input=/home/runner/work/iree-samples/iree-samples/tflitehub/tmp/mobilenet_ssd_quant_test.py/tflite.mlir --save-temp-iree-input=/home/runner/work/iree-samples/iree-samples/tflitehub/tmp/mobilenet_ssd_quant_test.py/tosa.mlir
  /opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/site-packages/iree/compiler/tools/../_mlir_libs/iree-compile - --iree-input-type=tosa --iree-vm-bytecode-module-output-format=flatbuffer-binary --iree-hal-target-backends=dylib-llvm-aot -o=/home/runner/work/iree-samples/iree-samples/tflitehub/tmp/mobilenet_ssd_quant_test.py/module.bytecode --iree-mlir-to-vm-bytecode-module --iree-llvm-embedded-linker-path=/opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/site-packages/iree/compiler/tools/../_mlir_libs/iree-lld --mlir-print-debuginfo --mlir-print-op-on-diagnostic=false
I0603 15:18:53.740416 140275194853184 test_util.py:75] Setting up tflite interpreter
I0603 15:18:53.744799 140275194853184 test_util.py:82] Setting up iree runtime
I0603 15:18:53.755876 140275194853184 test_util.py:131] Setting up test inputs
I0603 15:18:54.208817 140275194853184 test_util.py:134] Invoking TFLite
I0603 15:18:54.280681 140275194853184 test_util.py:96] Invocation time: 0.0713 seconds
I0603 15:18:54.281131 140275194853184 test_util.py:137] Invoke IREE
I0603 15:18:54.663150 140275194853184 test_util.py:[111](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:112)] Invocation time: 0.3817 seconds
I0603 15:18:54.663800 140275194853184 test_util.py:72] Max error (0): 2.000000
I0603 15:18:54.663979 140275194853184 test_util.py:72] Max error (1): 1.000000
I0603 15:18:54.664087 140275194853184 test_util.py:72] Max error (2): 2.000000
I0603 15:18:54.664179 140275194853184 test_util.py:72] Max error (3): 1.000000
I0603 15:18:54.664267 140275194853184 test_util.py:72] Max error (4): 2.000000
I0603 15:18:54.664351 140275194853184 test_util.py:72] Max error (5): 1.000000
I0603 15:18:54.664455 140275194853184 test_util.py:72] Max error (6): 1.000000
I0603 15:18:54.664557 140275194853184 test_util.py:72] Max error (7): 1.000000
I0603 15:18:54.664650 140275194853184 test_util.py:72] Max error (8): 1.000000
I0603 15:18:54.664749 140275194853184 test_util.py:72] Max error (9): 1.000000
I0603 15:18:54.664831 140275194853184 test_util.py:72] Max error (10): 1.000000
I0603 15:18:54.664909 140275194853184 test_util.py:72] Max error (11): 1.000000
[  FAILED  ] MobilenetSsdQuantTest.test_compile_tflite

Update tests to check for user-facing results rather than just numerical accuracy

Many of our regression tests currently test for numerical accuracy between IREE and TOSA results: https://github.com/google/iree-samples/blob/main/tflitehub/bird_classifier_test.py#L18

This has led to some test failures like in #45, where a TF bump caused a slight but expected change in quantized results due to switching CPU kernels. To avoid failures like these, it could be useful to expand these tests to look at a more user-centric metric rather than just comparing the output activations.

Regression tests failing due to numerical differences

lightning_i8_test.py, magenta_test.py, visual_wake_words_i8_test.py began failing on August 20th:

https://github.com/iree-org/iree-samples/runs/7944684175?check_suite_focus=true

Max error (0): 0.372814
[  FAILED  ] LightningI8Test.test_compile_tflite

Max error (0): 0.238547
[  FAILED  ] MagentaTest.test_compile_tflite

Max error (0): 2.000000
[  FAILED  ] VisualWakeWordsTest.test_compile_tflite

Seg fault causing failures in regression tests

lightning_test.py and lightning_fp16_test.py are failing due to a crash in the IREE runtime

https://github.com/iree-org/iree-samples/actions/runs/3826764148/jobs/6510806357

I0103 04:08:23.993968 140566536087360 test_util.py:119] Setting up for IREE
I0103 04:08:23.994825 140566536087360 binaries.py:214] Invoke IREE Pipeline:
  /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/iree/tools/tflite/iree-import-tflite /home/runner/work/iree-samples/iree-samples/tflitehub/tmp/lightning_test.py/model.tflite --mlir-print-debuginfo --save-temp-tfl-input=/home/runner/work/iree-samples/iree-samples/tflitehub/tmp/lightning_test.py/tflite.mlir --save-temp-iree-input=/home/runner/work/iree-samples/iree-samples/tflitehub/tmp/lightning_test.py/tosa.mlir
  /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/iree/compiler/tools/../_mlir_libs/iree-compile - --iree-input-type=tosa --iree-vm-bytecode-module-output-format=flatbuffer-binary --iree-hal-target-backends=llvm-cpu -o=/home/runner/work/iree-samples/iree-samples/tflitehub/tmp/lightning_test.py/module.bytecode --iree-llvm-embedded-linker-path=/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/iree/compiler/tools/../_mlir_libs/iree-lld --mlir-print-debuginfo --mlir-print-op-on-diagnostic=false
I0103 04:08:38.[122](https://github.com/iree-org/iree-samples/actions/runs/3826764148/jobs/6510806357#step:6:123)436 140566536087360 test_util.py:75] Setting up tflite interpreter
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
I0103 04:08:38.135419 140566536087360 test_util.py:82] Setting up iree runtime
I0103 04:08:38.147252 140566536087360 test_util.py:131] Setting up test inputs
I0103 04:08:38.147829 140566536087360 test_util.py:57] 	[  1 192 192   3], float32
I0103 04:08:38.148043 140566536087360 test_util.py:134] Invoking TFLite
I0103 04:08:38.160333 140566536087360 test_util.py:96] Invocation time: 0.0117 seconds
I0103 04:08:38.160587 140566536087360 test_util.py:137] Invoke IREE
Fatal Python error: Segmentation fault

Thread 0x00007fd832736740 (most recent call first):
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/iree/runtime/function.py", line 154 in _invoke
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/iree/runtime/function.py", line 130 in __call__
  File "/home/runner/work/iree-samples/iree-samples/tflitehub/test_util.py", line 109 in invoke_iree
  File "/home/runner/work/iree-samples/iree-samples/tflitehub/test_util.py", line [138](https://github.com/iree-org/iree-samples/actions/runs/3826764148/jobs/6510806357#step:6:139) in compile_and_execute
  File "/home/runner/work/iree-samples/iree-samples/tflitehub/lightning_test.py", line 16 in test_compile_tflite
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/unittest/case.py", line 550 in _callTestMethod
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/unittest/case.py", line 592 in run
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/unittest/case.py", line 651 in __call__
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/unittest/suite.py", line 122 in run
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/unittest/suite.py", line 84 in __call__
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/unittest/suite.py", line 122 in run
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/unittest/suite.py", line 84 in __call__
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/unittest/runner.py", line 184 in run
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/absl/testing/_pretty_print_reporter.py", line 82 in run
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/unittest/main.py", line 271 in runTests
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/unittest/main.py", line 101 in __init__
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/absl/testing/absltest.py", line 2524 in _run_and_get_tests_result
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/absl/testing/absltest.py", line 2558 in run_tests
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/absl/testing/absltest.py", line 2160 in main_function
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/absl/app.py", line 254 in _run_main
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/absl/app.py", line 308 in run
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/absl/testing/absltest.py", line 2162 in _run_in_app
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/absl/testing/absltest.py", line 2057 in main
  File "/home/runner/work/iree-samples/iree-samples/tflitehub/lightning_test.py", line 19 in <module>

Unknown command line argument '--iree-flow-enable-aggressive-fusion`.

I am trying to build and test and play with the transform-dialect directory, I followed the steps from the dumped commands I get this error, is these some flag changes in iree-compile not updated in the script.sh ?

$ benchmark-transform-run-iree-compile -b cuda /tmp/reduction_2d_static_123x456xf32.mlir /tmp/iree_transform_dialect_aca3d2.mlir
iree-transform-compile /tmp/reduction_2d_static_123x456xf32.mlir -b cuda -c /tmp/iree_transform_dialect_aca3d2.mlir
iree-compile: Unknown command line argument '--iree-flow-enable-aggressive-fusion'.  Try: 'iree-compile --help'
iree-compile: Did you mean '--iree-llvmcpu-enable-pad-consumer-fusion'?

Error creating iree runtime is causing regression test failures

https://github.com/iree-org/iree-samples/runs/7789826451?check_suite_focus=true

[  FAILED  ] AlbertTest.test_compile_tflite
======================================================================
ERROR: test_compile_tflite (__main__.AlbertTest)
AlbertTest.test_compile_tflite
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/iree-samples/iree-samples/tflitehub/albert_test.py", line 30, in test_compile_tflite
    self.compile_and_execute()
  File "/home/runner/work/iree-samples/iree-samples/tflitehub/test_util.py", line 129, in compile_and_execute
    self.setup_iree()
  File "/home/runner/work/iree-samples/iree-samples/tflitehub/test_util.py", line 86, in setup_iree
    vm_module = iree_rt.VmModule.from_flatbuffer(f.read())
TypeError: from_flatbuffer(): incompatible function arguments. The following argument types are supported:
    1. (arg0: iree._runtime.VmInstance, arg1: object) -> iree._runtime.VmModule

Invoked with: b'PK\x03\x04-\x00\x00\x00\x00\x00\x00\x00!\x00

No matching distribution found for iree-tools-tflite with python-3.10 on macOS.

$ source setup_venv.sh 
Setting up venv dir: /Users/benjamin.sergeant/src/foss/iree-samples/iree-samples.venv
Python: /Users/benjamin.sergeant/.pyenv/shims/python
Python version: Python 3.10.9
Requirement already satisfied: pip in ./iree-samples.venv/lib/python3.10/site-packages (22.3.1)
Collecting pip
  Using cached pip-23.0-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.3.1
    Uninstalling pip-22.3.1:
      Successfully uninstalled pip-22.3.1
Successfully installed pip-23.0
Looking in links: https://iree-org.github.io/iree/pip-release-links.html
Collecting iree-compiler
  Downloading https://github.com/iree-org/iree/releases/download/candidate-20230206.422/iree_compiler-20230206.422-cp310-cp310-macosx_11_0_universal2.whl (75.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 75.7/75.7 MB 14.7 MB/s eta 0:00:00
Collecting iree-runtime
  Downloading https://github.com/iree-org/iree/releases/download/candidate-20230206.422/iree_runtime-20230206.422-cp310-cp310-macosx_11_0_universal2.whl (3.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 14.2 MB/s eta 0:00:00
ERROR: Could not find a version that satisfies the requirement iree-tools-tflite (from versions: none)
ERROR: No matching distribution found for iree-tools-tflite
Activate venv with:
  source /Users/benjamin.sergeant/src/foss/iree-samples/iree-samples.venv/bin/activate

multi gpu

[runtime-library] CMake configuration of runtime-library causes warnings about CMP0069

Following steps in the README file of the runtime-library sample, the command cmake -GNinja -Bbuild . prints the following warnings.

Adding cmake_policy(SET CMP0096 NEW) in warned cmake files suppresses these warnings.

I do not know CMP0096 or CMAKE_INTERPROCEDURAL_OPTIMIZATION. Are they necessary for building the runtime library? Should I just ignore these warnings?

[runtime-library] Xcode complains undefined symbols

The landing of #97 enabled the building and running of a simple iOS app in Objective-C that calls iree_status_fprint.

Then, I copy-n-pasted the content of https://github.com/iree-org/iree-samples/blob/main/runtime-library/test_api.c to replace the call to iree_status_fprint. However, Xcode complains undefined symbols in flatcc and cpuinfo.

The command nm -C confirms that these symbols are not defined in all the target frameworks.

$ nm -C ~/w/l/ios-app-call-cmake-xcframework/SampleApp/iree.xcframework/ios-arm64/iree.framework/iree  | grep _cpuinfo_get_clusters_count
              U _cpuinfo_get_clusters_count
$ nm -C ~/w/l/ios-app-call-cmake-xcframework/SampleApp/iree.xcframework/ios-arm64-simulator/iree.framework/iree  | grep _cpuinfo_get_clusters_count
              U _cpuinfo_get_clusters_count
$ nm -C ~/w/l/ios-app-call-cmake-xcframework/SampleApp/iree.xcframework/macos-arm64/iree.framework/iree  | grep _cpuinfo_get_clusters_count
              U _cpuinfo_get_clusters_count

According to https://stackoverflow.com/a/32232609/724872, the letter U means that the symbol is used in the code. If they are defined, we should have got a letter T.

The build of test_api.c into ./build/bin/iree_test succeeded without complaining of undefined symbols. And nm -C reveals that the symbols exist in the executable binary:

$ nm -C build/bin/iree_test | grep _cpuinfo_get_clusters_count
0000000100035998 T _cpuinfo_get_clusters_count

Xcode didn't complain about iree symbols. And nm -C confirms that these iree symbols exist in target frameworks.

$ nm -C ~/w/l/ios-app-call-cmake-xcframework/SampleApp/iree.xcframework/macos-arm64/iree.framework/iree  | grep iree_hal_driver_registry_enumerate
/Users/y/w/l/ios-app-call-cmake-xcframework/SampleApp/iree.xcframework/macos-arm64/iree.framework/iree:tracing.cc.o: no symbols
---------------- T _iree_hal_driver_registry_enumerate

Dumping traces for iree-samples targets is not supported

Coming from openxla/iree#8712

I found that the traces are not generated when running the test target. It's really inconvenient when debugging the issue. I have a commit which dumps inputs myself. We should generalize it to testing module, and dump all the artifacts to a directory.

Example commit: d9ff438

BirdClassifierTest.test_compile_tflite failing in regression tests

https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true

FAIL: TFLITEHUB :: bird_classifier_test.py (3 of 53)
******************** TEST 'TFLITEHUB :: bird_classifier_test.py' FAILED ********************
Script:
--
: 'RUN: at line 1';   /opt/hostedtoolcache/Python/3.9.13/x64/bin/python /home/runner/work/iree-samples/iree-samples/tflitehub/bird_classifier_test.py
--
Exit Code: 1
Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "/opt/hostedtoolcache/Python/3.9.13/x64/bin/python" "/home/runner/work/iree-samples/iree-samples/tflitehub/bird_classifier_test.py"
# command output:
TMP_DIR = /home/runner/work/iree-samples/iree-samples/tflitehub/tmp/bird_classifier_test.py
# command stderr:
2022-06-03 15:07:26.094638: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.[11](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:12).0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.[13](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:14)/x64/lib
2022-06-03 15:07:26.094679: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Running tests under Python 3.9.13: /opt/hostedtoolcache/Python/3.9.13/x64/bin/python
2022-06-03 15:07:28.588283: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.13/x64/lib
2022-06-03 15:07:28.588377: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-06-03 15:07:28.588415: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (fv-az58-885): /proc/driver/nvidia/version does not exist
[ RUN      ] BirdClassifierTest.test_compile_tflite
I0603 15:07:28.784186 [14](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:15)0321084335936 test_util.py:119] Setting up for IREE
I0603 [15](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:16):07:28.785079 140321084335936 binaries.py:2[18](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:19)] Invoke IREE Pipeline:
  /opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/site-packages/iree/tools/tflite/iree-import-tflite /home/runner/work/iree-samples/iree-samples/tflitehub/tmp/bird_classifier_test.py/model.tflite --mlir-print-debuginfo --save-temp-tfl-input=/home/runner/work/iree-samples/iree-samples/tflitehub/tmp/bird_classifier_test.py/tflite.mlir --save-temp-iree-input=/home/runner/work/iree-samples/iree-samples/tflitehub/tmp/bird_classifier_test.py/tosa.mlir
  /opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/site-packages/iree/compiler/tools/../_mlir_libs/iree-compile - --iree-input-type=tosa --iree-vm-bytecode-module-output-format=flatbuffer-binary --iree-hal-target-backends=dylib-llvm-aot -o=/home/runner/work/iree-samples/iree-samples/tflitehub/tmp/bird_classifier_test.py/module.bytecode --iree-mlir-to-vm-bytecode-module --iree-llvm-embedded-linker-path=/opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/site-packages/iree/compiler/tools/../_mlir_libs/iree-lld --mlir-print-debuginfo --mlir-print-op-on-diagnostic=false
I0603 15:07:58.786024 1403[21](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:22)084335936 test_util.py:75] Setting up tflite interpreter
I0603 15:07:58.787774 140321084335936 test_util.py:82] Setting up iree runtime
I0603 15:07:58.795829 140321084335936 test_util.py:131] Setting up test inputs
I0603 15:07:59.056051 140321084335936 test_util.py:134] Invoking TFLite
I0603 15:07:59.084538 140321084335936 test_util.py:96] Invocation time: 0.0280 seconds
I0603 15:07:59.084826 140321084335936 test_util.py:137] Invoke IREE
I0603 15:07:59.[24](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:25)4467 140[32](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:33)1084[33](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:34)5936 test_util.py:111] Invocation time: 0.1594 seconds
I0603 15:07:59.245091 1403210843[35](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:36)9[36](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:37) test_util.py:72] Max error (0): 4.000000
[  FAILED  ] BirdClassifierTest.test_compile_tflite

nasnet_test.py failing in regression tests

https://github.com/iree-org/iree-samples/runs/7422252950?check_suite_focus=true

MnasnetTest.test_compile_tflite
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/iree-samples/iree-samples/tflitehub/nasnet_test.py", line 15, in test_compile_tflite
    self.compile_and_execute()
  File "/home/runner/work/iree-samples/iree-samples/tflitehub/test_util.py", line 120, in compile_and_execute
    iree_tflite_compile.compile_file(
  File "/opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/site-packages/iree/compiler/tools/tflite.py", line 154, in compile_file
    result = invoke_pipeline([import_cl, compile_cl])
  File "/opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/site-packages/iree/compiler/tools/binaries.py", line 259, in invoke_pipeline
    raise CompilerToolError(stage.completed)
iree.compiler.tools.binaries.CompilerToolError: Error invoking IREE compiler tool iree-compile

Error invoking iree-compile is causing regression test failures

https://github.com/iree-org/iree-samples/actions/runs/3826764148/jobs/6510806357

ERROR: test_compile_tflite (__main__.MobilenetV2Uint8Test)
MobilenetV2Uint8Test.test_compile_tflite
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/iree-samples/iree-samples/tflitehub/mobilenet_v2_uint8_test.py", line 27, in test_compile_tflite
    self.compile_and_execute()
  File "/home/runner/work/iree-samples/iree-samples/tflitehub/test_util.py", line 120, in compile_and_execute
    iree_tflite_compile.compile_file(
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/iree/compiler/tools/tflite.py", line 154, in compile_file
    result = invoke_pipeline([import_cl, compile_cl])
  File "/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/iree/compiler/tools/binaries.py", line 259, in invoke_pipeline
    raise CompilerToolError(stage.completed)
iree.compiler.tools.binaries.CompilerToolError: Error invoking IREE compiler tool iree-compile

mobilenet_v1_uint8_test, mobilenet_v2_uint8_test, and mobilenet_v3-large_uint8_test are impacted

How to run MNIST sample in ROCm?

I want to run MNIST sample(https://github.com/iree-org/iree-samples/tree/main/cpp/vision_inference) in ROCm system,but I do not know how to do it ?

Regression tests failing due to numerical differences

https://github.com/iree-org/iree-samples/actions/runs/3826764148/jobs/6510806357

midas_test.py and mirnet_test.py began failing on Jan 2nd:

Max error (0): 1183.290161
[  FAILED  ] MidasTest.test_compile_tflite

Max error (0): 0.053242
[  FAILED  ] MirnetTest.test_compile_tflite

[runtime-library] Cannot use functions in runtime/src/iree/tooling in app development

The Android demo app calls functions like

These functions are defined in the directory

runtime/src/iree/tooling

However, when we build the IREE runtime as a library in https://github.com/iree-org/iree-samples/tree/main/runtime-library, we trace the target iree::runtime::impl, which is

runtime/src/iree/runtime, which is in parallel with
runtime/src/iree/tooling

The result is if we build an app whose source code is out of the directory of the IREE repo and depends on the IREE runtime release built by https://github.com/iree-org/iree-samples/tree/main/runtime-library, when we build the app, the compiler will complain for undefined symbols that exists in

runtime/src/iree/tooling, but not in
runtime/src/iree/runtime

For example:

mobilebert_tf2_quant_test.py failing in regression tests

https://github.com/google/iree-samples/runs/6815385767?check_suite_focus=true

******************** TEST 'TFLITEHUB :: mobilebert_tf2_quant_test.py' FAILED ********************
Script:
--
: 'RUN: at line 1';   /opt/hostedtoolcache/Python/3.9.13/x64/bin/python /home/runner/work/iree-samples/iree-samples/tflitehub/mobilebert_tf2_quant_test.py
--
Exit Code: 1
Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "/opt/hostedtoolcache/Python/3.9.13/x64/bin/python" "/home/runner/work/iree-samples/iree-samples/tflitehub/mobilebert_tf2_quant_test.py"
# command output:
TMP_DIR = /home/runner/work/iree-samples/iree-samples/tflitehub/tmp/mobilebert_tf2_quant_test.py
# command stderr:
2022-06-09 15:22:17.187877: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.13/x64/lib
2022-06-09 15:22:17.187916: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Running tests under Python 3.9.13: /opt/hostedtoolcache/Python/3.9.13/x64/bin/python
2022-06-09 15:22:19.575135: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.13/x64/lib
2022-06-09 15:22:19.575178: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-06-09 15:22:19.575197: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (fv-az90-374): /proc/driver/nvidia/version does not exist
[ RUN      ] MobileBertTest.test_compile_tflite
I0609 15:22:19.918985 140036022593344 test_util.py:119] Setting up for IREE
I0609 15:22:19.919978 140036022593344 binaries.py:218] Invoke IREE Pipeline:
  /opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/site-packages/iree/tools/tflite/iree-import-tflite /home/runner/work/iree-samples/iree-samples/tflitehub/tmp/mobilebert_tf2_quant_test.py/model.tflite --mlir-print-debuginfo --save-temp-tfl-input=/home/runner/work/iree-samples/iree-samples/tflitehub/tmp/mobilebert_tf2_quant_test.py/tflite.mlir --save-temp-iree-input=/home/runner/work/iree-samples/iree-samples/tflitehub/tmp/mobilebert_tf2_quant_test.py/tosa.mlir
  /opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/site-packages/iree/compiler/tools/../_mlir_libs/iree-compile - --iree-input-type=tosa --iree-vm-bytecode-module-output-format=flatbuffer-binary --iree-hal-target-backends=dylib-llvm-aot -o=/home/runner/work/iree-samples/iree-samples/tflitehub/tmp/mobilebert_tf2_quant_test.py/module.bytecode --iree-mlir-to-vm-bytecode-module --iree-llvm-embedded-linker-path=/opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/site-packages/iree/compiler/tools/../_mlir_libs/iree-lld --mlir-print-debuginfo --mlir-print-op-on-diagnostic=false
I0609 15:32:50.786149 140036022593344 test_util.py:75] Setting up tflite interpreter
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
I0609 15:32:51.084219 140036022593344 test_util.py:82] Setting up iree runtime
I0609 15:32:51.247557 140036022593344 test_util.py:131] Setting up test inputs
I0609 15:32:51.294097 140036022593344 mobilebert_tf2_quant_test.py:17] 	[  1 384], int32
I0609 15:32:51.294435 140036022593344 mobilebert_tf2_quant_test.py:17] 	[  1 384], int32
I0609 15:32:51.29[45](https://github.com/google/iree-samples/runs/6815385767?check_suite_focus=true#step:6:46)82 140036022593344 mobilebert_tf2_quant_test.py:17] 	[  1 384], int32
I0609 15:32:51.297305 140036022593344 test_util.py:134] Invoking TFLite
I0609 15:32:51.870673 140036022593344 test_util.py:96] Invocation time: 0.5702 seconds
I0609 15:32:51.872726 140036022593344 test_util.py:137] Invoke IREE
I0609 15:32:53.257142 140036022593344 test_util.py:111] Invocation time: 1.3744 seconds
I0609 15:32:53.274341 140036022593344 test_util.py:72] Max error (0): 6.358604
I0609 15:32:53.27[46](https://github.com/google/iree-samples/runs/6815385767?check_suite_focus=true#step:6:47)25 140036022593344 test_util.py:72] Max error (1): 6.114042
[  FAILED  ] MobileBertTest.test_compile_tflite
======================================================================
FAIL: test_compile_tflite (__main__.MobileBertTest)
MobileBertTest.test_compile_tflite
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/iree-samples/iree-samples/tflitehub/mobilebert_tf2_quant_test.py", line 35, in test_compile_tflite
    self.compile_and_execute()
  File "/home/runner/work/iree-samples/iree-samples/tflitehub/test_util.py", line 1[47](https://github.com/google/iree-samples/runs/6815385767?check_suite_focus=true#step:6:48), in compile_and_execute
    self.compare_results(iree_results, tflite_results, self.output_details)
  File "/home/runner/work/iree-samples/iree-samples/tflitehub/mobilebert_tf2_quant_test.py", line 31, in compare_results
    self.assertTrue(np.isclose(iree_results[0], tflite_results[0], atol=5.0).all())
AssertionError: False is not true
----------------------------------------------------------------------
Ran 1 test in [63](https://github.com/google/iree-samples/runs/6815385767?check_suite_focus=true#step:6:64)3.[78](https://github.com/google/iree-samples/runs/6815385767?check_suite_focus=true#step:6:79)1s
FAILED (failures=1)
error: command failed with exit status: 1
--

iree-org / iree-samples Goto Github PK

iree-samples's People

Contributors

Stargazers

Watchers

Forkers

iree-samples's Issues

Recommend Projects

Recommend Topics

Recommend Org