Building executorch runtime about executorch HOT 16 CLOSED

After reading in detail the documentation on deployment on Android devices a number of times, I have some feedback which I hope you will find of some use:

1. The documentation is focused exclusively on deployment via the Android demo app (and it does state that). As such it is of limited (if any) use for deployment on Android devices via a developer's own application.

2. The documentation makes certain assumptions regarding starting points which are not clear and are not stated. This makes it difficult to use it, at least for developers like me who normally do not work with cmake.

3. Even for documentation focused on the demo app, it lacks what I think is necessary/useful information regarding the mechanisms used in the Android demo app for running inference. Clearly, a developer can dig through the code and figure it out, not a big deal. However, when it comes to documentation (as I think of documentation), I think it is missing some important content.

Thanks

from executorch.

These are great suggestions. Let me get to them to either address or understand better.

1. The documentation is focused exclusively on deployment via the Android demo app (and it does state that). As such it is of limited (if any) use for deployment on Android devices via a developer's own application.

This is absolutely fair. Particularly documentation around integrating executorch in your own app is something I would call WIP. This is because for our first release, we did not get around to focusing on this aspect largely due to competing priorities and resource constraints. I love that you have raised it here as this is something we can target to improve upon immediately.

2. The documentation makes certain assumptions regarding starting points which are not clear and are not stated. This makes it difficult to use it, at least for developers like me who normally do not work with cmake.

Can you say a little more on this? Documentation does list "pre-requisite", but perhaps something is not clear and we can improve that.

3. Even for documentation focused on the demo app, it lacks what I think is necessary/useful information regarding the mechanisms used in the Android demo app for running inference. Clearly, a developer can dig through the code and figure it out, not a big deal. However, when it comes to documentation (as I think of documentation), I think it is missing some important content.

This is a good point. I think as part of addressing your 1st concern, we should cover this.

from executorch.

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/buck2'

Hi @adonnini, in our repo we usually assume the cwd is root directory (executorch/), and we often assume that buck2 is extracted to /tmp/buck2 (see https://pytorch.org/executorch/stable/getting-started-setup.html#building-a-runtime).

You brought up a good point about we should make these assumptions clear.

from executorch.

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/buck2'

Hi @adonnini, in our repo we usually assume the cwd is root directory (executorch/), and we often assume that buck2 is extracted to /tmp/buck2 (see https://pytorch.org/executorch/stable/getting-started-setup.html#building-a-runtime).

You brought up a good point about we should make these assumptions clear.

@adonnini let us know if this helps. As @kirklandsign pointed out this is related to where buck2 is installed.

Totally sympathize with your pain points btw. Some of these may not be communicated clearly. @kirklandsign can we fix the documentation for a few different things that have been brought up here.

from executorch.

cc @kirklandsign

from executorch.

@adonnini for your immediate question on which directory, it should be the "executorch" directory. Same as what is listed in the steps at https://pytorch.org/executorch/stable/getting-started-setup. I do think we can update the documentation to make this clear and you feedback is helpful here.

from executorch.

@kimishpatel, Thanks. Below you will find the commands I ran at "executorch" together with the traceback for the failure.

What am I doing wrong?

euralNetworks/trajectory-prediction-transformers-master/executorch
(base) adonnini1@actlnxlptp6:~/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch$ rm -rf cmake-out && mkdir cmake-out && cd cmake-out 
cmake .. \
    -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
    -DANDROID_ABI=arm64-v8a \
    -DBUCK2=/tmp/buck2 \
    -DEXECUTORCH_BUILD_ANDROID_DEMO_APP_JNI=ON \
    -DEXECUTORCH_BUILD_XNNPACK=ON \
    -DEXECUTORCH_BUILD_FLATC=OFF \
    -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON
-- executorch: Generating source lists
Error while generating /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/executorch_srcs.cmake. Exit code: 1
Output:

Error:
Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/build/extract_sources.py", line 235, in <module>
    main()
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/build/extract_sources.py", line 220, in main
    target_to_srcs[name] = sorted(target.get_sources(graph, runner))
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/build/extract_sources.py", line 133, in get_sources
    sources: set[str] = set(runner.run(["cquery", query]))
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/build/extract_sources.py", line 79, in run
    cp: subprocess.CompletedProcess = subprocess.run(
  File "/home/adonnini1/anaconda3/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/home/adonnini1/anaconda3/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/home/adonnini1/anaconda3/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/buck2'

CMake Error at build/Utils.cmake:103 (message):
  executorch: source list generation failed
Call Stack (most recent call first):
  CMakeLists.txt:170 (extract_sources)


-- Configuring incomplete, errors occurred!

from executorch.

@kimishpatel With regards to your question about my point on "starting points", I apologize. I was not clear.

I was referring to the fact that the documentation as I read it does not say something as simple as:

"...Starting from //executorch run the following commands..."

In other words, set the stage, give the coordinates so to speak.

My wording is poor but I hope you understand what I mean.

By the way, I completely understand about your priorities. Developers benefit from what you are prioritizing. Keeping this in mind, I would say that the documentation is really very good. https://pytorch.org/executorch/stable/demo-apps-android.html#building-an-executorch-android-demo-app seems to have been put together in a hurry, and not by an Android app developer.

from executorch.

In other words, set the stage, give the coordinates so to speak.

100% agree on this.

from executorch.

UPDATE
I decided to set up executorch from scratch following all the instructions in
https://pytorch.org/executorch/stable/getting-started-setup.html

All the steps in
https://pytorch.org/executorch/stable/getting-started-setup.html
were completed successfully

Next I tried to build the runtime library for the executorch Android demo app again
(https://pytorch.org/executorch/stable/demo-apps-android.html#runtime).

Before doing that I ran

export FLATC_EXECUTABLE=$(realpath third-party/flatbuffers/cmake-out/flatc)
python3 -m examples.xnnpack.aot_compiler --model_name="dl3" --delegate
mkdir -p examples/demo-apps/android/ExecuTorchDemo/app/src/main/assets/
cp dl3_xnnpack_fp32.pte examples/demo-apps/android/ExecuTorchDemo/app/src/main/assets/

as per the instructions in
https://pytorch.org/executorch/stable/demo-apps-android.html#ahead-of-time

The attempt to build the runtime ended in failure. Please see the traceback log below.

I fulfilled the prerequisites described in
https://pytorch.org/executorch/stable/demo-apps-android.html#building-an-executorch-android-demo-app

Does it matter that I am running Debian 12 and not one of the Linux variants listed in the prerequisites column?

I am at a loss as to what I should do next, and what I did wrong.

Please let me know what you think I should do. Thanks

(base) adonnini1@actlnxlptp6:~/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch$ export PATH="/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/third-party/flatbuffers/cmake-out:${PATH}"
(base) adonnini1@actlnxlptp6:~/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch$ bash /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/build/install_flatc.sh
Version of /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/third-party/flatbuffers is 23.5.26
A compatible version of flatc is on the PATH and ready to use.
(base) adonnini1@actlnxlptp6:~/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch$ rm -rf cmake-out && mkdir cmake-out && cd cmake-out
cmake .. \
    -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
    -DANDROID_ABI=arm64-v8a \
    -DBUCK2=/tmp/buck2 \
    -DEXECUTORCH_BUILD_ANDROID_DEMO_APP_JNI=ON \
    -DEXECUTORCH_BUILD_XNNPACK=ON \
    -DEXECUTORCH_BUILD_FLATC=OFF \
    -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON
-- ANDROID_PLATFORM not set. Defaulting to minimum supported version
19.
-- The C compiler identification is Clang 14.0.6
-- The CXX compiler identification is Clang 14.0.6
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/adonnini1/Development/android-sdk-linux_x86-android-studio/ndk/25.1.8937393/toolchains/llvm/prebuilt/linux-x86_64/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/adonnini1/Development/android-sdk-linux_x86-android-studio/ndk/25.1.8937393/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- executorch: Generating source lists
-- executorch: Using sources file /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/cmake-out/executorch_srcs.cmake
CMake Error at CMakeLists.txt:232 (message):
  FLATC_EXECUTABLE must be set when EXECUTORCH_BUILD_FLATC is disabled.  Note
  that EXECUTORCH_BUILD_FLATC may be disabled implicitly when cross-compiling
  or when EXECUTORCH_BUILD_HOST_TARGETS is disabled.


-- Configuring incomplete, errors occurred!

from executorch.

I was able to complete
https://pytorch.org/executorch/stable/demo-apps-android.html#building-an-executorch-android-demo-app

by changing
-DEXECUTORCH_BUILD_FLATC=OFF
to
-DEXECUTORCH_BUILD_FLATC=ON \

in

cmake ..
-DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake
-DANDROID_ABI=arm64-v8a
-DBUCK2=/tmp/buck2
-DEXECUTORCH_BUILD_ANDROID_DEMO_APP_JNI=ON
-DEXECUTORCH_BUILD_XNNPACK=ON
-DEXECUTORCH_BUILD_FLATC=OFF
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON

NEXT STEPS

I ran my android application using libexecutorchdemo.so
Not surprsingly, it falied with the following error:

E/log(24873): error java.lang.ClassNotFoundException: Didn't find class "com.example.executorchdemo.executor.NativePeer" on path: DexPathList[[zip file "/data/app/~~f1h9wO_NO3lHM6SjE1G7OA==/com.android.contextq-z80angDwYmHd0nyne1m1qA==/base.apk"],nativeLibraryDirectories=[/data/app/~~f1h9wO_NO3lHM6SjE1G7OA==/com.android.contextq-z80angDwYmHd0nyne1m1qA==/lib/arm64, /data/app/~~f1h9wO_NO3lHM6SjE1G7OA==/com.android.contextq-z80angDwYmHd0nyne1m1qA==/base.apk!/lib/arm64-v8a, /system/lib64, /system_ext/lib64]]
11-12 22:02:14.030: A/lNetworkServic(24873): java_vm_ext.cc:591] JNI DETECTED ERROR IN APPLICATION: JNI NewGlobalRef called with pending exception java.lang.ClassNotFoundException: Didn't find class "com.example.executorchdemo.executor.NativePeer" on path: DexPathList[[zip file "/data/app/~~f1h9wO_NO3lHM6SjE1G7OA==/com.android.contextq-z80angDwYmHd0nyne1m1qA==/base.apk"],nativeLibraryDirectories=[/data/app/~~f1h9wO_NO3lHM6SjE1G7OA==/com.android.contextq-z80angDwYmHd0nyne1m1qA==/lib/arm64, /data/app/~~f1h9wO_NO3lHM6SjE1G7OA==/com.android.contextq-z80angDwYmHd0nyne1m1qA==/base.apk!/lib/arm64-v8a, /system/lib64, /system_ext/lib64]]

As I thought, in order to run inference in my application using the executorch runtime, I need a "generic" executorch naitve library. Is such a library in your plans?

Alternatively, I could try to set up a ""com.example.executorchdemo.executor" package in my application. This could be a shor-term solution.

In any event, I would appreciate knowing whether a general purpose native library for running the executorch runtime in an Android application is planned.

Thanks

from executorch.

@adonnini

As I thought, in order to run inference in my application using the executorch runtime, I need a "generic" executorch naitve library. Is such a library in your plans?

Currently we hardcoded the JNI code in https://github.com/pytorch/executorch/blob/main/examples/demo-apps/android/jni/jni_layer.cpp and the corresponding java code is https://github.com/pytorch/executorch/tree/main/examples/demo-apps/android/ExecuTorchDemo/app/src/main/java/com/example/executorchdemo/executor.

The official API for java apps is not released yet at the time, but yes it is planned.

from executorch.

@kirklandsign I understand. Thanks. In the meantime, I will add the com.example.executorchdemo.executor package to my app.

Is the best way for me to keep track of progress towards the API for Java apps to monitor the executor github?.

from executorch.

@adonnini the java APIs as well as android distributions are very much works in progress. Our first release in Oct'23 was just MVP. We will be following this up with subsequent releases and please stay tuned on the progress there. In the meantime, we would appreciate your feedback as well as feature requests

from executorch.

@kimishpatel Is a pre-release of the Java API available? Where could I find it? I thought @kirklandsign said that it was not released. Even a pre-release (with all the caveats that go along with it) would be very useful because right now, I cannot continue working on this as libexecutorchdemo.so is hard coded to work with the demo app only.
I did try a work around. It did not work.
I will be glad to send feedback and ideas about features.
Thanks

from executorch.

@adonnini yeah so unfortunately java bindings (JNI) are hardcoded in the demo app. Let me get back to you on factoring that out to make it work outside of the demo app. Do you mind filing another issue specifically for this and we can close this one?

from executorch.