Comments (14)
guangy10
commented on June 9, 2024
1
@shoumikhin can you take a look?
from executorch.
shoumikhin
commented on June 9, 2024
Hi @kyungsoo3lee, can you please try the following BUCK file:
def define_common_targets():
for aten in (True, False):
aten_suffix = "_aten" if aten else ""
runtime.cxx_library(
name = "runner" + aten_suffix,
srcs = [
"runner.cpp",
],
exported_headers = [
"runner.h",
"util.h",
],
preprocessor_flags = [
"-DUSE_ATEN_LIB",
] if aten else [],
visibility = [
"@EXECUTORCH_CLIENTS",
],
exported_deps = [
"//executorch/examples/models/llama2/sampler:sampler" + aten_suffix,
"//executorch/examples/models/llama2/tokenizer:tokenizer",
"//executorch/extension/evalue_util:print_evalue" + aten_suffix,
"//executorch/extension/module:module" + aten_suffix,
"//executorch/kernels/portable:" + ("generated_lib_aten" if aten else "generated_lib_all_ops"),
"//executorch/runtime/core/exec_aten:lib" + aten_suffix,
],
external_deps = [
"libtorch",
] if aten else [],
)
from executorch.
kyungsoo3lee
commented on June 9, 2024
Hi @shoumikhin, I changed BUCK(targets.bzl) file for main.cpp.
Build success. :)
1. My build command is
/tmp/buck2 build //examples/models/llama2:main --show-output
2. The BUCK file(targets.bzl) is
load("@fbsource//xplat/executorch/build:runtime_wrapper.bzl", "get_oss_build_kwargs", "runtime")
def define_common_targets():
for aten in (True, False):
aten_suffix = "_aten" if aten else ""
runtime.cxx_binary(
name = "main" + aten_suffix,
srcs = [
"main.cpp",
],
preprocessor_flags = [
"-DUSE_ATEN_LIB",
] if aten else [],
deps = [
"//executorch/examples/models/llama2/runner:runner" + aten_suffix,
],
external_deps = [
"gflags",
],
visibility = [
"//executorch/examples/...",
],
define_static_target = True,
**get_oss_build_kwargs()
)
from executorch.
kyungsoo3lee
commented on June 9, 2024
I used the model in the provided param folder and the conversion was successful.
When I am running llama2 pte model. Tokenizer.bin file is required.
Where can I get tokenizer.bin file?
I 00:00:00.000484 executorch:runner.cpp:39] Reading metadata from model
I 00:00:00.000958 executorch:runner.cpp:104] get_vocab_size: 512
I 00:00:00.000970 executorch:runner.cpp:102] The model does not contain get_bos_id method, using default value 1
I 00:00:00.000978 executorch:runner.cpp:104] get_bos_id: 1
I 00:00:00.000985 executorch:runner.cpp:102] The model does not contain get_eos_id method, using default value 2
I 00:00:00.000990 executorch:runner.cpp:104] get_eos_id: 2
I 00:00:00.000998 executorch:runner.cpp:102] The model does not contain get_n_bos method, using default value 1
I 00:00:00.001004 executorch:runner.cpp:104] get_n_bos: 1
I 00:00:00.001010 executorch:runner.cpp:102] The model does not contain get_n_eos method, using default value 1
I 00:00:00.001016 executorch:runner.cpp:104] get_n_eos: 1
I 00:00:00.001036 executorch:runner.cpp:104] get_max_seq_len: 128
E 00:00:00.001073 executorch:tokenizer.cpp:56] couldn't load tokenizer.bin
E 00:00:00.001082 executorch:tokenizer.cpp:193] Tokenizer not initialized
F 00:00:00.001088 executorch:runner.cpp:123] In function generate(), assert failed (num_prompt_tokens >= 1): Expected at least 1 prompt token
Aborted (core dumped)
from executorch.
kyungsoo3lee
commented on June 9, 2024
Hi @shoumikhin.
Do u have any comment? :)
from executorch.
larryliu0820
commented on June 9, 2024
Hi @kyungsoo3lee thanks for trying out our llama offerings. Please be aware that this is largely still WIP so we don't have any guarantee on this will work flawlessly as of now.
To get tokenizer.bin, first thing you need is the tokenizer.model from your Llama2 official checkpoint or huggingface checkpoint. Then you can run the following script to modify it into something readable by runner.cpp:
python -m examples.models.llama2.tokenizer --tokenizer-model <tokenizer.model> --output-path <tokenizer.bin>
Let us know how it goes.
from executorch.
kyungsoo3lee
commented on June 9, 2024
Hi.
Build fail is occurred.
This is error message.
buck2 build //examples/models/llama2:main --show-output
Action failed: root//backends/xnnpack/third-party:ukernels_avxvnni (cxx_compile xnnpack_wrappers/amalgam/gen/avxvnni.c)
Local command returned non-zero exit code 1
Reproduce locally: `clang -o buck-out/v2/gen/root/524f8da68ea2a374/backends/xnnpack/third-party/__ukernels_avxvnni__/__o ...<omitted>... kernels_avxvnni__/.c.argsfile -c backends/xnnpack/third-party/xnnpack_wrappers/amalgam/gen/avxvnni.c (run `buck2 log what-failed` to get the full command)`
stdout:
stderr:
clang: error: unknown argument: '-mavxvnni'
Build ID: c70ed4e3-a14f-4942-bab2-2c97766e8952
Network: Up: 0 B Down: 0 B
Jobs completed: 3. Time elapsed: 0.0s.
Cache hits: 0%. Commands: 1 (cached: 0, remote: 0, local: 1)
BUILD FAILED
Failed to build 'root//backends/xnnpack/third-party:ukernels_avxvnni (prelude//platforms:default#524f8da68ea2a374)'
from executorch.
larryliu0820
commented on June 9, 2024
@kyungsoo3lee hello can you make sure your submodule is up to date?
git submodule update
from executorch.
kyungsoo3lee
commented on June 9, 2024
@larryliu0820 Yes, I updated submodule.
but error is note fixed yet.
Showing commands from: buck2 build //examples/models/llama2:main --show-output
build root//backends/xnnpack/third-party:ukernels_avxvnni (prelude//platforms:default#524f8da68ea2a374) (cxx_compile xnnpack_wrappers/amalgam/gen/avxvnni.c) local env -- "TMPDIR=/home/platformsw/work/executorch-main-0216/executorch/buck-out/v2/tmp/root/524f8da68ea2a374/backends/xnnpack/third-party/__ukernels_avxvnni__/cxx_compile/xnnpack_wrappers/amalgam/gen/avxvnni.c" "BUCK2_DAEMON_UUID=7f032016-0563-49a9-91a2-9aa3349e8b34" "BUCK_BUILD_ID=a495a2a1-e292-444c-894f-f1a2efb98065" clang -o buck-out/v2/gen/root/524f8da68ea2a374/backends/xnnpack/third-party/__ukernels_avxvnni__/__objects__/xnnpack_wrappers/amalgam/gen/avxvnni.c.o @buck-out/v2/gen/root/524f8da68ea2a374/backends/xnnpack/third-party/__ukernels_avxvnni__/.c.argsfile -c backends/xnnpack/third-party/xnnpack_wrappers/amalgam/gen/avxvnni.c
from executorch.
larryliu0820
commented on June 9, 2024
@digantdesai do you know what’s going on here? I still think this is a repo setup issue
from executorch.
digantdesai
commented on June 9, 2024
clang: error: unknown argument: '-mavxvnni'
What is the clang version? It might be too old?
IIRC there should be a fix in upstream xnnpack to allow you to actually not build these kernels on older platform/compilers.
If you update XNNPACK submodule and try again, it might work with XNNPACK_ENABLE_AVXVNNI=OFF.
from executorch.
kyungsoo3lee
commented on June 9, 2024
Hi @digantdesai
My clang version is 10.0.0
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
I am trying to update to clang v18.
How to update XNNPACK submodule?
I always use 'git submodule update' .
Is there another way?
from executorch.
digantdesai
commented on June 9, 2024
So upgrading XNNPACK is already a WIP.
For your local setup, you can do something like,
1 - go to the backends/xnnpack/third-party/xnnpack dir
2 - checkout the XNNPACK version you prefer
3 - cd back to ExecuTorch root
4 - check-in the submodule update
from executorch.
kyungsoo3lee
commented on June 9, 2024
Hi @digantdesai
I upgrade to clang v18.
But another error is occurred.
How to link ptherad library?
- Run command is:
buck2 run examples/models/llama2:main -- --model_path=llama2.pte --tokenizer_path=tokenizer.bin --prompt="Once"
- Error Mssage:
Action failed: root//examples/models/llama2:main (cxx_link_executable)
Local command returned non-zero exit code 1
Reproduce locally: `env -- 'BUCK_SCRATCH_PATH=buck-out/v2/tmp/root/524f8da68ea2a374/examples/models/llama2/__main__/cxx_link_executable' clang++ @buck-out/v2/gen/root/524f8da68ea2a374/examples/models/llama2/__main__/main.linker.argsfile`
stdout:
stderr:
ld.lld: error: undefined symbol: pthread_once
>>> referenced by init.c
>>> init.c.o:(xnn_initialize) in archive buck-out/v2/gen/root/524f8da68ea2a374/backends/xnnpack/third-party/__XNNPACK__/libXNNPACK.a
>>> referenced by hardware-config.c
>>> hardware-config.c.o:(xnn_init_hardware_config) in archive buck-out/v2/gen/root/524f8da68ea2a374/backends/xnnpack/third-party/__XNNPACK__/libXNNPACK.a
>>> referenced by unary-elementwise-config.c
>>> unary-elementwise-config.c.o:(xnn_init_f16_abs_config) in archive buck-out/v2/gen/root/524f8da68ea2a374/backends/xnnpack/third-party/__XNNPACK__/libXNNPACK.a
>>> referenced by unary-elementwise-config.c
>>> unary-elementwise-config.c.o:(xnn_init_f16_clamp_config) in archive buck-out/v2/gen/root/524f8da68ea2a374/backends/xnnpack/third-party/__XNNPACK__/libXNNPACK.a
>>> referenced by unary-elementwise-config.c
>>> unary-elementwise-config.c.o:(xnn_init_f16_elu_config) in archive buck-out/v2/gen/root/524f8da68ea2a374/backends/xnnpack/third-party/__XNNPACK__/libXNNPACK.a
>>> referenced by unary-elementwise-config.c
>>> unary-elementwise-config.c.o:(xnn_init_f16_hswish_config) in archive buck-out/v2/gen/root/524f8da68ea2a374/backends/xnnpack/third-party/__XNNPACK__/libXNNPACK.a
>>> referenced by unary-elementwise-config.c
>>> unary-elementwise-config.c.o:(xnn_init_f16_lrelu_config) in archive buck-out/v2/gen/root/524f8da68ea2a374/backends/xnnpack/third-party/__XNNPACK__/libXNNPACK.a
>>> referenced by unary-elementwise-config.c
>>> unary-elementwise-config.c.o:(xnn_init_f16_neg_config) in archive buck-out/v2/gen/root/524f8da68ea2a374/backends/xnnpack/third-party/__XNNPACK__/libXNNPACK.a
>>> referenced by unary-elementwise-config.c
>>> unary-elementwise-config.c.o:(xnn_init_f16_rndd_config) in archive buck-out/v2/gen/root/524f8da68ea2a374/backends/xnnpack/third-party/__XNNPACK__/libXNNPACK.a
>>> referenced by unary-elementwise-config.c
>>> unary-elementwise-config.c.o:(xnn_init_f16_rndne_config) in archive buck-out/v2/gen/root/524f8da68ea2a374/backends/xnnpack/third-party/__XNNPACK__/libXNNPACK.a
>>> referenced 120 more times
ld.lld: error: undefined symbol: pthread_create
>>> referenced by pthreads.c
>>> pthreads.c.o:(pthreadpool_create) in archive buck-out/v2/gen/root/524f8da68ea2a374/backends/xnnpack/third-party/__pthreadpool__/libpthreadpool.a
ld.lld: error: undefined symbol: pthread_join
>>> referenced by pthreads.c
>>> pthreads.c.o:(pthreadpool_destroy) in archive buck-out/v2/gen/root/524f8da68ea2a374/backends/xnnpack/third-party/__pthreadpool__/libpthreadpool.a
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Build ID: 18b5190e-92f4-459d-8ce7-b7376d0380cd
Network: Up: 0B Down: 0B
Jobs completed: 3. Time elapsed: 0.5s.
Cache hits: 0%. Commands: 1 (cached: 0, remote: 0, local: 1)
BUILD FAILED
Failed to build 'root//examples/models/llama2:main (prelude//platforms:default#524f8da68ea2a374)'
from executorch.
Related Issues (20)
- Does executorch support qat quant for qualcomm qnn backend?
- Remove Arm SDK step from docker image
- Qualcomm AI Engine Direct - support 4bit quantization HOT 1
- buck2 build failure HOT 7
- Unable to run the iOS demo app HOT 4
- ImportError: cannot import name '_transform' from 'executorch.exir.program._program' HOT 3
- AttributeError: module 'mpmath' has no attribute 'rational' HOT 1
- How do you get executorch to run within Mbed OS? HOT 3
- mpmath package version HOT 1
- ExecuTorch: Failed to install inmemoryfs extension. HOT 6
- How to analyze executorch .pte file performance? HOT 6
- Operator torch._ops.quantized_decomposed.quantize_per_tensor.default is not Aten Canonical HOT 4
- Move Arm bare-metal platform builds to C++17, and include arm builds and aot flow in standard precommits HOT 1
- ’xnn_executor_runner' is missing in the documentation 'Building and Running ExecuTorch with XNNPACK Backend'
- Unknown target `generated_lib_all_ops` from package `root//kernels/portable`. HOT 5
- The core has been dumped after alternately reasoning three times about the two preloaded models
- ExecuTorch main branch setup fails while stable branch works HOT 4
- nn.BatchNorm2d is not supported at exir.to_edge HOT 9
- How to perform inference and gathering accuracy metrics on executorch model
- For the Runtime build for QNN
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from executorch.