wind-river / meta-tensorflow Goto Github PK

License: Other

BitBake 40.60% Python 16.60% Shell 18.97% Assembly 1.03% PHP 6.44% Smarty 14.03% Starlark 2.32%

meta-tensorflow's Introduction

meta-tensorflow

Overview

TensorFlow is an open source software library for high performance numerical
computation primarily used in machine learning. Its flexible architecture
allows easy deployment of computation across a variety of types of platforms
(CPUs, GPUs, TPUs), and a range of systems from single desktops to clusters
of servers to mobile and edge devices.
(https://www.tensorflow.org/)

The build system of TensorFlow is Bazel (https://bazel.build/).

This layer integrates TensorFlow to OE/Yocto platform
- Integrate Google's bazel to Yocto
- Add Yocto toolchain for bazel to support cross compiling.
- Replace python package system(pip/wheel) with Yocto package system(rpm/deb/ipk).

Prerequisite(s)

1. Based on Yocto

Yocto layer dependencies

URI: git://github.com/openembedded/openembedded-core.git
branch: master
revision: HEAD

URI: git://github.com/openembedded/bitbake.git
branch: master
revision: HEAD

URI: git://github.com/openembedded/meta-openembedded.git
layers: meta-python, meta-oe
branch: master
revision: HEAD

URI: git://git.openembedded.org/meta-python2
branch: master
revision: HEAD

2. Based on Wind River Linux

Wind River Linux (CI/CD branch)

Installation

Build and Run

Demo 1. Facial recognition

1.1 Based on Yocto

Face recognition on Yocto

1.2 Based on Wind River Linux

Face recognition on Wind River Linux

Demo 2. Neural Machine Translation

Demo 3. Minigo: A minimalist Go engine modeled after AlphaGo Zero

Project License

Copyright (C) 2019 Wind River Systems, Inc.

All metadata is MIT licensed unless otherwise stated. Source code included
in tree for individual recipes is under the LICENSE stated in each recipe
(.bb file) unless otherwise stated.

Legal Notices

If product is based on Wind River Linux:

All product names, logos, and brands are property of their respective owners.
All company, product and service names used in this software are for identification
purposes only. Wind River are registered trademarks of Wind River Systems.

Disclaimer of Warranty / No Support: Wind River does not provide support and
maintenance services for this software, under Wind River’s standard Software
Support and Maintenance Agreement or otherwise. Unless required by applicable
law, Wind River provides the software (and each contributor provides its
contribution) on an “AS IS” BASIS, WITHOUT WARRANTIES OF ANY KIND, either express
or implied, including, without limitation, any warranties of TITLE, NONINFRINGEMENT,
MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible
for determining the appropriateness of using or redistributing the software
and assume ay risks associated with your exercise of permissions under the license.

Attribution

TensorFlow, the TensorFlow logo and any related marks are trademarks of Google Inc.

meta-tensorflow's People

Contributors

Stargazers

Watchers

Forkers

mhaldrich nobuotsukamoto mauzil prashantchikhalkar chenqi1989 nguyenvuhung ejonesni flovo89 ossystems paroque28 dorozcom jacksonlau-mi justeph mrlhumanoid sandy-lcq fabiorafaelcoutada

meta-tensorflow's Issues

Build fails with Yocto zeus

Building tensorflow fails with the following error:
undefined symbol: _ZNK6google8protobuf10TextFormat17FieldValuePrinter9PrintBoolB5cxx11Eb
Build environment:
Ubuntu 18.04
Target: ARM 8

log.do_compile.18822.txt

Bazel Checksum missmatch @icu component

Hi!

I was trying to add meta-tensorflow to my build but when tensorflow is compiled i get the following error message

:ERROR: .../aarch64-poky-linux/tensorflow/1.13.0-r0/git/tensorflow/tools/pip_package/BUILD:149:1:
 no such package '@icu//': java.io.IOException: Error downloading 
[https://mirror.bazel.build/github.com/unicode-org/icu/archive/release-62-1.tar.gz, 
https://github.com/unicode-org/icu/archive/release-62-1.tar.gz] to .../aarch64-poky-linux
/tensorflow/1.13.0-r0/bazel/output_base/external/icu/release-62-1.tar.gz: 
Checksum was 86b85fbf1b251d7a658de86ce5a0c8f34151027cc60b01e1b76f167379acf181 but
 wanted e15ffd84606323cbad5515bf9ecdf8061cc3bf80fb883b9e6aa162e485aa9761 and 
referenced by '//tensorflow/tools/pip_package:licenses'

i needed to modify git/third_party/icu/workspace.bzl within my build tmp/work
i think this is a tensorflow-repo issue but i wanted to post it here if someone has the same problem...
After this change:

./git/third_party/icu/workspace.bzl:14: sha256 = "86b85fbf1b251d7a658de86ce5a0c8f34151027cc60b01e1b76f167379acf181"

Error isn't showed up anymore, but new ones appeared...
I'm currentrly working with meta-* zeus version.

tensorflow 2.4 on armv7 does not build

After building a couple of hours I get this error on arm32.

It builds on x86-64.

Linking of rule '//tensorflow/lite/examples/label_image:label_image' failed (Exit 1): arm-resy-linux-gnueabi-gcc failed: error executing command

  /workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/bazel/output_base/external/yocto_compiler/recipe-sysroot-native/usr/bin/arm-resy-linux-gnueabi/arm-resy-linux-gnueabi-ar @bazel-out/arm-opt/bin/tensorflow/lite/tools/delegates/libnnapi_delegate_provider.lo-2.params) 
SUBCOMMAND: # //tensorflow/lite/examples/label_image:label_image [action 'Linking tensorflow/lite/examples/label_image/label_image', configuration: fe514663a42db0ce8ce2e417bb1bd7373bcf85317a2e2587c9e7bf739b053306, execution platform: @local_execution_config_platform//:platform] 
(cd /workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/bazel/output_base/execroot/org_tensorflow && \ 
  exec env - \ 
    PATH=/workdir/build/container-arm-v7-tensorflow-master/tmp/sysroots-uninative/x86_64-linux/usr/bin:/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/recipe-sysroot-native/usr/bin/python3-native:/workdir/sources/poky-master/scripts:/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/recipe-sysroot-native/usr/bin/arm-resy-linux-gnueabi:/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/recipe-sysroot/usr/bin/crossscripts:/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/recipe-sysroot-native/usr/sbin:/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/recipe-sysroot-native/usr/bin:/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/recipe-sysroot-native/sbin:/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/recipe-sysroot-native/bin:/workdir/sources/poky-master/bitbake/bin:/workdir/build/container-arm-v7-tensorflow-master/tmp/hosttools \ 
    PWD=/proc/self/cwd \ 
    PYTHON_BIN_PATH=/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/recipe-sysroot-native/usr/bin/python3-native/python3 \ 
    PYTHON_LIB_PATH=/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/recipe-sysroot-native/usr/lib/python3.9/site-packages \ 
    TF2_BEHAVIOR=1 \ 
    TF_CONFIGURE_IOS=0 \ 
  /workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/bazel/output_base/external/yocto_compiler/recipe-sysroot-native/usr/bin/arm-resy-linux-gnueabi/arm-resy-linux-gnueabi-gcc @bazel-out/arm-opt/bin/tensorflow/lite/examples/label_image/label_image-2.params) 
ERROR: /workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/git/tensorflow/lite/examples/label_image/BUILD:15:10: Linking of rule '//tensorflow/lite/examples/label_image:label_image' failed (Exit 1): arm-resy-linux-gnueabi-gcc failed: error executing command  
  (cd /workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/bazel/output_base/execroot/org_tensorflow && \ 
  exec env - \ 
    PATH=/workdir/build/container-arm-v7-tensorflow-master/tmp/sysroots-uninative/x86_64-linux/usr/bin:/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/recipe-sysroot-native/usr/bin/python3-native:/workdir/sources/poky-master/scripts:/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/recipe-sysroot-native/usr/bin/arm-resy-linux-gnueabi:/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/recipe-sysroot/usr/bin/crossscripts:/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/recipe-sysroot-native/usr/sbin:/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/recipe-sysroot-native/usr/bin:/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/recipe-sysroot-native/sbin:/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/recipe-sysroot-native/bin:/workdir/sources/poky-master/bitbake/bin:/workdir/build/container-arm-v7-tensorflow-master/tmp/hosttools \ 
    PWD=/proc/self/cwd \ 
    PYTHON_BIN_PATH=/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/recipe-sysroot-native/usr/bin/python3-native/python3 \ 
    PYTHON_LIB_PATH=/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/recipe-sysroot-native/usr/lib/python3.9/site-packages \ 
    TF2_BEHAVIOR=1 \ 
    TF_CONFIGURE_IOS=0 \

/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/bazel/output_base/external/yocto_compiler/recipe-sysroot-native/usr/bin/arm-resy-linux-gnueabi/../../libexec/arm-resy-linux-gnueabi/gcc/arm-resy-linux-gnueabi/10.2.0/ld: neon_tensor_utils.cc:(.text._ZN3ruy7RunPackILNS_4PathE16ENS_17FixedKernelLayoutILNS_5OrderE0ELi16ELi4EEEaaEEvNS_6TuningERKNS_4EMatEPNS_5PEMatEii[_ZN3ruy7RunPackILNS_4PathE16ENS_17FixedKernelLayoutILNS_5OrderE0ELi16ELi4EEEaaEEvNS_6TuningERKNS_4EMatEPNS_5PEMatEii]+0x2fe): undefined reference to `ruy::Pack8bitColMajorForNeonOutOfOrder4Cols(ruy::PackParams8bit const&)' 
/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/bazel/output_base/external/yocto_compiler/recipe-sysroot-native/usr/bin/arm-resy-linux-gnueabi/../../libexec/arm-resy-linux-gnueabi/gcc/arm-resy-linux-gnueabi/10.2.0/ld: neon_tensor_utils.cc:(.text._ZN3ruy7RunPackILNS_4PathE16ENS_17FixedKernelLayoutILNS_5OrderE0ELi16ELi4EEEaaEEvNS_6TuningERKNS_4EMatEPNS_5PEMatEii[_ZN3ruy7RunPackILNS_4PathE16ENS_17FixedKernelLayoutILNS_5OrderE0ELi16ELi4EEEaaEEvNS_6TuningERKNS_4EMatEPNS_5PEMatEii]+0x384): undefined reference to `ruy::Pack8bitColMajorForNeonOutOfOrder4Cols(ruy::PackParams8bit const&)' 
collect2: error: ld returned 1 exit status 
INFO: Elapsed time: 22848.401s, Critical Path: 3315.50s 
INFO: 23289 processes: 2757 internal, 20532 local. 
FAILED: Build did NOT complete successfully 
FAILED: Build did NOT complete successfully 
WARNING: /workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/temp/run.do_compile.28847:162 exit 1 from '/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/bazel/bazel build --config=v2 -c opt --cpu=arm --subcommands --explain=/workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/temp/explain.log --verbose_explanations --verbose_failures --crosstool_top=@local_config_yocto_compiler//:toolchain --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --verbose_failures --copt -DTF_LITE_DISABLE_X86_NEON //tensorflow:libtensorflow.so //tensorflow:libtensorflow_cc.so //tensorflow:libtensorflow_framework.so //tensorflow/tools/benchmark:benchmark_model //tensorflow/tools/pip_package:build_pip_package tensorflow/examples/label_image/... //tensorflow/lite/examples/label_image:label_image' 
WARNING: Backtrace (BB generated script):  
        #1: do_compile, /workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/temp/run.do_compile.28847, line 162 
        #2: main, /workdir/build/container-arm-v7-tensorflow-master/tmp/work/armv7at2hf-vfp-resy-linux-gnueabi/tensorflow/2.4.0-r0/temp/run.do_compile.28847, line 323 

Backtrace (metadata-relative locations): 
        #1: do_compile, /workdir/sources/poky-master/../meta-tensorflow-master/recipes-framework/tensorflow/tensorflow_2.4.0.bb, line 94

Is there any easy way to skip this test case?

Build for GPU rather than CPU

Thanks for creating these recipes! Got meta-tensorflow to build with yocto warrior and run on the CPU. What needs to be done to build for a GPU (such as CUDA 10)?

build fail with bazel-native

| Executing genrule //src:embedded_tools_nojdk; 0s local
| [-----] Extracting interface //.../fakebuildapi/repository:repository
| ERROR: /home/lijie/try/tsts/build/tmp-glibc/work/x86_64-linux/bazel-native/0.21.0-r0/src/BUILD:237:1: Executing genrule //src:embedded_tools_nojdk failed (Exit 127): bash failed: error executing command
| (cd /home/lijie/try/tsts/build/bazel/bazel_w81DtPDN/out/execroot/io_bazel &&
| exec env -
| PATH=/home/lijie/try/tsts/ts/oe-core/scripts/native-intercept:/home/lijie/try/tsts/ts/oe-core/scripts:/home/lijie/try/tsts/build/tmp-glibc/work/x86_64-linux/bazel-native/0.21.0-r0/recipe-sysroot-native/usr/bin/x86_64-linux:/home/lijie/try/tsts/build/tmp-glibc/work/x86_64-linux/bazel-native/0.21.0-r0/recipe-sysroot-native/usr/bin:/home/lijie/try/tsts/build/tmp-glibc/work/x86_64-linux/bazel-native/0.21.0-r0/recipe-sysroot-native/usr/sbin:/home/lijie/try/tsts/build/tmp-glibc/work/x86_64-linux/bazel-native/0.21.0-r0/recipe-sysroot-native/usr/bin:/home/lijie/try/tsts/build/tmp-glibc/work/x86_64-linux/bazel-native/0.21.0-r0/recipe-sysroot-native/sbin:/home/lijie/try/tsts/build/tmp-glibc/work/x86_64-linux/bazel-native/0.21.0-r0/recipe-sysroot-native/bin:/home/lijie/try/tsts/ts/bitbake/bin:/home/lijie/try/tsts/build/tmp-glibc/hosttools
| /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/host/bin/src/create_embedded_tools "bazel-out/k8-opt/genfiles/src/embedded_tools_nojdk.zip" bazel-out/k8-opt/bin/src/embedded_tools_nojdk.params')
| Execution platform: @bazel_tools//platforms:host_platform
| [1,344 / 1,520] 4 actions running
| Compiling .../3.6.1/src/google/protobuf/compiler/parser.cc; 11s local
| //src/main/java/com/google/devtools/build/lib:packages-internal; 6s worker
| Executing genrule //src:embedded_tools_nojdk; 0s local
| Compiling third_party/grpc/src/core/iomgr/closure.c; 0s local
| /usr/bin/env: 'python': No such file or directory
| [1,344 / 1,520] 4 actions running
| Compiling .../3.6.1/src/google/protobuf/compiler/parser.cc; 11s local
| //src/main/java/com/google/devtools/build/lib:packages-internal; 6s worker
| Executing genrule //src:embedded_tools_nojdk; 0s local
| Compiling third_party/grpc/src/core/iomgr/closure.c; 0s local
| [1,347 / 1,520] .../com/google/devtools/build/lib:packages-internal; 6s worker
| [1,347 / 1,520] .../com/google/devtools/build/lib:packages-internal; 7s worker
| [1,348 / 1,520] no action
| Target //src:bazel_nojdk failed to build
| [1,348 / 1,520] no action
| INFO: Elapsed time: 1322.654s, Critical Path: 229.74s
| [1,348 / 1,520] no action
| INFO: 1266 processes: 1024 local, 242 worker.
| [1,348 / 1,520] no action
| FAILED: Build did NOT complete successfully
| FAILED: Build did NOT complete successfully
|
| ERROR: Could not build Bazel
| WARNING: exit code 1 from a shell command.
|
ERROR: Task (/home/lijie/try/tsts/ts/meta-tensorflow/recipes-devtools/bazel/bazel-native_0.21.0.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 858 tasks of which 0 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
/home/lijie/try/tsts/ts/meta-tensorflow/recipes-devtools/bazel/bazel-native_0.21.0.bb:do_compile
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

meta-tensorflow/meta-demo/conf/layer.conf needs zeus compatibility added

In File:
meta-tensorflow/meta-demo/conf/layer.conf
Change:
LAYERSERIES_COMPAT_tensorflow-demo = "warrior"
To:
LAYERSERIES_COMPAT_tensorflow-demo = "warrior zeus"
Test:
Make sure the Master Branch builds with WRL LTS19.

https://mirror.bazel.build/github.com/unicode-org/icu/archive/release-62-1.tar.gz

Not found anymore.

tensorflow building is failed with glibc 2.29.

I think "0001-grpc-Define-gettid-only-for-glibc-2.30.patch" has problem.

gettid is defined since glibc 2.30. however it seems the condition includes 2.29 in below.

++#if defined(__GLIBC__) && (__GLIBC_MINOR__ >= 29)

in upstream, It seems that gettid will be renamed to sys_gettid.

tensorflow/tensorflow#33758

thanks.

meta-python2 is dead

*) meta-python2 is dead and will not be maintained anymore
*) Are there any plans to update this layer and get rid of this dependency?

using `bazel sync` in `do_fetch`

there are many third party modules used in tensorflow, many of them are cloned to the work directory not as a part of the do_fetch task.
according to bazel documentation one can simply fetch all third party requirements by simply running the command bazel sync.
I strongly suggest that tensorflow's recipe will add the command bazel sync to the do_fetch task via do_fetch_append.

Archive script failure

Python script /oe-core/scripts/native-intercept/ar can't support @file option.

===========================================================
#!/usr/bin/env python3

Wrapper around 'ar' that defaults to deterministic archives

import os
import shutil
import sys

calculate path to the real 'ar'

path = os.environ['PATH']
path = path.replace(os.path.dirname(sys.argv[0]), '')

real_ar = shutil.which('ar', path=path)

if len(sys.argv) == 1:
os.execl(real_ar, 'ar')

modify args to mimic 'ar' configured with --default-deterministic-archives

argv = sys.argv

if argv[1].startswith('--'):
# No modifier given
None
else:
# remove the optional '-'
if argv[1][0] == '-':
argv[1] = argv[1][1:]
if 'U' in argv[1]:
sys.stderr.write("ar: non-deterministic mode requested\n")
else:
argv[1] = argv[1].replace('u', '')
argv[1] = 'D' + argv[1]

os.execv(real_ar, argv)

When argv[1] equals "@bazel-out/k8-opt/bin/external/llvm-project/llvm/libSupport.a-2.params", the "else" clause was executed. As a result argv[1] was modified to "D@bazel-ot/k8-opt/bin/external/llvm-project/llvm/libSpport.a-2.params". Thus, build failed.

My question is, why do we want to remove the character "u" from the file path/name?
Second, should we change "if" in "if 'U' in argv[1]" into "elif"?
Third, when the first byte of is "@", we shouldn't modify argv[1], right?
And last, why do we insert command "D" as "D" is already in the first line of file "libSpport.a-2.params"? (see first a few lines of params file below)

===========================================================
rcsD
bazel-out/k8-opt/bin/external/llvm-project/llvm/libSupport.a
bazel-out/k8-opt/bin/external/llvm-project/llvm/_objs/Support/AArch64TargetParser.o
bazel-out/k8-opt/bin/external/llvm-project/llvm/_objs/Support/ABIBreak.o

tensorflow: build for genericx86-64 target fails

Issue

Building MACHINE=genericx86-64 bitbake tensorflow fails to build (eg, wrong architecture: armeabi (fails during eigen). Note: I am using a patched recipe to build using sumo toolchain

Any pointers on extending the file BUILD or CROSSTOOL files are appreciated.

Primary interest is building the shared objects (eg, libtensorflow.so)
I have forked the project, removed extra dependencies, removed extra build outputs, and successfully built for AArch64 targets (eg, MACHINE=<aarch64-target> bitbake tensorflow using the sumo toolchain.

tensorflow on riscv64

Hi,

When can we expect the tensorflow support on riscv64 architecture , Reply will be highly appreciated.