ilbers / isar Goto Github PK
View Code? Open in Web Editor NEWIntegration System for Automated Root filesystem generation
License: Other
Integration System for Automated Root filesystem generation
License: Other
Provide such functionaliity
Specifically for large packages like the kernel during development cycle, it would be more efficient to allow building those packages from source via the Debian cross-toolchain. Should be selectable in the recipe, maybe also globally via local.conf.
What is required to allow this? Any concrete ideas? We may have someone here who could contribute to this.
I'm quite new to using isar and bitbake but so far I found working with it a quite smooth experience.
However, I observed that the isar-setup-builddir script is only copying the content of bblayers.conf.sample
and local.conf.sample
to build/conf/
.
See here
isar/scripts/isar-setup-builddir
Lines 69 to 70 in 596732a
This will display an error message that the directory configured in .templateconf
does not contain a local.conf file and also no bblayers.conf.
My naive assumption would be that I could provide a meta-isar/conf/local.conf
right next to the meta-isar/conf/local.conf.sample
and isar-setup-builddir would take this one instead and also not show the warning message.
Something like this:
if [ -r "$TEMPLATECONF/bblayers.conf"]; then
ISARCORELAYERCONF="$TEMPLATECONF/bblayers.conf"
else
ISARCORELAYERCONF="$TEMPLATECONF/bblayers.conf.sample"
fi
Or is this a design decision because of how Bitbake works? I myself like the approach of just deleting the whole build folder and then let isar take care of copying the files to the right locations.
To: [email protected]
From: Jan Kiszka
Subject: Jessie: Host configuration leakage around apt-get
Date: Tue, 27 Jun 2017 19:38:46 +0200
Hi all,
for days, I was hunting a nasty bug around multistrap mystically failing in a jessie docker container while working fine in a stretch container as well as a regular jessie VM installation. It turned out that the docker image (debian:jessie-slim) contains
/etc/apt/apt.conf.d/docker-gzip-indexes
which disables the unpacking of packages index files after downloading. Due to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=774698, this also affected the multistrap call of apt-get. So, everyone trying to build Isar images with jessie can run into this bug. Maybe we should
detect the affect version (<2.2.1) and warn at least?
Considering to pull in 2.2.9 from stretch into my builder image now...
Jan
Building multiconfig:qemuarm-wheezy:isar-image-base
and multiconfig:rpi-jessie:isar-image-base
in a single bitbake
call takes 12m7s, the same in two different build dirs and two bitbake instances takes 8m57s.
In addition, http://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html 2.4 says:
"Support for multiple configurations in this current release of the Yocto Project (Tbd 2.4) has some known issues:
No inter-multi-configuration dependencies exist.
Shared State (sstate) optimizations do not exist. Consequently, if the build uses the same object twice in, for example, two different TMPDIR
directories, the build will either load from an existing sstate cache at the start or build the object twice."
libhello build failed:
$ bitbake multiconfig:qemuarm-stretch:isar-image-base multiconfig:de0-nano-soc-stretch:isar-image-base
mc:de0-nano-soc-stretch:libhello-0.1-98f2e41-r0 do_build:
| configure: creating ./config.status
| /bin/bash ./config.status
| config.status: creating Makefile
| config.status: creating config.h
| config.status: executing depfiles commands
| config.status: executing libtool commands
| make all-am
| make[2]: Entering directory '/home/builder/libhello/libhello-0.1-98f2e41'
| /bin/bash ./libtool --tag=CC --mode=compile arm-linux-gnueabihf-gcc -DHAVE_CONFIG_H -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/home/builder/libhello/libhello-0.1-98f2e41=. -fstack-protector-strong -Wformat -Werror=format-security -c -o hello.lo hello.c
| libtool: compile: arm-linux-gnueabihf-gcc -DHAVE_CONFIG_H -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/home/builder/libhello/libhello-0.1-98f2e41=. -fstack-protector-strong -Wformat -Werror=format-security -c hello.c -fPIC -DPIC -o .libs/hello.o
| libtool: compile: arm-linux-gnueabihf-gcc -DHAVE_CONFIG_H -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/home/builder/libhello/libhello-0.1-98f2e41=. -fstack-protector-strong -Wformat -Werror=format-security -c hello.c -o hello.o >/dev/null 2>&1
| /bin/bash ./libtool --tag=CC --mode=link arm-linux-gnueabihf-gcc -g -O2 -fdebug-prefix-map=/home/builder/libhello/libhello-0.1-98f2e41=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -o libhello.la -rpath /usr/lib/arm-linux-gnueabihf hello.lo
| libtool: link: `hello.lo' is not a valid libtool object
| Makefile:430: recipe for target 'libhello.la' failed
| make[2]: *** [libhello.la] Error 1
| make[2]: Leaving directory '/home/builder/libhello/libhello-0.1-98f2e41'
| Makefile:340: recipe for target 'all' failed
| make[1]: *** [all] Error 2
| make[1]: Leaving directory '/home/builder/libhello/libhello-0.1-98f2e41'
| dh_auto_build: make -j1 returned exit code 2
| debian/rules:22: recipe for target 'build' failed
| make: *** [build] Error 2
| dpkg-buildpackage: error: debian/rules build gave error exit status 2
| WARNING: exit code 2 from a shell command.
| ERROR: Function failed: do_build (log file is located at isar/build/tmp/work/debian-stretch-armhf/libhello-0.1-98f2e41-r0/temp/log.do_build.206
$ cat /etc/debian_version
9.0
$ grep -r CROSS conf/local.conf
ISAR_CROSS_COMPILE ?= "1"
$ bitbake -c cache_base_repo multiconfig:qemuarm-buster:isar-image-base
$ sudo rm -rf tmp
$ vim conf/local.conf
$ grep -r CACHE conf/local.conf
ISAR_USE_CACHED_BASE_REPO ?= "1"
$ bitbake multiconfig:qemuarm-buster:isar-image-base
<snip>
NOTE: Executing RunQueue Tasks
ERROR: mc:qemuarm-buster:isar-bootstrap-host-1.0-r0 do_bootstrap: Function failed: do_bootstrap (log file is located at /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-buster-armhf/isar-bootstrap-host-debian-stretch-amd64/temp/log.do_bootstrap.24832)
ERROR: Logfile of failure stored in: /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-buster-armhf/isar-bootstrap-host-debian-stretch-amd64/temp/log.do_bootstrap.24832
Log data follows:
| DEBUG: Executing shell function do_bootstrap
| W: Target architecture is the same as host architecture; disabling QEMU support
| I: Running command: debootstrap --arch amd64 --verbose --variant=minbase --include=locales --components=main,contrib,non-free --keyring /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-buster-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-keyring.gpg stretch /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-buster-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs file:////home/myo/work/isar/src/trunk/isar/build/downloads/base-apt/apt/debian
| I: Retrieving InRelease
| I: Retrieving Release
| E: Failed getting release file file:////home/myo/work/isar/src/trunk/isar/build/downloads/base-apt/apt/debian/dists/stretch/Release
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_bootstrap (log file is located at /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-buster-armhf/isar-bootstrap-host-debian-stretch-amd64/temp/log.do_bootstrap.24832)
ERROR: Task (multiconfig:qemuarm-buster:/home/myo/work/isar/src/trunk/isar/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb:do_bootstrap) failed with exit code '1'
Provide such functionality
I saw it in the project README. Should start_armhf_vm
be start_vm
?
Error log:
==================================================================
ERROR: hello-1.0-r0 do_unpack: Build of do_unpack failed
ERROR: hello-1.0-r0 do_unpack: Traceback (most recent call last):
File "/home/pfefferl/projects/isar/bitbake/lib/bb/build.py", line 642, in exec_task
return _exec_task(fn, task, d, quieterr)
File "/home/pfefferl/projects/isar/bitbake/lib/bb/build.py", line 580, in _exec_task
exec_func(task, localdata)
File "/home/pfefferl/projects/isar/bitbake/lib/bb/build.py", line 210, in exec_func
bb.utils.mkdirhier(adir)
File "/home/pfefferl/projects/isar/bitbake/lib/bb/utils.py", line 740, in mkdirhier
raise e
File "/home/pfefferl/projects/isar/bitbake/lib/bb/utils.py", line 737, in mkdirhier
os.makedirs(directory)
File "/usr/lib/python3.4/os.py", line 227, in makedirs
makedirs(head, mode, exist_ok)
File "/usr/lib/python3.4/os.py", line 237, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/pfefferl/projects/isar/build/tmp/work/buildchroot/debian-stretch/rootfs//home/builder'
The mentions directory "..../home/builder" does really not exist!!
. isar-init-build-env ../b
bitbake isar-image-base
start_vm -d jessie
The above produces a working image, but fails due to:
/home/ibr/w/work/j/isar/src/b/tmp/deploy/images/vmlinuz-3.16.0-4-armmp: No such file or directory
qemu: could not load kernel '/home/ibr/w/work/j/isar/src/b/tmp/deploy/images/vmlinuz-3.16.0-4-armmp'
The docker image debian:stretch-slim is only 55 MB on x86 (granted, without kernel and modules), isar-image-base rootfs is 540 MB. It should be possible to provide a similarly slim image via Isar as well.
The user should be able to retrieve a list of packages and their exact versions that Isar installed in an image. With that list, it should then be possible to regenerate the very same image later on.
This also means we will need repository servers that preserver all those packages, even if normal mirrors dropped them with the availability of newer versions.
The project has 12 branches. And at one point @alexbluesman advised me to use asmirnov/next
, though he has edited that to next
Question: Is it best to just always use next
branch as using `master seems to be problematic?
Instead use http/https protocol instead
As bitbake/lib/bb/build.py
's shell_trap_code()
injects set -e
into scripts, the premature end of scripts leaves dangling mounts around. This may be fixed by hooking into the EXIT
trap handler also setup by shell_trap_code()
. See the patch and discussion on the Mailing List for details.
Currently, by the patches sent to the Mailing List, dangling mount handling is local to the scripts. Eventually, a more thorough approach may be taken, e.g., based on this proposal using an "unmount stack".
To reproduce:
. isar-init-build-env ../build
time bitbake multiconfig:qemuarm-stretch:isar-image-base
start_vm
Boot log:
qemuarm-stretch-fail.txt
Currently multiconfigs in meta-isar contain more than just machine (MACHINE) and distribution (DISTRO) settings.
That sets a bad example, since bitbake and isar cannot distinguish between other possible settings that could have effected the build.
To fix this, all other configuration options should move into distro or machine configuration files.
See 376b2f0 "meta/conf: Split bitbake config" and https://groups.google.com/d/msg/isar-users/kUN6lM03WpM/OXA0q6XBBQAJ.
Date: Thu, 29 Jun 2017 17:52:24 +0200
From: Jan Kiszka
To: isar-users [email protected]
...because users do not have /sbin in their PATH. Where to fix this best? I don't want to fork imported wic files.
I seem to be unable to get one due to #22 . Can I confirm if anyone else had success or is it just me?
The project is missing contact information to reach out to in case of problems or suggestions. An issue tracker cannot replace a mailing list or something similar. However, even the github home isn't referenced.
How about creating a list? Could be on googlegroups if no mailman instance is available.
Following the user manual steps and running them in a Debian Stretch Virtualbox VM using Vagrant on Mac OS X 10.12.6, is unsuccessful
Expecting a successful build
Build fails. Error msg
chroot: failed to run command ‘/configscript.sh’: Exec format error
WARNING: exit code 126 from a shell command.
vagrant box add debian/stretch64
vagrant init debian/stretch64
vagrant ssh
git clone http://github.com/ilbers/isar.git
cd isar/
. isar-init-build-env ../build
time bitbake isar-image-base isar-image-debug
This issue could be reproduce by building target: multiconfig:qemuarm-wheezy:isar-image-base
.
After build is complete, the following folder stays mounted:
tmp/work/debian-wheezy-armhf/buildchroot/rootfs/run/shm
.
We should start updating master like other projects do, ie. after qualifying next via reviews and CI to be probably stable. It's very misleading for observers that master is not normally maintained.
Isar creates root filesystem directories on the host with root privileges. Those are also required for tools like multistrap, mount and chroot. Currently, Isar requires using sudo ALL=NOPASSWD: ALL for simplicity.
The goal is to build without root privileges. Motivation:
Candidate tools:
See also:
Currently we don't have a clearly define pipeline of the tasks, especially for image recipes. This makes it difficult to introduce new tasks that should be run at certain points of the build process.
For image recipes I would like to propose the following pipeline:
do_rootfs
a (virtual) task after which the rootfs
directory contains the desired root file system with all packages installed, etc.do_rootfs_postprocess
a (virtual) task after which all post processing steps that can not be defined in a package are done.do_image
a virtual task after which the images are created in the work directory of the image.do_deploy
a (virtual) task after which the output artifacts of the recipe are deployed to the deploy directory.do_build
a virtual task that defines the default build process.This way it should be simpler to order custom and currently implemented task better into the structure.
No more lines like these
addtask ubi_image before do_build after do_copy_boot_files do_install_imager_deps do_unpack do_transform_template
but instead
addtask ubi_image before do_image after do_rootfs_postprocess
I have not looked into optimizing the package build pipeline and other recipe types.
The wic tool calls mkfs.ext4 with option "-d". This option does not exist in the Debian jessie version. So it should be mentioned in the readme file that at least Debian stretch is required.
Build fails silently without qemu-arm-static
. Please consider building without it or add an according check and at least a message in meta-isar/recipes-core/images/files/setup.sh
and meta/recipes-devtools/buildchroot/files/setup.sh
to make this failure reason easier to find.
See 376b2f0 "meta/conf: Split bitbake config" and https://groups.google.com/d/msg/isar-users/kUN6lM03WpM/OXA0q6XBBQAJ.
On 1/2/19 9:07 PM, Jan Kiszka wrote:
Hi all,
just saw this in CI build:
[...]
2019-01-02 19:37:55 - INFO - | localepurge: Removing disabled auto-generated config file.
2019-01-02 19:37:55 - INFO - |
2019-01-02 19:37:55 - INFO - | To reinstall all the packages which localepurge has been taking care
2019-01-02 19:37:55 - INFO - | of before, you can use the following command:
2019-01-02 19:37:55 - INFO - |
2019-01-02 19:37:55 - INFO - | apt-get -u --reinstall --fix-missing install $(dpkg -S LC_MESSAGES | cut -d: -f1 | tr ', ' '' | sort -u)
2019-01-02 19:37:55 - INFO - |
2019-01-02 19:37:55 - INFO - | mknod: /dev/console: File exists
2019-01-02 19:37:55 - INFO - | WARNING: exit code 1 from a shell command.
2019-01-02 19:37:55 - INFO - | ERROR: Function failed: do_rootfs (log file is located at /builds/ebsy/debian/build/tmp/work/jailhouse-demo-amd64/demo-image-qemu-amd64/temp/log.do_rootfs.66135)I strongly suspect it comes from debian-configscript.sh:
Create console device if [ ! -e /dev/console ]; then mknod /dev/console c 5 1 fi
Something must be creating /dev/console in the background while the
config script runs. That is weird. Any ideas what could be the race
partner?Do we still need this mknod? It date back to e7e5915. Removing this
would be the safest option because the case might be hard to reproduce -
I've only seen this once so far.Jan
What I noticed is that I cannot overwrite DISTRO_APT_SOURCE in local.conf with a local mirror because of
DISTRO_APT_SOURCE = "..."
Any reason why we should not make them all
DISTRO_APT_SOURCE ??= "..."
? Or is the plan rather to allow setting a set of mirrors and let the machinery pull from them when available, without local.conf overwrites?
root@sertac-build:/DATA/ISAR-Test2# git clone https://github.com/ilbers/isar.git
Cloning into 'isar'...
remote: Counting objects: 4069, done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 4069 (delta 27), reused 39 (delta 22), pack-reused 4020
Receiving objects: 100% (4069/4069), 18.03 MiB | 5.31 MiB/s, done.
Resolving deltas: 100% (2124/2124), done.
root@sertac-build:/DATA/ISAR-Test2# ^C
root@sertac-build:/DATA/ISAR-Test2# cd isar/
root@sertac-build:/DATA/ISAR-Test2/isar# . isar-init-build-env ../build
You had no conf/local.conf file. This configuration file has therefore been
created for you with some default values. You may wish to edit it to, for
example, select a different MACHINE (target hardware). See conf/local.conf
for more information as common configuration options are commented.
You had no conf/bblayers.conf file. This configuration file has therefore been
created for you with some default values. To add additional metadata layers
into your configuration please add entries to conf/bblayers.conf.
For more information about isar, contact the developers at:
https://lists.debian.org/debian-embedded/
You can now run 'bitbake '
Common targets are:
isar-image-base
isar-image-debug
root@sertac-build:/DATA/ISAR-Test2/build# bitbake -s
Traceback (most recent call last):
File "/DATA/ISAR-Test2/isar/bitbake/bin/bitbake", line 31, in
import bb
File "/DATA/ISAR-Test2/isar/bitbake/lib/bb/init.py", line 79, in
from bb import fetch2 as fetch
File "/DATA/ISAR-Test2/isar/bitbake/lib/bb/fetch2/init.py", line 1846, in
from . import clearcase
File "/DATA/ISAR-Test2/isar/bitbake/lib/bb/fetch2/clearcase.py", line 72, in
from distutils import spawn
ImportError: cannot import name 'spawn'
root@sertac-build:/DATA/ISAR-Test2/build# bitbake isar-image-base
Traceback (most recent call last):
File "/DATA/ISAR-Test2/isar/bitbake/bin/bitbake", line 31, in
import bb
File "/DATA/ISAR-Test2/isar/bitbake/lib/bb/init.py", line 79, in
from bb import fetch2 as fetch
File "/DATA/ISAR-Test2/isar/bitbake/lib/bb/fetch2/init.py", line 1846, in
from . import clearcase
File "/DATA/ISAR-Test2/isar/bitbake/lib/bb/fetch2/clearcase.py", line 72, in
from distutils import spawn
ImportError: cannot import name 'spawn'
root@sertac-build:/DATA/ISAR-Test2/build# ^C
root@sertac-build:/DATA/ISAR-Test2/build#
The kernel package version has changed in the debian repository. So the build fails now.
Better to use package "linux-image-amd64" instead of versioned package. The same is with vmlinuz and initrd. Better use /vmlinuz and /initrd.img instead of versioned names from /boot directory.
It would be great, if isar (bitbake) is able to get the checksum for JForg Artifactory's using [1], automatically, not from "SRC_URI[sha256sum]" Or do we have a workaround to avoid hard coded checksum values in bb recipes.
[1] http://forums.jfrog.org/How-to-verify-curl-wget-download-artifact-is-correct-td7580081.html
E.g., bitbake multiconfig:qemuarm-wheezy:isar-image-base multiconfig:qemuarm-jessie:isar-image-base
fails in isar-image-base.bb
with the following error:
cp: cannot stat ‘/home/ibr/w/work/j/isar/src/b-master-1/tmp/work/isar-image-base/qemuarm/rootfs/boot/vmlinuz-3.2.0-4-vexpress’: No such file or directory
tmp/work/isar-image-base/qemuarm
is used for both wheezy and jessie.
Currently, the kernel version is hard-coded in the multiconfig file. Whenever Debian releases a new revision, the multiconfig file has to be updated. 52b8d35 determines the version automatically. It should be rebased onto the current master and updated to cover all multiconfigs.
Further use cases:
asmirnov/next 6f689d7
Include option
omitdebsrc=true
into files
meta-isar/recipes-core/images/files/multistrap.conf.in
meta/recipes-devtools/buildchroot/files/multistrap.conf.in
to avoid import of Debian source repos.
Background: I want to use a company internal Debian mirror. But we do not mirror the source repository part of it.
Specifying the environment variable DL_DIR doesn't work as expected, because ISAR default bitbake configuration overwrites it. DL_DIR = "
is being used instead of DL_DIR ?= "
.
Currently the main page readme says only Debian 8 is supported. May I know what are the plans for supporting Debian 9 if any?
Build every package in its individual buildchroot to prevent dependency problems.
Use cases: TODO
Date: Thu, 29 Jun 2017 17:52:24 +0200
From: Jan Kiszka
To: isar-users [email protected]
I'm looking into IMAGE_TYPE = "wic" so that the user no longer has to call wic explicitly. We do this successfully in the yocto layer meta-iot2000 for quite a while.
Provide such functionality
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.