rpi-distro / pi-gen Goto Github PK

View Code? Open in Web Editor NEW

2.5K 2.5K 1.6K 1.28 MB

Tool used to create the official Raspberry Pi OS images

License: BSD 3-Clause "New" or "Revised" License

Shell 97.72% sed 0.82% Dockerfile 1.46%

pi-gen's People

Contributors

Stargazers

Watchers

Forkers

openrov-dev tjstyle lurch dirkakrid ashang privacylabs aibar staroselskii robertmcelha williamren forevergenin rasppleii priestd09 diglos scds brimstone home-assistant dabonneville gregmac masteinhauser peteryule chzchzchz lgtmcu joeyparrish kartstig timtrittipo jonasrosland turbobytes psema4 alastairhm protothis chiangfamily prae5 frjaraur hj91 mikkyornyx moodlebox mdegrazia cikey exhar uspike leonardok devslashnull davejm cfstras hypermit dride sam3d lazing viteos techwolfy pinge tinkertinker dcwangmit01 gunawanw9 thrilleratplay andragabr ngwese eins78 leetcodes sealsystems damianmoore wlanthermo franke1276 jsr westwork-org jayjlawrence devicehubnet thebagel maxnet kai5263499 khasmek philippmayrth itavero benjaminehowe friskbybergen brocaar olaan andregasser agredain binjun liviuflore gsemet jacen92 glucaci mizuno-as mbcolbert kingbrewer mattsoftware dreisiger-metz eisler colardynit mkolod jsiverskog andrewmichaelsmith jipumarino nubix-de g-e-r landeru tobybatch

pi-gen's Issues

apt-get failures

This may be an issue with the archive.raspberrypi.org server, but I'm inconsistently seeing build errors:

Now at patch 02-persistant-net.diff
[22:43:40] End /work/stage1/02-net-tweaks/00-patches
[22:43:40] Begin /work/stage1/02-net-tweaks/00-run.sh
Adding 'local diversion of /lib/udev/rules.d/75-persistent-net-generator.rules to /lib/udev/rules.d/75-persistent-net-generator.rules.distrib'
[22:43:40] End /work/stage1/02-net-tweaks/00-run.sh
[22:43:40] End /work/stage1/02-net-tweaks
[22:43:40] Begin /work/stage1/03-install-packages
[22:43:40] Begin /work/stage1/03-install-packages/00-packages
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  alsa-utils device-tree-compiler libasound2 libasound2-data libparted2 libsamplerate0 lua5.1 parted triggerhappy
Suggested packages:
  libasound2-plugins libparted-dev libparted-i18n parted-doc
The following NEW packages will be installed:
  alsa-utils device-tree-compiler libasound2 libasound2-data libparted2 libraspberrypi-bin libraspberrypi0 libsamplerate0 lua5.1 parted raspi-config triggerhappy
0 upgraded, 12 newly installed, 0 to remove and 0 not upgraded.
Need to get 4384 kB of archives.
After this operation, 11.4 MB of additional disk space will be used.
Get:1 http://archive.raspberrypi.org/debian/ jessie/main libasound2-data all 1.0.28-1+rpi1 [65.1 kB]
Get:2 http://archive.raspberrypi.org/debian/ jessie/main libasound2 armhf 1.0.28-1+rpi1 [321 kB]
Get:3 http://mirrordirector.raspbian.org/raspbian/ jessie/main libparted2 armhf 3.2-7 [259 kB]
Get:4 http://archive.raspberrypi.org/debian/ jessie/main device-tree-compiler armhf 1.4.1-1+rpi1 [351 kB]
Get:5 http://mirrordirector.raspbian.org/raspbian/ jessie/main libsamplerate0 armhf 0.1.8-8 [939 kB]  
Get:6 http://mirrordirector.raspbian.org/raspbian/ jessie/main alsa-utils armhf 1.0.28-1 [960 kB]
Err http://archive.raspberrypi.org/debian/ jessie/main libraspberrypi0 armhf 1.20161215-1
  404  Not Found [IP: 93.93.130.104 80]
Get:7 http://mirrordirector.raspbian.org/raspbian/ jessie/main lua5.1 armhf 5.1.5-7.1 [75.1 kB]
Get:8 http://mirrordirector.raspbian.org/raspbian/ jessie/main parted armhf 3.2-7 [191 kB]
Err http://archive.raspberrypi.org/debian/ jessie/main libraspberrypi-bin armhf 1.20161215-1
  404  Not Found [IP: 93.93.130.104 80]
Get:9 http://mirrordirector.raspbian.org/raspbian/ jessie/main triggerhappy armhf 0.3.4-2 [31.2 kB]
Get:10 http://archive.raspberrypi.org/debian/ jessie/main raspi-config all 20161207 [17.7 kB]   
Fetched 3211 kB in 2s (1154 kB/s)                          
E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/libraspberrypi0_1.20161215-1_armhf.deb  404  Not Found [IP: 93.93.130.104 80]

E: Failed to fetch http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/libraspberrypi-bin_1.20161215-1_armhf.deb  404  Not Found [IP: 93.93.130.104 80]

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Not really a bug (unless filled under documentation update request) but my question is when building a raspberry pi image can this be done on any Linux PC or does it need to be build on the pi itself (I know certain dependencies need to be installed)?

To not finish with a question 3 quick things I noticed:

Tried doing a build which gave me some errors (only to realize I was starting to script with sh build.sh instead of bash build.sh)
config file contents where unclear to me (figured it out after reading some previous bugs might be nice to add this to the docs :-))
After checking out I noticed the prerun.sh scripts weren't running after a bit of research noticed the sh files needed to be chmodded to have execute rights else they aren't executed.

Not really good things but would be nice stuff to add to the docs (ow yeah and of course my question ;-))

With kind regards

StaticCube

Error --> sudo: unable to resolve host hassbian

The hostname file is correct but the hosts file should be updated to match. The last line in hosts should be 127.0.1.1 hassbian and that will eliminate the sudo: unable to resolve host hassbian error.

Thank you!
Jon

pi@hassbian:~ $ sudo systemctl restart [email protected]
sudo: unable to resolve host hassbian

pi@hassbian:~ $ cat /etc/hostname
hassbian

pi@hassbian:~ $ cat /etc/hosts
127.0.0.1	localhost
::1		localhost ip6-localhost ip6-loopback
ff02::1		ip6-allnodes
ff02::2		ip6-allrouters

127.0.1.1	raspberry

pi@hassbian:~ $ sudo nano /etc/hosts

pi@hassbian:~ $ cat /etc/hosts
127.0.0.1	localhost
::1		localhost ip6-localhost ip6-loopback
ff02::1		ip6-allnodes
ff02::2		ip6-allrouters

127.0.1.1	hassbian

pi@hassbian:~ $

EDIT: This is for the 2017-02-03-HASSbian.img

Build script hangs when trying to install libglib2.0-cil

Hey Team,
I have added few packages to the build script that I want to have installed. In stage2 in one of the files containing list of packages to install I have added libglib2.0-cil. It downloads and unpacks fine but it hangs up on Setting up libglib2.0-cil (2.12.10-5.1) ....

What could be the reason for it ?

Error on build: "Invalid filesystem option set: ^metadata_csum,^huge_file" with docker build

Hey,
I am trying to build pi-gen with the docker build and without it I get the following error. There is no image at the end in the deploy folder.
Using Ubuntu 17.04:

I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xb2a851a7  

Device                                                                      Boot 
Start     End Sectors  Size Id Type
/pi-gen/work/2017-07-13-Raspbian/export-image/2017-07-13-Raspbian-lite.img1      
 8192   94207   86016   42M  c W95 FAT32 (LBA)
/pi-gen/work/2017-07-13-Raspbian/export-image/2017-07-13-Raspbian-lite.img2      
94208 3241835 3147628  1.5G 83 Linux


Command (m for help): The partition table has been altered.
Syncing disks.

/boot: offset 4194304, length 44040192
/:     offset 48234496, length 1611585536
mkfs.fat: warning - lowercase labels might not work properly with DOS or Windows
mke2fs 1.42.12 (29-Aug-2014) 
Invalid filesystem option set: ^metadata_csum,^huge_file
Done! Your image(s) should be in deploy/
/home/ubuntu
/var/lib/docker/volumes/66fb28dda4b5110681d778b8eacbb516798772ed703b99341caa9334b
de50762/_data/2017-07-13-Raspbian/export-image

current workaround is to build without docker

No recovery after failure

There is a horrible issue when the script fails from network issues during any stage of the apt-get processes.

When attempting to resume, the error is "WORK_DIR/?/apt' is not a directory or an apt error that cannot seem to be recovered.

Most of the time resuming is not possible, so a removal of the work directory is required.. this causes a cycle of issues since the original failure was a network issue (in this case DNS randomly failing) and the network issue may or may not occur again. I have also seen an error "cannot verify package" from apt.. which seems that it should be re downloaded instead of reporting an error..

Re downloading the packages every time the script is run with a new image name, or new date, or changing something else where the work directory has to be removed is very time consuming when trying to debug and build a deployable image that can be put in an embedded environment with no user intervention.

Creating an apt-cache directory for the downloaded packages in work/.cache or something and setting apt to use that instead of the temporary root fs being created is probably a good way around re downloading the same packages unnecessarily so the only packages being downloaded are the ones that failed before - even after cleaning the working image directory.

Slow builds

Building an image takes a very long time. Are there any suggestions for areas where improvements could be made to speed up the process.

packages cannot be authenticated during stage1/03-install-packages/00-packages

When I run sudo IMG_NAME=test.img ./build.sh I get an error during 03-install-packages of stage1. I can add --force-yes to the apt-get in build.sh and the error goes away, but it is probably not the safest solution. Is there something else wrong?

Snippet of stdout from the above command.

[...]
[20:25:35] End /home/pi/pi-gen/stage1/02-net-tweaks/00-run.sh
[20:25:36] End /home/pi/pi-gen/stage1/02-net-tweaks
[20:25:36] Begin /home/pi/pi-gen/stage1/03-install-packages
[20:25:36] Begin /home/pi/pi-gen/stage1/03-install-packages/00-packages
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  alsa-utils device-tree-compiler libasound2 libasound2-data libparted2 libsamplerate0 lua5.1 parted triggerhappy
Suggested packages:
  libasound2-plugins libparted-dev libparted-i18n parted-doc
The following NEW packages will be installed:
  alsa-utils device-tree-compiler libasound2 libasound2-data libparted2 libraspberrypi-bin libraspberrypi0 libsamplerate0 lua5.1 parted raspi-config triggerhappy
0 upgraded, 12 newly installed, 0 to remove and 0 not upgraded.
Need to get 4384 kB of archives.
After this operation, 11.4 MB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  libparted2 libsamplerate0 alsa-utils lua5.1 parted triggerhappy
E: There are problems and -y was used without --force-yes

Here is some other information that may or may not be related.

pi@raspberrypi:~ $ cat /etc/apt/sources.list
deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi
deb-src http://archive.raspbian.org/raspbian/ jessie main contrib non-free rpi
pi@raspberrypi:~ $ sudo apt-key update
gpg: key 90FDDD2E: "Mike Thompson (Raspberry Pi Debian armhf ARMv6+VFP) <[email protected]>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
Warning: removed keys keyring  /usr/share/keyrings/raspbian-archive-removed-keys.gpg missing or not readable
pi@raspberrypi:~ $ sudo apt-get update
Hit http://mirrordirector.raspbian.org jessie InRelease
Hit http://archive.raspbian.org jessie InRelease
Hit http://archive.raspberrypi.org jessie InRelease
Hit https://dl.yarnpkg.com stable InRelease
Get:1 http://archive.raspbian.org jessie/main Sources [7,746 kB]
Get:2 http://archive.raspberrypi.org jessie/main armhf Packages [130 kB]
Hit http://archive.raspberrypi.org jessie/ui armhf Packages
Get:3 http://archive.raspbian.org jessie/contrib Sources [64.5 kB]
Get:4 http://archive.raspbian.org jessie/non-free Sources [113 kB]
Get:5 http://archive.raspbian.org jessie/rpi Sources [1,132 B]
Ign http://archive.raspberrypi.org jessie/main Translation-en_GB
Ign http://archive.raspberrypi.org jessie/main Translation-en
Ign http://archive.raspberrypi.org jessie/ui Translation-en_GB
Ign http://archive.raspberrypi.org jessie/ui Translation-en
Hit http://mirrordirector.raspbian.org jessie/main armhf Packages
Hit http://mirrordirector.raspbian.org jessie/contrib armhf Packages
Hit http://mirrordirector.raspbian.org jessie/non-free armhf Packages
Hit http://mirrordirector.raspbian.org jessie/rpi armhf Packages
Ign http://mirrordirector.raspbian.org jessie/contrib Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/contrib Translation-en
Ign http://mirrordirector.raspbian.org jessie/main Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/main Translation-en
Ign http://mirrordirector.raspbian.org jessie/non-free Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/non-free Translation-en
Ign http://mirrordirector.raspbian.org jessie/rpi Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/rpi Translation-en
Fetched 8,054 kB in 3min 37s (37.1 kB/s)
W: Failed to fetch https://dl.yarnpkg.com/debian/dists/stable/InRelease  Unable to find expected entry 'main/binary-armhf/Packages' in Release file (Wrong sources.list entry or malformed file)

W: Failed to fetch http://archive.raspberrypi.org/debian/dists/jessie/main/binary-armhf/Packages  Hash Sum mismatch

E: Some index files failed to download. They have been ignored, or old ones used instead.

First boot configuration file/script

This'd be a relatively major feature, and I know it's been suggested before but I'm unsure if there's an intentional reason it hasn't been implemented (and if so feel free to ignore the rest of this proposal).

Description of the problem

There are many reasons this would be beneficial, but the most commonly cited issue is configuring a Raspberry Pi through just the use of wifi. As it stands right now, you'd have to do one of the following:

Compile your own image with the wifi configuration baked in
Use a third party utility that relies on custom images such as PiBakery
Connect ethernet on initial boot to add appropriate wifi configuration
Use an OS with write support for ext4 (neither Mac or Windows can do this natively)

All of the above solutions have frustrating aspects to them; a native solution that requires no third party utilities or faffing around with ethernet cables would be ideal.

Feature proposal

In essence, a configuration file/script placed in the boot partition of the SD card. It's FAT which means that almost every operating system in existence can read and write to it natively.

Much like with the empty ssh file, this would only take effect on first boot (thus hopefully eliminating many of the security concerns with having such a configuration file).

File specifics

This could theoretically be implemented in two ways:

A config.json, config.txt or config.yml with structured config data
A config.sh of some kind that simply executes on first boot

It could be either/or, as there are advantages and disadvantages to both. A structured config file would be easier to setup and have fewer security concerns (with the limited number of properties that could be modified), but with the greater overhead of having to parse the file contents.

Likewise a config.sh would be slightly harder to configure but offer the user much greater customisability over their Pi. The security concerns would be slightly more-so but if the user takes care to make sure it's freshly written image (with validated checksums) it's unlikely it could be used maliciously. It'd be fast though.

Perhaps the option could be allowed for both, so that a user can take whichever approach they are most comfortable with.

Implementation

Placing a file that executes the script (or parses the structured config data) in /etc/rc.local or /etc/init.d/... and then deletes itself so that it is guaranteed to only run once would probably be the best way to go.

I personally think this would be a fantastic feature to have natively, and would eliminate a lot of headache with setting up a new Raspberry Pi. I would be more than happy to do all of the scripting, implementation and documentation for it myself. Just wanted to run it by you and see what your thoughts were before attempting 😄

Paths with spaces do not work

Upon inspecting the code and testing the build system, there are many many places in several scripts where if the current path (pwd) has spaces in it, the script will fail.
These should be cleaned up so we don't have to rely on the pwd having no spaces.

I have seen talk of changing the build script to python, but I have seen the dependency check script fail too. Looks like the whole build system needs to be modified to handle such paths.

Python_Openzwave autoinstall fails (HASSBian 1.23)

Forgive my pull req... Should have been an issue.

Fresh flash of hassbian 1.23
added zwave: to configuration.yaml
Receive the following error in log:

Jun 07 12:35:49 hassbian hass[738]: 2017-06-07 12:35:49 INFO (Thread-4) [homeassistant.util.package] Attempting install of python_openzwave==0.4.0.31
Jun 07 12:38:19 hassbian hass[738]: 2017-06-07 12:38:19 ERROR (Thread-4) [homeassistant.util.package] Unable to install package python_openzwave==0.4.0.31: Command "/srv/homeassistant/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-92041o8k/python-openzwave/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-uses1i9y-record/install-record.txt --single-version-externally-managed --compile --install-headers /srv/homeassistant/include/site/python3.4/python-openzwave --home=/tmp/tmpyxonxd5x" failed with error code 1 in /tmp/pip-build-92041o8k/python-openzwave/
Jun 07 12:38:19 hassbian hass[738]: 2017-06-07 12:38:19 ERROR (MainThread) [homeassistant.setup] Not initializing zwave because could not install dependency python_openzwave==0.4.0.31
Jun 07 12:38:19 hassbian hass[738]: 2017-06-07 12:38:19 ERROR (MainThread) [homeassistant.setup] Setup failed for zwave: Could not install all requirements.

dependencies_check fail

so i was testing the new build with sudo IMG_NAME="raspbian" ./build.sh and there is an error tossed from the dependencies_check file
/home/ssfdre38/pi-gen/scripts/dependencies_check: line 27: unexpected EOF while looking for matching `"'
Line 27 is echo "$missing"

Question: What are steps 01 and 02 in export-image?

I notice there are two intermediate steps in the export-image stage called set-sources and network. Is there a reason these were added here as opposed to one of the earlier stages? 01-set-sources also has:

on_chroot sh -e - <<EOF
apt-get update
apt-get -y dist-upgrade
apt-get clean
EOF

The update and dist-upgrade seem to be done already in stage0/01-configure-apt/00-run.sh and the clean is done in stage2/03-cleanup/00-run.sh. Wondering if this ends up being redundant.

Suggestion: Use sudoers.d instead of patching /etc/sudoers

Currently stage2/01-sys-tweaks applies a patch to /etc/sudoers, which is a Debian conffile. If at all possible, we shouldn't be modifying Debian conffiles. The reason for that is that should /etc/sudoers ever change in Debian, the user is going to be asked whether they want to keep "their" local changes or adopt Debian's.

Instead, I propose the solution I use on my own systems:

diff --git a/stage2/01-sys-tweaks/00-patches/00-sudoers.diff b/stage2/01-sys-tweaks/00-patches/00-sudoers.diff
deleted file mode 100644
index 3fda3d3..0000000
--- a/stage2/01-sys-tweaks/00-patches/00-sudoers.diff
+++ /dev/null
@@ -1,10 +0,0 @@
-Index: jessie-stage2/rootfs/etc/sudoers
-===================================================================
---- jessie-stage2.orig/rootfs/etc/sudoers
-+++ jessie-stage2/rootfs/etc/sudoers
-@@ -25,3 +25,5 @@ root ALL=(ALL:ALL) ALL
- # See sudoers(5) for more information on "#include" directives:
- 
- #includedir /etc/sudoers.d
-+
-+pi ALL=(ALL) NOPASSWD: ALL
diff --git a/stage2/01-sys-tweaks/00-patches/series b/stage2/01-sys-tweaks/00-patches/series
index b1381c8..5cb4c55 100644
--- a/stage2/01-sys-tweaks/00-patches/series
+++ b/stage2/01-sys-tweaks/00-patches/series
@@ -1,4 +1,3 @@
-00-sudoers.diff
 01-useradd.diff
 02-swap.diff
 03-console-setup.diff
diff --git a/stage2/01-sys-tweaks/01-run.sh b/stage2/01-sys-tweaks/01-run.sh
index 9356ce6..7a9759c 100755
--- a/stage2/01-sys-tweaks/01-run.sh
+++ b/stage2/01-sys-tweaks/01-run.sh
@@ -8,6 +8,8 @@ install -d                      ${ROOTFS_DIR}/etc/systemd/system/rc-local.service.d
 install -m 644 files/ttyoutput.conf            ${ROOTFS_DIR}/etc/systemd/system/rc-local.service.d/

 install -m 644 files/50raspi               ${ROOTFS_DIR}/etc/apt/apt.conf.d/
+install -m 440 files/group_sudo                ${ROOTFS_DIR}/etc/sudoers.d
+


 on_chroot sh -e - <<EOF
@@ -25,7 +27,7 @@ on_chroot sh -e - << \EOF
 for GRP in input spi i2c gpio; do
    groupadd -f -r $GRP
 done
-for GRP in adm dialout cdrom audio users sudo video games plugdev input gpio spi i2c netdev; do
+for GRP in adm dialout cdrom audio users sudo video games plugdev input gpio spi i2c netdev sudo; do
   adduser pi $GRP
 done
 EOF
diff --git a/stage2/01-sys-tweaks/files/group_sudo b/stage2/01-sys-tweaks/files/group_sudo
new file mode 100644
index 0000000..44f8752
--- /dev/null
+++ b/stage2/01-sys-tweaks/files/group_sudo
@@ -0,0 +1,2 @@
+# Allow members of group sudo to execute any command without a password
+%sudo  ALL=(ALL:ALL) NOPASSWD: ALL

The file /etc/sudoers.d/README explains the rules for this.

NB: This patch does provide a small functionality change. Instead of giving the pi user explicitly unlimited access to use sudo, it adds pi to the group sudo and then overrides the rule in /etc/sudoers to make that access passwordless.

You could put the group_sudo file into one of the Raspbian mods packages, but that might change behavior for existing systems. I'd suggest doing it only on new systems as I've done here. In any event, if you opt to add it to a package instead of doing it here as I've done, mark it as a conffile so that user changes to the file will be preserved unless they choose not to.

Raspberry Pi Debian release key is unchecked.

In the bootstrap function in scripts/common the --no-check-gpg flag is provided to deboostrap. Seems like it would be a little better to embed the key signature somewhere in the repository, get the key, check it against the signature, and then if it is good add it to a keyring so that deboostrap can validate that the repository is authentic.

ioctl failures

When doing successive build, I get ioctl errors using docker or build.sh script.
I tried using virtualbox or a native debian distro

I think that the chroot created is not properly unmounted.

Raspbian Stretch?

Now that Stretch has been officially released, do we know what's happening with Raspbian being updated to Stretch?

Suggestion: package changes

Should git-core be removed from stage3/00-install-packages/00-packages and replaced with git in either stage2 or stage4? In stage2 it makes -lite a little less lite, and it does fit with the more complete development tools suggested by stage4.

The gvfs-daemons package now depends on udisks2. Anything else that uses it in Raspbian requires that or udisks. Currently stage3 installs udisks. Perhaps it is time to update that to udisks2? Doing this will cause a few other packages not to be installed: hdparm, powermgmt-base, and cryptsetup-bin.

./build-docker.sh failed raspbian release file

Hello,
I'm currently working on a custom image of Raspbian at stage2 level with some specific tweaks. It works well with ./build.sh
I would like testing it with docker support :

./build-docker.sh 
Sending build context to Docker daemon 1.597 MB
Step 1 : FROM debian:jessie
 ---> 8cedef9d7368
Step 2 : RUN apt-get -y update &&     apt-get -y install         git vim parted         quilt realpath qemu-user-static debootstrap zerofree pxz zip dosfstools         bsdtar libcap2-bin rsync grep     && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 19a0cb25957d
Step 3 : COPY . /pi-gen/
 ---> Using cache
 ---> 82bd079b4475
Step 4 : VOLUME /pi-gen/work /pi-gen/deploy
 ---> Using cache
 ---> bbe618019985
Step 5 : CMD /bin/sh
 ---> Using cache
 ---> 3211122be11f
Successfully built 3211122be11f
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
update-binfmts: warning: qemu-aarch64 already enabled in kernel.
[14:53:26] Begin /pi-gen
[14:53:26] Begin /pi-gen/stage0
[14:53:26] Begin /pi-gen/stage0/prerun.sh
I: Running command: debootstrap --arch armhf --foreign --components=main,contrib,non-free --no-check-gpg jessie /pi-gen/work/2017-04-20-Raspbian/stage0/rootfs http://mirrordirector.raspbian.org/raspbian/
I: Retrieving Release 
E: Failed getting release file http://mirrordirector.raspbian.org/raspbian/dists/jessie/Release
Done! Your image(s) should be in deploy/

when i launch container manually and tries to wget this file it works fine.

I think it is a very easy issue but don't find where i fail.

Thanks !

Systemd services

This may be irrelevant/too hard/wrong approach.

Some of my Pi provisioning mechanisms interact with running services to confirm that updates have worked as expected. The approach requires systemd within the container. Is this an approach that is a/ useful in general, b/ likely to be supported?

tia
Tim

Controller or keyboard not detected

hey just downloaded and installed the image,

it boots and i end up on the meny, but none of my remote (SNES or PS3) are detected, and, also, my keyboad...does not seem to be detected...any way to help me out?

Printing "Done! Your image(s) should be in deploy/" after failure

After build fails to create raspbian image, pi-gen no matter prints that image should be in deploy/ folder. This is a small bug, but it is annoying when your build fails and you must look into deploy folder expecting that image is there... But it's not true =)

-bash: startx: command not found

After using pi-gen and build.sh to construct (what I am to believe) is an image of a desktop Raspbian system and subsequently copying this image onto a 16GB SD card, I am unable to access the GUI on my Raspberry Pi. Attempting to run startx returns the error in the title, whilst attempting to start the GUI on boot via raspi-config boots the pi into the CLI without providing any error messages/feedback.

I am relatively new to Rapsbian, and as such, I have two queries relating to this:

Am I missing something obvious in the Raspberry Pi installation process? After loading the image onto an SD card, I have been placing it in to my Raspberry Pi and powering it up.
How do I know which Raspbian Stage I have built to? In the "Stage Anatomy" section of readme.md, it is stated that the build of Raspbian is divided up into several stages from 0 to 4 inclusive, where stages 3 and 4 refer to the full Raspbian desktop systems as opposed to the Raspbian-Lite system. However, pi-gen contains stages 0-5 inclusive and I was unable to determine what Stage 5 relates to. I have constructed images up to (and including) Stage 3 and Stage 4 of pi-gen, and I experience the same issue detailed above with both images.

Thank you for all the great work!

export-noobs failure if you use /dev/mapper

I posted part of this in my run of pi-gen in #13, and I've noticed I always get the same result:

[05:13:41] Begin /home/tjcarter/Source/RPi-Distro/pi-gen/export-noobs
[05:13:41] Begin /home/tjcarter/Source/RPi-Distro/pi-gen/export-noobs/prerun.sh
umount: /: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)

real    72m34.969s
user    70m47.640s
sys     2m21.440s

Note this was my RPi-Distro tree, which has no patches.

You'll note that there is no End /home/tjcarter/Source/RPi-Distro/pi-gen/export-noobs/prerun.sh to match, and it ends with umount returning an error. (An error in any stage causes pi-gen to bail, leaving filesystems mounted and other issues, but that's a slightly more complex issue.) The prerun.sh is failing in its final line to unmount_image ${IMG_FILE}. At first glance, IMG_FILE was unmounted, but a second look spots the kpartx call which would've placed the image back in use.

Again, this is a problem I see in RPi-Distro's tree, however I'm going to use my own tree to try and diagnose the problem because it has a completely populated work/ from today, resulting in a 15 minute build instead of a 70-90 minute build depending on disk caching.

My debug procedure mostly consists of running build.sh with the following patch:

diff --git a/export-noobs/prerun.sh b/export-noobs/prerun.sh
index a531be6..bde31b7 100755
--- a/export-noobs/prerun.sh
+++ b/export-noobs/prerun.sh
@@ -1,4 +1,4 @@
-#!/bin/bash -e
+#!/bin/bash -ex

 IMG_FILE="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.img"
 NOOBS_DIR="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}"

Relevant output follows:

[00:35:45] End /home/tjcarter/Source/RasppleII/pi-gen/stage4
[00:35:45] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image
[00:35:45] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/prerun.sh
mkfs.fat: warning - lowercase labels might not work properly with DOS or Windows
mke2fs 1.43.3 (04-Sep-2016)
mount: /dev/mapper/loop1p2 mounted on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-image/rootfs.
mount: /dev/mapper/loop1p1 mounted on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-image/rootfs/boot.
[00:35:57] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/prerun.sh
[00:35:57] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/00-allow-rerun
[00:35:57] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/00-allow-rerun/00-run.sh
[00:35:57] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/00-allow-rerun/00-run.sh
[00:35:57] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/00-allow-rerun
[00:35:57] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources
[00:35:57] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/00-patches
Converting meta-data to version 2
Applying patch /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/00-patches/0-sources.diff
patching file rootfs/etc/apt/sources.list.d/raspi.list

Now at patch /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/00-patches/0-sources.diff
[00:35:57] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/00-patches
[00:35:57] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/01-run.sh
Hit http://archive.raspberrypi.org jessie InRelease
Get:1 http://mirrordirector.raspbian.org jessie InRelease [14.9 kB]
Hit http://archive.raspberrypi.org jessie/main armhf Packages 
Get:2 http://mirrordirector.raspbian.org jessie/main armhf Packages [8981 kB]
Hit http://archive.raspberrypi.org jessie/ui armhf Packages           
Ign http://archive.raspberrypi.org jessie/main Translation-en          
Ign http://archive.raspberrypi.org jessie/ui Translation-en
Get:3 http://mirrordirector.raspbian.org jessie/contrib armhf Packages [37.5 kB]          
Get:4 http://mirrordirector.raspbian.org jessie/non-free armhf Packages [70.3 kB]         
Get:5 http://mirrordirector.raspbian.org jessie/rpi armhf Packages [1356 B]               
Ign http://mirrordirector.raspbian.org jessie/contrib Translation-en                      
Ign http://mirrordirector.raspbian.org jessie/main Translation-en                         
Ign http://mirrordirector.raspbian.org jessie/non-free Translation-en                     
Ign http://mirrordirector.raspbian.org jessie/rpi Translation-en                          
Fetched 9105 kB in 20s (449 kB/s)                                                         
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
[00:36:45] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/01-run.sh
[00:36:45] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources
[00:36:45] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/02-network
[00:36:45] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/02-network/01-run.sh
[00:36:45] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/02-network/01-run.sh
[00:36:45] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/02-network
[00:36:45] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/03-finalise
[00:36:45] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/03-finalise/01-run.sh
Stopping fake hwclock: saving system time.
Mode:     real
Files:    2228
Linked:   326 files
Compared: 0 xattrs
Compared: 568 files
Saved:    9.13 MiB
Duration: 0.19 seconds
14081/119135/321792
zip /home/tjcarter/Source/RasppleII/pi-gen/deploy/image_2016-09-16-RasppleII-lite.zip /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-image/2016-09-16-RasppleII-lite.img
  adding: 2016-09-16-RasppleII-lite.img (deflated 78%)
[00:37:24] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/03-finalise/01-run.sh
[00:37:24] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/03-finalise
[00:37:24] End /home/tjcarter/Source/RasppleII/pi-gen/export-image
[00:37:24] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image
[00:37:24] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/prerun.sh
mkfs.fat: warning - lowercase labels might not work properly with DOS or Windows
mke2fs 1.43.3 (04-Sep-2016)
mount: /dev/mapper/loop1p2 mounted on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-image/rootfs.
mount: /dev/mapper/loop1p1 mounted on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-image/rootfs/boot.
[00:39:06] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/prerun.sh
[00:39:06] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/00-allow-rerun
[00:39:06] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/00-allow-rerun/00-run.sh
[00:39:06] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/00-allow-rerun/00-run.sh
[00:39:06] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/00-allow-rerun
[00:39:06] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources
[00:39:06] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/00-patches
Converting meta-data to version 2
Applying patch /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/00-patches/0-sources.diff
patching file rootfs/etc/apt/sources.list.d/raspi.list

Now at patch /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/00-patches/0-sources.diff
[00:39:06] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/00-patches
[00:39:06] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/01-run.sh
Hit http://archive.raspberrypi.org jessie InRelease
Get:1 http://mirrordirector.raspbian.org jessie InRelease [14.9 kB]
Hit http://archive.raspberrypi.org jessie/main armhf Packages 
Get:2 http://mirrordirector.raspbian.org jessie/main armhf Packages [8981 kB]
Hit http://archive.raspberrypi.org jessie/ui armhf Packages            
Ign http://archive.raspberrypi.org jessie/main Translation-en          
Ign http://archive.raspberrypi.org jessie/ui Translation-en
Get:3 http://mirrordirector.raspbian.org jessie/contrib armhf Packages [37.5 kB]          
Get:4 http://mirrordirector.raspbian.org jessie/non-free armhf Packages [70.3 kB]         
Get:5 http://mirrordirector.raspbian.org jessie/rpi armhf Packages [1356 B]               
Ign http://mirrordirector.raspbian.org jessie/contrib Translation-en                      
Ign http://mirrordirector.raspbian.org jessie/main Translation-en                         
Ign http://mirrordirector.raspbian.org jessie/non-free Translation-en                     
Ign http://mirrordirector.raspbian.org jessie/rpi Translation-en                          
Fetched 9105 kB in 18s (483 kB/s)                                                         
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
[00:39:52] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources/01-run.sh
[00:39:52] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/01-set-sources
[00:39:52] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/02-network
[00:39:52] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/02-network/01-run.sh
[00:39:52] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/02-network/01-run.sh
[00:39:52] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/02-network
[00:39:52] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/03-finalise
[00:39:52] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-image/03-finalise/01-run.sh
Stopping fake hwclock: saving system time.
Mode:     real
Files:    6984
Linked:   1348 files
Compared: 0 xattrs
Compared: 3866 files
Saved:    36.81 MiB
Duration: 0.61 seconds
35687/105480/964096
zip /home/tjcarter/Source/RasppleII/pi-gen/deploy/image_2016-09-16-RasppleII.zip /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-image/2016-09-16-RasppleII.img
  adding: 2016-09-16-RasppleII.img (deflated 65%)
[00:42:09] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/03-finalise/01-run.sh
[00:42:09] End /home/tjcarter/Source/RasppleII/pi-gen/export-image/03-finalise
[00:42:09] End /home/tjcarter/Source/RasppleII/pi-gen/export-image
[00:42:09] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-noobs
[00:42:09] Begin /home/tjcarter/Source/RasppleII/pi-gen/export-noobs/prerun.sh
+ IMG_FILE=/home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII.img
+ NOOBS_DIR=/home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII
+ unmount_image /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII.img
+ sync
+ sleep 1
++ losetup -j /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII.img
++ cut -f1 -d:
+ local LOOP_DEV=/dev/loop0
+ '[' -n /dev/loop0 ']'
++ mount
+++ basename /dev/loop0
+++ head -n 1
+++ cut -f 3 -d ' '
++ grep loop0
+ local MOUNTED_DIR=
+ '[' -n '' ']'
+ sleep 1
+ kpartx -ds /dev/loop0
+ losetup -d /dev/loop0
+ mkdir -p /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs
+ cp /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-image/2016-09-16-RasppleII.img /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/
+ rm -rf /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII
++ kpartx -asv /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII.img
++ grep -E -o -m1 'loop[[:digit:]]+'
++ head -n 1
+ LOOP_DEV=loop0
+ BOOT_DEV=/dev/mapper/loop0p1
+ ROOT_DEV=/dev/mapper/loop0p2
+ mkdir -p /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs
+ mkdir -p /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/bootfs
+ mkdir -p /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII
+ mount /dev/mapper/loop0p2 /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs
+ mount /dev/mapper/loop0p1 /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-Raspple$
I/export-noobs/bootfs
+ tar -I pxz -C /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-no$
bs/bootfs -cpf /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noo$
s/2016-09-16-RasppleII/boot.tar.xz .
+ tar -I pxz -C /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noo
bs/rootfs -cpf /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noob
s/2016-09-16-RasppleII/root.tar.xz .
+ unmount_image /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noo
bs/2016-09-16-RasppleII.img
+ sync
+ sleep 1
++ losetup -j /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs
/2016-09-16-RasppleII.img
++ cut -f1 -d:
+ local LOOP_DEV=/dev/loop0
+ '[' -n /dev/loop0 ']'
++ mount
+++ basename /dev/loop0
+++ head -n 1
+++ cut -f 3 -d ' '
++ grep loop0
+ local MOUNTED_DIR=
+ '[' -n '' ']'
+ sleep 1
+ kpartx -ds /dev/loop0
+ losetup -d /dev/loop0
+ mkdir -p /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs
+ cp /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-image/2016-09-16-RasppleII.img /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/
+ rm -rf /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII
++ kpartx -asv /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII.img
++ grep -E -o -m1 'loop[[:digit:]]+'
++ head -n 1
+ LOOP_DEV=loop0
+ BOOT_DEV=/dev/mapper/loop0p1
+ ROOT_DEV=/dev/mapper/loop0p2
+ mkdir -p /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs
+ mkdir -p /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/bootfs
+ mkdir -p /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII
+ mount /dev/mapper/loop0p2 /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs
+ mount /dev/mapper/loop0p1 /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/bootfs
+ tar -I pxz -C /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/bootfs -cpf /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII/boot.tar.xz .
+ tar -I pxz -C /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs -cpf /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII/root.tar.xz .
+ unmount_image /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII.img
+ sync
+ sleep 1
++ losetup -j /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/2016-09-16-RasppleII.img
++ cut -f1 -d:
+ local LOOP_DEV=/dev/loop0
+ '[' -n /dev/loop0 ']'
++ mount
+++ basename /dev/loop0
+++ head -n 1
+++ cut -f 3 -d ' '
++ grep loop0
+ local 'MOUNTED_DIR=/dev/mapper/loop0p2 on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs type ext4 (rw,relatime,data=ordered)
/dev/mapper/loop0p1 on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/bootfs type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)'
+ '[' -n '/dev/mapper/loop0p2 on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs type ext4 (rw,relatime,data=ordered)
/dev/mapper/loop0p1 on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/bootfs type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)' ']'
++ dirname /dev/mapper/loop0p2 on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs type ext4 '(rw,relatime,data=ordered)' /dev/mapper/loop0p1 on /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/bootfs type vfat '(rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)'
+ unmount /dev/mapper . /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs . . . /dev/mapper . /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs . . .
+ '[' -z /dev/mapper ']'
+ DIR=/dev/mapper
+ mount
+ grep -q /dev/mapper
++ mount
++ grep /dev/mapper
++ cut -f 3 -d ' '
++ sort -r
+ local 'LOCS=/home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs
/home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/bootfs/'
+ for loc in '$LOCS'
+ umount /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/rootfs
+ for loc in '$LOCS'
+ umount /home/tjcarter/Source/RasppleII/pi-gen/work/2016-09-16-RasppleII/export-noobs/bootfs
+ for loc in '$LOCS'
+ umount /
umount: /: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)

real    14m32.352s
user    35m18.548s
sys     0m33.020s

As you can see, unmount_image is doing all kinds of not intended things there.

Building forward from Stage 2

I'm looking to go forward from a lite system toward a server-type setup (so, not a graphical direction). Is the right way to do this in pi-gen to just remove the existing Stage 3/4 and build a new Stage 3 that installs what I need with the config I'm looking for, or am I thinking about this wrong?

Suggest: Comments in packages files

Currently ##-packages and ##-packages-nr files are just lists of packages to include, one or more per line. Might it be useful to have comments in these files for later maintenance to indicate why packages are there, or to provide some logical grouping?

Here's what I have in mind:

diff --git a/build.sh b/build.sh
index 5b745e8..84daeda 100755
--- a/build.sh
+++ b/build.sh
@@ -16,7 +16,7 @@ EOF
        fi
        if [ -f ${i}-packages-nr ]; then
            log "Begin ${SUB_STAGE_DIR}/${i}-packages-nr"
-           PACKAGES=`cat $i-packages-nr | tr '\n' ' '`
+           PACKAGES=`sed -e '/[[:space:]]*#.*$/d' < ${i}-packages-nr | tr '\n' ' '`
            if [ -n "$PACKAGES" ]; then
                on_chroot sh -e - << EOF
 apt-get install --no-install-recommends -y $PACKAGES
@@ -26,7 +26,7 @@ EOF
        fi
        if [ -f ${i}-packages ]; then
            log "Begin ${SUB_STAGE_DIR}/${i}-packages"
-           PACKAGES=`cat $i-packages | tr '\n' ' '`
+           PACKAGES=`sed -e '/[[:space:]]*#.*$/d' < ${i}-packages | tr '\n' ' '`
            if [ -n "$PACKAGES" ]; then
                on_chroot sh -e - << EOF
 apt-get install -y $PACKAGES

That will strip out anything starting with # on a line (and leading whitespace). This will allow stuff like:

# X.org compositing effects for LXDE
libgl1-mesa-dri libgles1-mesa libgles2-mesa xcompmgr

Scripts can explain what they're doing so that the process is understood by people looking at it later on. Right now, packages files cannot. They ought to, just so that they can explain WHY certain packages are being installed and perhaps sort packages into logical groups.

The alternative to doing this would be to have multiple parts of any given stage that install packages, each named for the bundle of packages being installed. That's semantically quite possible and quite useful, however it's pragmatically a mess simply because Debian has approximately 2^512 packages (only slightly exaggerated) in the database which must be processed every time you run apt-get.

Processing databases takes so long in Debianish systems nowadays that sid's version of aptitude now asks you after finishing an install if you want to quit immediately, without bothering to grind through the package list to regenerate the state graph for the UI before allowing you to do so. It's ugly and inelegant, but even on fast systems it's become kinda welcome.

The comment removal is fast and orders of magnitude faster than running multiple instances of apt-get when not necessary.

Probably worthwhile to make sed also do the tr stage above, but I didn't bother for comments on the suggestion. :)

Question: read-only root filesystem

I'm trying to modify pi-gen to create an image which mounts / and /boot read-only. This means, for example, removing resize2fs_once. In a nutshell, I'm trying to take the steps in this article and do them at generation time so that the system is read-only from the very first boot.

Has anyone tried this, or am I in new territory? I'm sure I'll get it working eventually, but I wanted to see if anyone has any experience here before I spend too much time hacking away at it.

Shrink the size of the image file

The size of the final image files seems to be determined in "export-image/prerun.sh" script file with the following command.
IMG_SIZE=$(expr $BOOT_SIZE \* 2 \+ $TOTAL_SIZE \+ 512)M

I understand the boot size and total size part in this command, but I'm not sure why an extra 512M space is needed.

For now, I'm trying to create a image file with size as small as possible. I know that a lite version image is created in stage2, however, its size still over 1GB. How can I shrink the size more?

Error when executing build.sh

I am constantly getting IMG_NAME not set and can't get build process to start.

What can be the possible cause for this error? I even checked local env settings.

Code is cloned directly from github.

The build date is always UTC, even with different time zone settings

This took me some time to find out.
My timezone - PDT - is UTC-7
So if I start a build and it fails after 5pm, the date will change...
That happens very often in the US West coast.
there should be a better strategy for this.

Force password change on first login

I tried to see if there was already a request for this, but saw none.
I do not know if this will impact a GUI environment, but in a lite environment, it would be perfect.

The passwd command has an option to force the password to be changed on the next login. Is it possible this can be toggled for a fresh image, to help mitigate compromised machines?

as root: passwd -e pi
or edit /etc/shadow, and change the 3rd field to a 0
Example: pi:xxx:0:0:99999:7:::

This, by default, will force the person logging in as pi to change the password, and will not allow the setting of the current password.

Missing dependencies

rsync and realpath are missing as dependencies in depends. I am trying to build inside a docker container pulled from ubuntu:trusty.

In addition, would the maintainers be interested in a PR for a docker-ized build system? It would be an addition in the form of a Makefile and Dockerfile only.

Build fails at stage2/01-sys-tweaks/03-console-setup.diff

My build on Ubuntu 14.04.4 LTS is failing with the following error:

Applying patch 03-console-setup.diff
patching file rootfs/etc/default/console-setup
Hunk #1 FAILED at 6.
1 out of 1 hunk FAILED -- rejects in file rootfs/etc/default/console-setup
Patch 03-console-setup.diff does not apply (enforce with -f)

It seems like the patch is expecting the following:

-CODESET="Lat15"
-FONTFACE="Fixed"
-FONTSIZE="8x16"

but the file to patch actually contains:

CODESET="guess"
FONTFACE="Fixed"
FONTSIZE="8x16"

As a part of the build process there was a graphical chooser for the codeset where the default was "guess", so I chose that. Manually editing work/rootfs/etc/default/console-setup fixes the issue and lets the build continue.

Exporting at Stage 1 fails

Trying to export "truly minimal system" of Stage 1 fails:

[18:24:14] Begin /path/to/export-image/04-finalise/01-run.sh
/bin/bash: line 1: /etc/init.d/fake-hwclock: No such file or directory
/bin/bash: line 2: hardlink: command not found

include usb-modeswitch in default install

This is required to get usb dongles internet devises like Tata Photon working.

http://www.tataphoton.com/photon-max-wifi.aspx its a very common method to get internet connection in India. It is included in desktop images of debian.

lightdm.conf patch fails at stage4

Using the 2016-09-23 release, the patch stage4/02-extras/00-patches/0-autologin.diff attempts to set lightdm for autologin-user=pi, but at least on my build, it fails because that has already been set at stage3.
Not sure what modified /etc/lightdm/lightdm.conf at stage3. The config file as delivered in lightdm_1.18.2-2_armhf.deb has autologin-user commented out as expected. Nothing obvious in the logs indicate that line being modified by another package install or by an explicit action of stage3.

Way to specify which stage to run to?

Had a brief look over build.sh and was wondering if there was a good way to execute the build script up to a certain stage (stage 2, to get just the lite system, for example).

[Suggestion] ${i}-run.sh must be .sh?

Noticed a few things looking at build.sh and scripts/common:

${i}-run-chroot (no suffix) is a thing that could be used, but never is even where it makes sense.
${i}-run.sh (.sh suffix) is used, and could be any language of course, though bash is obviously assumed.
on_chroot everywhere uses sh -e (dash) rather than bash -e.

While contemplating what if anything to do about any of the above, it crossed my mind: What if for some reason you wanted or needed to have a -run script that wasn't written in bash? How would that work?

Answer: Something like this:

#! /bin/bash

i=00

for run_script in ./${i}-run{.*,}; do
        if [ -x "$run_script" ]; then
                echo "Begin running ${run_script}"
                $run_script
                echo "End running ${run_script}"
        fi
done

Okay, let's try that out:

00-run.py: a /usr/bin/env python3 script, ASCII text executable
00-run.sh: Bourne-Again shell script, ASCII text executable
00-run:    a /bin/cat script, ASCII text executable

... /bin/cat!? Sure, why not.

Begin running ./00-run.py
./00-run.py has been run!
End running ./00-run.py
Begin running ./00-run.sh
./00-run.sh has been run!
End running ./00-run.sh
Begin running ./00-run
#! /bin/cat

I'm pretty useless, but I could exist and be anything at all...
End running ./00-run

At all wanted or cared about, or is this feature creep? I've got thoughts about how -run-chroot could be useful, if it were a file copied to the rootfs, run there, and then deleted, which would allow for more flexibility than an inline script. That's particularly useful to me for now since my Raspple II stage 5 presently uses -run to arbitrarily run random scripts from the chroot, though how that's done will change over time.

Repeated use of build-docker.sh eats disk space extremely fast

Each use of build-docker.sh leaves behind tens of GBs of files in /var/lib/docker which are never deleted, not even with docker rm pigen_work.

This eats up disk space extremely fast, wasting easily hundreds of GB, with no way to get rid of these files (except sudo rm -rf which might destroy integrity of docker).

Build fails at export-image/04-finalise/01-run.sh

Building with current git version fails:

$ cd pi-gen-dev
$ echo "IMG_NAME='MyRaspbian'" > config
$ ./build-docker.sh
...
[10:25:03] Begin /pi-gen/export-image/04-finalise
[10:25:03] Begin /pi-gen/export-image/04-finalise/01-run.sh
Stopping fake hwclock: saving system time.
Mode:     real
Files:    2257
Linked:   344 files
Compared: 0 xattrs
Compared: 590 files
Saved:    9.44 MiB
Duration: 0.11 seconds
fatal: Not a git repository (or any of the parent directories): .git
Done! Your image(s) should be in deploy/

And there is no image in deploy.

Build failed at stage 4

ownership of `/home/harshad/pi-gen-dev/work/2016-12-04-harry/stage4/rootfs/home/pi/python_games' retained as harshad:harshad

rsync: - - chown=1000:1000: unknown option
rsync error:syntax or usage error (code 1) at main.c(1453) [client=3.0.9]

What has gone wrong here?

resize2fs_once.service startup fails at boot

Noticed in the boot sequence for a freshly built image that resize2fs_once was failing to get started by systemd. Checked its status and found that resize2fs_once.service simply didn't exist. Running raspi-config and expanding the filesystem fixes it and the message doesn't appear again.

How do you use this tool?

Please do not default to using Google nameservers

See discussion here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=761658

Changing default language gives errors

Perhaps the locale package should be installed eralier in the process?

Set the locale to en_US.UTF-8 UTF-8

A number of errors appear for several packages before seeing "generating locale"

/usr/bin/locale: Cannot set LC_CTYPE to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_MESSAGES to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_ALL to default locale: No such file or directory

Suggestion to build.sh

This script adds a check to make sure the user is under root to run the script and adds all dependencies before starting the scripting process and if IMG_NAME is not set, will echo how to set IMG_NAME in the command.
build.txt

Including `certbot` from jessie-backports fails authentication

I'm trying to include certbot, the Let's Encrypt certificate client, in stage3. I have included the jessie-backports repository in sources.list, but when the script tries to install certbot, the following error occurs:

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  dialog python-acme python-certbot python-cffi-backend python-chardet python-configargparse python-configobj python-cryptography python-dialog python-dnspython python-enum34
  python-funcsigs python-idna python-ipaddress python-mock python-ndg-httpsclient python-openssl python-parsedatetime python-pbr python-pkg-resources python-psutil python-pyasn1
  python-pyicu python-requests python-rfc3339 python-setuptools python-six python-tz python-urllib3 python-zope.component python-zope.event python-zope.interface
Suggested packages:
  python-certbot-apache python-certbot-doc python-acme-doc python-configobj-doc python-cryptography-doc python-cryptography-vectors python-enum34-doc python-funcsigs-doc
  python-mock-doc python-openssl-doc python-openssl-dbg python-psutil-doc doc-base python-socks python-setuptools-doc python-ntlm
Recommended packages:
  letsencrypt
The following NEW packages will be installed:
  certbot dialog python-acme python-certbot python-cffi-backend python-chardet python-configargparse python-configobj python-cryptography python-dialog python-dnspython python-enum34
  python-funcsigs python-idna python-ipaddress python-mock python-ndg-httpsclient python-openssl python-parsedatetime python-pbr python-pkg-resources python-psutil python-pyasn1
  python-pyicu python-requests python-rfc3339 python-setuptools python-six python-tz python-urllib3 python-zope.component python-zope.event python-zope.interface
0 upgraded, 33 newly installed, 0 to remove and 46 not upgraded.
Need to get 2313 kB of archives.
After this operation, 11.1 MB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  python-cffi-backend python-pkg-resources python-configargparse python-six python-idna python-ipaddress python-pyasn1 python-setuptools python-cryptography python-dialog
  python-dnspython python-funcsigs python-openssl python-ndg-httpsclient python-pbr python-psutil python-urllib3 python-requests python-rfc3339 python-mock python-acme python-certbot
  certbot
E: There are problems and -y was used without --force-yes

I have tried adding the main Debian Jessie signing key to apt-key and running apt-key update but neither has worked so far.

Happy for any thoughts!

Question: Best practices to customise it?

Hi all,
the project doesn't seem to have discussion group, so I will ask here.

Given I am working on my own step and I sometimes make mistakes, is there a way to avoid running all previous steps and speed up the development process?
If I was to use a devops tool like Ansible or Chef, what would be the best approach?

Thanks,
Franklin

Remove dependency on kernel support for binfmt

In order to build for docker, the underlying kernel has to have support for binfmt, which is required by qemu-user-static. On Docker for mac, this works fantastically because the kernel already has the required support. However, when attempting to run on a fresh Debian VPS many kernels don't have this support.

I'm not sure if there's an easy solve. Many on the internet have suggested to use a version of qemu with the QEMU_EXECVE set so compiling for ARM doesn't escape out and cause the build to fail. A special docker image has been created that uses this prebuilt special version of qemu but I'm not sure how much use this is in this context.

Attempting to install qemu-user-static produces the following output:

Setting up qemu-user-static (1:2.1+dfsg-12+deb8u6) ...
update-binfmts: warning: Couldn't load the binfmt_misc module.
update-binfmts: warning: Couldn't load the binfmt_misc module.
...
update-binfmts: warning: Couldn't load the binfmt_misc module.
Setting up quilt (0.63-3) ...

And some people on circleci have documented similar behaviour as a failed result of cross-compilation for ARM architectures.

The actual error with running IMG_NAME=example ./build.sh occurs during the initial file extractions:

...
I: Extracting zlib1g...
I: Running command: chroot /root/pi-gen/work/2017-02-07-tiny/stage0/rootfs /debootstrap/debootstrap --second-stage
chroot: failed to run command '/debootstrap/debootstrap': Exec format error

While this isn't an issue inherent to pi-gen, it would be good to remove the dependency on underlying kernel support that isn't always present.

Trouble building when "pi-gen/work" dir is not a symlink after fix has been applied

Git commit 40b5af9 seems to be causing issues for me.

➜ ~ git clone https://github.com/RPi-Distro/pi-gen
Cloning into 'pi-gen'...
remote: Counting objects: 567, done.
remote: Compressing objects: 100% (61/61), done.
remote: Total 567 (delta 20), reused 0 (delta 0), pack-reused 506
Receiving objects: 100% (567/567), 641.79 KiB | 302.00 KiB/s, done.
Resolving deltas: 100% (193/193), done.
Checking connectivity... done.
➜ ~ cd pi-gen
➜ pi-gen git:(dev) sudo IMG_NAME="test" ./build.sh
mkdir: missing operand
Try 'mkdir --help' for more information.

The problem seems to be that WORK_DIR is now returning empty:

export WORK_DIR=`readlink -f "${BASE_DIR}/work/${IMG_DATE}-${IMG_NAME}"`

This was a fix that was added for symlink support, but is now interfering with my ability to run pi-gen without a symlinked work directory.