OSTree is a tool for atomic full file system upgrades with rollback capability. OSTree has several advantages over traditional dual-bank systems, but the most important one is that it minimizes network bandwidth and data storage footprint by sharing files with the same contents across file system deployments.
Aktualizr (and RVI SOTA client) add authentication and provisioning capabilities to OTA and are integrated with OSTree. You can connect with these open-source applications or sign up for a free account at HERE OTA Connect to get started.
If you don’t already have a Yocto project that you want to add OTA to, you can use the HERE OTA Connect Quickstart project to rapidly get up and running on a Raspberry Pi. It takes a standard poky distribution, and adds OTA and OSTree capabilities.
If you already have a Yocto-based project and you want to add atomic filesystem updates to it, you just need to do three things:
-
Clone the
meta-updater
layer and add it to your bblayers.conf. -
Clone BSP integration layer (
meta-updater-${PLATFORM}
, e.g. meta-updater-raspberrypi) and add it to yourconf/bblayers.conf
. If your board isn’t supported yet, you could write a BSP integration for it yourself. See the Adding support for your board section for the details. -
Set up your distro. If you are using "poky", the default distro in Yocto, you can change it in your
conf/local.conf
to "poky-sota". Alternatively, if you are using your own or third party distro configuration, you can addINHERIT += " sota"
to it, thus combining capabilities of your distro with meta-updater features.
You can then build your image as usual, with bitbake. After building the root file system, bitbake will then create an OSTree-enabled version of it, commit it to your local OSTree repo and (optionally) push it to a remote server. Additionally, a live disk image will be created (normally named ${IMAGE_NAME}.-sdimg-ota
e.g. core-image-raspberrypi3.rpi-sdimg-ota
). You can control this behaviour through variables in your local.conf.
With AGL you can just add agl-sota feature while configuring your build environment:
source meta-agl/scripts/aglsetup.sh -m porter agl-demo agl-appfw-smack agl-devel agl-sota
You can then run:
bitbake agl-demo-platform
and get as a result an ostree_repo
folder in your images directory (tmp/deploy/images/${MACHINE}/ostree_repo
). It will contain:
-
your OSTree repository, with the rootfs committed as an OSTree deployment,
-
an
ota-ext4
bootstrap image, which is an OSTree physical sysroot as a burnable filesystem image, and optionally -
some machine-dependent live images (e.g.
.wic
for Raspberry Pi or.porter-sdimg-ota
Renesas Porter board).
Although aglsetup.sh
hooks provide reasonable defaults for SOTA-related variables, you may want to tune some of them.
Currently supported platforms are
If your board isn’t supported yet, you can add board integration code yourself. The main purpose of this code is to provide a bootloader that will be able to use OSTree’s boot directory. In the meta-updater integration layers we have written so far, the basic steps are:
-
Make the board boot into U-Boot
-
Make U-boot import variables from /boot/loader/uEnv.txt and load the kernel with initramfs and kernel command line arguments according to what is set in this file.
You may take a look into Minnowboard or Raspberry Pi integration layers for examples.
Although we have used U-Boot so far, other boot loaders can be configured work with OSTree as well.
Your images will also need network connectivity to be able to reach an actual OTA backend. Our 'poky-sota' distribution does not mandate or install a default network manager but our supported platforms use the virtual/network-configuration
recipe, which can be used as a starting example.
-
OSTREE_REPO
- path to your OSTree repository. Defaults to${DEPLOY_DIR_IMAGE}/ostree_repo
-
OSTREE_OSNAME
- OS deployment name on your target device. For more information about deployments and osnames see the OSTree documentation. Defaults to "poky". -
OSTREE_COMMIT_BODY
- Message attached to OSTree commit. Empty by default. -
OSTREE_COMMIT_SUBJECT
- Commit subject used by OSTree. Defaults toCommit-id: ${IMAGE_NAME}
-
OSTREE_UPDATE_SUMMARY
- Set this to '1' to update summary of OSTree repository on each commit. '0' by default. -
OSTREE_DEPLOY_DEVICETREE
- Set this to '1' to include devicetree(s) to boot -
GARAGE_SIGN_AUTOVERSION
- Set this to '1' to automatically fetch the last version of the garage tools installed by the aktualizr-native. Otherwise use the fixed version specified in the recipe. -
INITRAMFS_IMAGE
- initramfs/initrd image that is used as a proxy while booting into OSTree deployment. Do not change this setting unless you are sure that your initramfs can serve as such a proxy. -
SOTA_PACKED_CREDENTIALS
- when set, your ostree commit will be pushed to a remote repo as a bitbake step. This should be the path to a zipped credentials file in the format accepted by garage-push. -
SOTA_DEPLOY_CREDENTIALS
- when set to '1' (default value), deploys credentials to the built image. Override it inlocal.conf
to built a generic image that can be provisioned manually after the build. -
SOTA_CLIENT_PROV
- which provisioning method to use. Valid options areaktualizr-auto-prov
,aktualizr-ca-implicit-prov
, andaktualizr-hsm-prov
. For more information on these provisioning methods, see the OTA Connect documentation. The default isaktualizr-auto-prov
. This can also be set to an empty string to avoid using a provisioning recipe. -
SOTA_CLIENT_FEATURES
- extensions to aktualizr. The only valid options arehsm
(to build with HSM support) andsecondary-network
(to set up a simulated 'in-vehicle' network with support for a primary node with a DHCP server and a secondary node with a DHCP client). -
SOTA_SECONDARY_CONFIG_DIR
- a directory containing JSON configuration files for virtual secondaries on the host. These will be installed into/etc/sota/ecus
on the device and automatically provided to aktualizr. -
SOTA_HARDWARE_ID
- a custom hardware ID that will be written to the aktualizr config. Defaults to MACHINE if not set. -
SOTA_MAIN_DTB
- base device tree to use with the kernel. Used together with FIT images. You can change it, and the device tree will also be changed after the update. -
SOTA_DT_OVERLAYS
- whitespace-separated list of used device tree overlays for FIT image. This list is OSTree-updateable as well. -
SOTA_EXTRA_CONF_FRAGS
- extra configuration fragments for FIT image. -
RESOURCE_xxx_pn-aktualizr
- controls maximum resource usage of the aktualizr service, whenaktualizr-resource-control
is installed on the image. See aktualizr service resource control for details. -
SOTA_POLLING_SEC
- sets polling interval for aktualizr to check for updates if aktualizr-polling-sec is included in the image.
OSTree used to include a simple HTTP server as part of the ostree binary, but this has been removed in more recent versions. However, OSTree repositories are self-contained directories, and can be trivially served over the network using any HTTP server. For example, you could use Python’s SimpleHTTPServer:
cd tmp/deploy/images/qemux86-64/ostree_repo python -m SimpleHTTPServer <port> # port defaults to 8000
You can then run ostree from inside your device by adding your repo:
# This behaves like adding a Git remote; you can name it anything ostree remote add --no-gpg-verify my-remote http://<your-ip>:<port> # If OSTREE_BRANCHNAME is set in local.conf, that will be the name of the # branch. If not set, it defaults to the value of MACHINE (e.g. qemux86-64). ostree pull my-remote <branch> # poky is the OS name as set in OSTREE_OSNAME ostree admin deploy --os=poky my-remote:<branch>
After restarting, you will boot into the newly deployed OS image.
For example, on the raspberry pi you can try this sequence:
# add remote ostree remote add --no-gpg-verify agl-snapshot https://download.automotivelinux.org/AGL/snapshots/master/latest/raspberrypi3/deploy/images/raspberrypi3/ostree_repo/ agl-ota # pull ostree pull agl-snapshot agl-ota # deploy ostree admin deploy --os=agl agl-snapshot:agl-ota
The aktualizr repo contains a tool, garage-push, which lets you push the changes in OSTree repository generated by bitbake process. It communicates with an http server capable of querying files with HEAD requests and uploading them with POST requests. In particular, this can be used with HERE OTA Connect. garage-push is used as follows:
garage-push --repo=/path/to/ostree-repo --ref=mybranch --credentials=/path/to/credentials.zip
You can set SOTA_PACKED_CREDENTIALS
in your local.conf
to automatically synchronize your build results with a remote server. Credentials are stored in an archive as described in the aktualizr documentation.
Aktualizr supports a variety of configuration options via a configuration file and the command line. There are two primary ways to control aktualizr’s configuration from meta-updater.
First, you can set SOTA_CLIENT_PROV
to control which provisioning recipe is used. Each recipe installs an appropriate sota.toml
file from aktualizr according to the provisioning needs. See the SOTA-related variables in local.conf section for more information.
Second, you can write recipes to install additional config files with customized options. A few recipes already exist to address common needs and provide an example:
-
aktualizr-auto-reboot.bb configures aktualizr to automatically reboot after new updates are installed in order to apply the updates immediately. This is only relevant for package managers (such as OSTree) that require a reboot to complete the installation process. If this is not enabled, you will need to reboot the system through other means.
-
aktualizr-disable-send-ip.bb disables the reporting of networking information to the server. This is enabled by default and supported by HERE OTA Connect. However, if you are using a different server that does not support this feature, you may want to disable it in aktualizr.
-
aktualizr-log-debug.bb sets the log level of aktualizr to 0 (trace). The default is 2 (info). This recipe is intended for development and debugging purposes.
To use these recipes, you will need to add them to your image with a line such as IMAGE_INSTALL_append = " aktualizr-log-debug "
in your local.conf
.
With systemd based images, it is possible to set resource policies for the aktualizr service. The main use case is to provide a safeguard against resource exhaustion during an unforeseen failure scenario.
To enable it, install aktualizr-resource-control
on the target image and optionally override the default resource limits set in aktualizr_git.bb, from your local.conf
.
For example:
IMAGE_INSTALL_append += " aktualizr-resource-control " RESOURCE_CPU_WEIGHT_pn-aktualizr = "50"
There are a few settings that can be controlled in local.conf
to simplify the development process:
Option | Effect |
---|---|
|
Build the latest head (by default, using the master branch) of Aktualizr |
|
Build |
|
Build the specified revision of Aktualizr. Note that both of these need to be set. This can be used in conjunction with |
|
Use with |
Warning: overriding target version is a dangerous operation, make sure you understand this section completely before doing it.
Every time you build an image with SOTA_PACKED_CREDENTIALS
set, a new entry in your Uptane metadata is created and you can see it in the OTA Garage UI if you’re using one. Normally this version will be equal to OSTree hash of your root file system. If you want it to be different though you can override is using one of two methods:
-
Set
GARAGE_TARGET_VERSION
variable in yourlocal.conf
. -
Write a recipe or a bbclass to write the desired version to
${STAGING_DATADIR_NATIVE}/target_version
. An example of such bbclass can be found inclasses/target_version_example.bbclass
.
Please note that [target name, target version] pairs are expected to be unique in the system. If you build a new target with the same target version as a previously built one, the old package will be overwritten on the update server. It can have unpredictable effect on devices that have this version installed, and it is not guaranteed that information will be reported correctly for such devices or that you will be able to update them (we’re doing our best though). The easiest way to avoid problems is to make sure that your overriding version is as unique as an OSTree commit hash.
This layer relies on the test framework oe-selftest for quality assurance. Currently, you will need to run this in a build directory with MACHINE
set to qemux86-64
. Follow the steps below to run the tests:
-
Append the line below to
conf/local.conf
to disable the warning about supported operating systems:SANITY_TESTED_DISTROS = ""
-
If your image does not already include an ssh daemon such as dropbear or openssh, add this line to
conf/local.conf
as well:IMAGE_INSTALL_append = " dropbear "
-
Some tests require that
SOTA_PACKED_CREDENTIALS
is set in yourconf/local.conf
. See the SOTA-related variables in local.conf section. -
To be able to build an image for the grub tests, you will need to install TianoCore’s ovmf package on your host system. On Debian-like systems, you can do so with this command:
sudo apt install ovmf
-
Run oe-selftest:
oe-selftest -r updater_native updater_qemux86_64 updater_minnowboard updater_raspberrypi updater_qemux86_64_ptest
For more information about oe-selftest, including details about how to run individual test modules or classes, please refer to the Yocto Project wiki.
The meta-updater layer includes support for running parts of the aktualizr test suite on deployed devices through Yocto’s ptest functionality. Since it adds significant build time cost, it is currently disabled by default. To enable it, add the following to your conf/local.conf
:
PTEST_ENABLED_pn-aktualizr = "1"
IMAGE_INSTALL_append += " aktualizr-ptest ptest-runner "
Be aware that it will add several hundreds of MB to the generated file system.
The aktualizr tests will now be part of the deployed ptest suite, which can be run by calling ptest-runner
. Alternatively, the required files and run script can be found in /usr/lib/aktualizr/ptest
.
As described in SOTA-related variables in local.conf section you can set SOTA_DEPLOY_CREDENTIALS
to 0
to prevent deploying credentials to the built wic
image. In this case you get a generic image that you can use e.g. on a production line to flash a series of devices. The cost of this approach is that this image is half-baked and should be provisioned before it can connect to the backend.
Provisioning procedure depends on your provisioning recipe, i.e. the value of SOTA_CLIENT_PROV
(equal to aktualizr-auto-prov
by default):
-
For
aktualizr-auto-prov
put yourcredentials.zip
to/var/sota/sota_provisioning_credentials.zip
on the filesystem of a running device. If you have the filesystem of our device mounted to your build machine, prefix all paths with/ostree/deploy/poky
as in/ostree/deploy/poky/var/sota/sota_provisioning_credentials.zip
. -
For
aktualizr-ca-implicit-prov
-
put URL to the backend server (together with protocol prefix and port number) at
/var/sota/gateway.url
. If you’re using HERE OTA Connect, you can find the URL in theautoprov.url
file in your credentials archive. -
put client certificate, private key and root CA certificate (for the server, not for the device) at
/var/sota/import/client.pem
,/var/sota/import/pkey.pem
and/var/sota/import/root.crt
respectively.
-
-
For
aktualizr-hsm-prov
-
put URL to the server backend (together with protocol prefix and port number) at
/var/sota/gateway.url
. If you’re using HERE OTA Connect, you can find the URL in theautoprov.url
file in your credentials archive. -
put root CA certificate (for the server, not for the device) at
/var/sota/import/root.crt
. -
put client certificate and private key to slots 1 and 2 of the PKCS#11-compatible device.
-