If the sign_image function is defined, the image certificate needs to be placed in the staged config folder alongside the Swupd_Root.pem.

Content needs to be aware of all inputs to mixer. This mean packages AND bundles. Because content.sh has a well-defined interface with content providers, need to create a bundle provider interface and implement this. This bundle provider interface should be able to clone bundles from a parameterized location to a defined location that will be pulled from in the mix. Whether the mix is an upstream or a downstream will determine if the mixing process also attempt to pull bundles from an external source during the build process.

When starting a Jenkins job, or when running scripts via the Makefile, want to allow the user to override MIX_VERSION and DISTRO_LATEST variables to their choosing. This gives flexibility on how to start a build, which is useful for when issues arise.

If an image fails to generate, rather than catching the update in the next build of clear, want to have the option of selectively attempting to regenerate images again. This may come in the form of allowing re-execution through Jenkins pipeline, or maybe we simply put clr-installer into a loop attempting to build the image multiple times (no more than a total of 2 attempts maybe)?

Want to consider using a run_and_log approach for these commands because the console output is too large to go through most times. For mixer and clr-installer, we want to archive the logs in the Jenkins job so they may be referenced in the future.

Have a flag for the release workflow that would allow a format bump to be triggered by necessity of the Downstream itself instead of only tracking Upstream's format bumps.

This is required to allow for deletion of bundles that are exclusive to a given Downstream.

make release
make: *** No rule to make target 'prologue', needed by 'release'. Stop.

A file named os-core-update-extra-files needs to be created in mixer's server state directory containing the paths that need to be added to the os-core-update manifest. These files need to be created with the proper permissions inside mixer's full chroot. This needs done after mixer build bundles but before mixer build update. Without this, users are not able to perform swupd update from the content that mixer creates when custom update signing is used.

At a minimum, need the following:

/usr/share/clear/update-ca
/usr/share/clear/update-ca/Swupd_Root.pem

This is because swupd-client has an option in it's configure.ac for:

default_cert_path="/usr/share/clear/update-ca/Swupd_Root.pem"

Some operations for building Clear Linux vary depending on whether the build is an upstream or a downstream. Create this flag in the config repository and use this when calling to mixer to properly create update content.

To start, the --offline flags should be applied to mixer if an upstream.

Add header to license file that contains the following text to make it more clear as to what users are looking at:

# This list is automatically generated.
#
# To read the full license text for these licenses, please visit
# https://spdx.org/licenses. A few license identifiers in this list
# are not declared on the https://spdx.org/licenses website:
# - Copyright
# - Distributable
# - Public-Domain
#
# These are used for projects that have explicitly granted redistribution
# of the project source code, but don't have a typical OSI-approved
# license identifier.

A static files repository should be a variable in the config repo. This repository is expected to have the following directory structure:

• toplevel
• releases

That controls which files get placed in the toplevel directory and which files get places in the directory for each release.

When using the --no-signing option on mixer, mixer will not sign the latest file. This is not being done today by our process and is required for supporting future enhancements to swupd. The first file is not used by mixer or swupd, but supports creating delta packs from the first build in every format, and also supports MCA. This file should also be signed.

• Consider the pros and cons of having the orchestration tool (usually Jenkins) to also fetch the config repository
  • One big con would be the increased reliance on the tool itself
• Double-check Namespace vs Dstream Name precedence
  • For workspace, stagging and publishing, namespace is to be preferred
  • For content and version URL dstream_name is preferred
    • Than a symbolic link from namespace -> dstream_name is expected on the webserver
    • This would allow us to switch a downstream publish content very easily in case of problem with a given namespace release workflow
    • This does allow developers workflows to reuse a 'workflow_config' but "breaks" swupd (fixable with -u) for their streams
    • Today's approach have swupd looking directly to developers streams

Update our CI files to github actions that is performing much better.

This should be done in a separate pipeline running on an agent different than the build agent. Do not want debuginfo generation to be in the critical path of the build because it is optional for non-developers who are just running or hosting applications.

Start with the script from https://github.com/clearlinux/clr-debug-info in scripts/clr_debug_prepare along with the following code, replacing/removing undefined variables as necessary:

RAW=${DEBUGINFODIR}.raw

refresh_clr-debug-info() {
	local "${@}"
	TYPE=projects
	PROJ=clr-debug-info
	if [[ -d "${BUILDDIR}/${TYPE}/${PROJ}" ]] ; then
        	sudo rm -rf "${BUILDDIR}/${TYPE}/${PROJ}/"
        fi
        git clone "${GITHOST}:${TYPE}/${PROJ}" "${BUILDDIR}/${TYPE}/${PROJ}"
        
	PROJDIR=${REPO_BASE_DIR}/${PROJ}
        pushd ${PROJDIR}

        # clr-debug-info HEAD moves forward at will, but releases are tagged 
        # with one, two , or three didgets if it ever reaches release 100.
        TAG=$(git tag -l | grep -E "^[0-9]{1,3}$" | sort -n | tail -1)
        git checkout ${TAG}     || error "git special checkout failed"

        autoreconf -fiv
        if [[ $? -ne 0 ]]; then
                error "autoreconf -fiv failed"
        fi
        ${PROJDIR}/configure --prefix=${BUILDDIR}/usr
        if [[ $? -ne 0 ]]; then
                error "${PROJDIR}/configure ... failed"
        fi
        make
        sudo make install
        if [[ $? -ne 0 ]]; then
                error "make install failed"
        fi

        popd
}


create_debug() {
	local "${@}"
	log "Creating debug info for ${VER} in $0"
	#Begin debug data creation ----------------------
	set +x
	mkdir -p ${RAW}/done
	pushd ${RAW}
	DCNT=0
	for i in ${RELEASEDIR}/${VER}/clear/x86_64/debug/*.rpm ; do
        	base=`basename $i`;
        	if [[ ! -e ${RAW}/done/$base ]] ; then
			echo "Extracting ${base} ..."
			# Extract to a temporary staging dir first
			TMP="$(mktemp -d)"
			sudo rpm2cpio "${i}" | ( cd "${TMP}"; sudo cpio -idu || : )
			# Packages are now shipping debuginfo under
			# /usr/share/debug/ instead of /usr/{lib,src}/debug/.
			# Move back to /usr/{lib,src}/debug for compatibility.
			sudo mkdir -p "${TMP}"/usr/lib/debug
			sudo mkdir -p "${TMP}"/usr/src
			sudo find "${TMP}"/usr/share/debug/ -mindepth 1 -maxdepth 1 -type d -regextype awk -regex '.*/(.build-id|boot|lib|sbin|usr)$' | xargs -r -I '{}' sudo mv -v '{}' "${TMP}"/usr/lib/debug/
			sudo find "${TMP}"/usr/share/debug/ -mindepth 1 -maxdepth 1 -type d -name 'src' | xargs -r -I '{}' sudo mv -v '{}' "${TMP}"/usr/src/debug
			sudo rm -rf "${TMP}"/usr/share
			# Then copy only changed content into debuginfo.raw.
			# Symlinks are copied verbatim, since new ones are
			# needed whenever build IDs change, and some of the
			# symlinks are expected to be broken (so we can't use
			# the -L option) ...
			sudo rsync --info=stats2 -i -rlc "${TMP}"/ "${RAW}"/
			sudo rm -rf "${TMP}"
                	touch ${RAW}/done/$base
        	fi
	done
	popd
	set -x

	time nice sudo -E bash -x ${BUILDDIR}/usr/bin/clr_debug_prepare /home/releases/debuginfo.raw /home/releases/debuginfo || error "Unable to run clr_debug_prepare"

	#Validate files updated
	if [[ $(find ${RAW}/ -type f -atime 0 -amin -6 |wc -l) -eq 0 ]] ; then 
		log "ERROR: No new files in ${RAW}, debug is not working correctly!"
	fi
	
	if [[ $(find ${DEBUGINFODIR}/ -type f -atime 0 -amin -6 |wc -l) -eq 0 ]] ; then 
		log "ERROR: No new files in ${DEBUGINFODIR}, debug is not working correctly!"
	fi
	
	log "Finished creating debug info in $0"
}


main () {
	local "${@}"

	refresh_clr-debug-info ${@} 

        create_debug ${@}
}

main ${@}

Prior to doing releases of tools on which the clr-distro-factory build process is dependent, those tools should be tested in this environment to identify any changes needed to the build process to support their release.

Given clr-distro-factory enables reproducible testing in namespaced environments on the same machine (or even running on a separate machine all together), we want to provide a test plan for these tools that if their updated versions allow for the build process to complete successfully, they may be released.

Given the integration with GitHub we have with clr-distro-factory now, I'm not sure if this is something that may be done as Travis checks on our dependent tools. I'm thinking the more automation, the better since we want to always make sure this is tested.

There are one or two find command that become expensive as debuginfo build up over time. If we know the packages from the previous build, we might be able to identify which debuginfo packages have changed and generate debuginfo only for these in a staging directory, and then merge it into the final directory.

When working in a full DevOps environment, a git repository projects/common will be used. This contains a packages file that keeps track of the current packages for the distro. This needs updated as a part of every build to ensure that when developers do a make pull/make clone from their tooling, they are able to automatically clone new packages. This should be done as a part of the publishing step, and be done if a config repo for the URL to this repository is set to allow operation without a full DevOps setup.

Most use cases will have some sort of QA, so the breakpoint would be required. This don't need to be in replacement to the workflow definition we have today. We can keep both options and user can pick the one.

content.sh should be able to look at a directory and add a rpm repository for each folder in the directory to mixer. This allows users to configure any number of rpm repositories to be used as sources to support more complex downstream use cases.

Today, it can be done on Distro-Factory side by handling the bundles repository to fit mixer needs. But the most correct approach is for mixer to support the same layout used by mainline (upstream) Clear Linux so we don't have to move files for nothing.

To allow for one to maintain a Downstream that is just a subset of Upstream.

I started from 27000 (format 26), and 27190 (format 26) ,and 27200 (format 26), all successful.
But It failed like when I tried 27210 (format 27).
My clear version on the build machine is 29280.
The failure log as below.

== Build 'Update' Content ==
The upstream version for this build (27210) is outside the format range of your last mix (format 26, upstream versions 25770 to 27200). This build cannot be done until you complete an upstream format build. Please run the following command to complete the format bump:
mixer build upstream-format
Once this has completed you can re-run this build you are attempting to create.

• Please note that if you are multiple format bumps behind, mixer will detect you are still behind and re-run the command until the proper format is reached.
  Error: ERROR: Format bump required

Content needs to be able to be staged on a remote machine. Rather than just moving content into a new location, it should be rsync'd to a path. This path may be local, or it may be another machine. This allows for more complex infrastructure setups and requirements when dealing with larger amounts of data that need mirrored as a part of the publishing process.

Delta packs and delta manifests (from here on 'deltas' refers to both packs and manifests) should be generated for the past 10 builds to give users quick updates from releases over the past week or two.

mixer build delta-packs --previous-versions 10
mixer build delta-manifests --previous-versions 10

Additionally, for systems that are on a very old version and need to update across format bumps, need to always generate deltas from the first build in the current format to help users get to the current build faster.

mixer build delta-packs --from ${FIRST_BUILD}
mixer build delta-manifests --from ${FIRST_BUILD}

As a part of release note generation, bundle changes for the local bundles added to the mix should be shown. This means showing added, updated, and removed bundles. For now, this should only apply to the bundles for the particular distribution. If a downstream, it should not show changes for upstream bundles. While an improvement for the future, these may be resolved by looking at the release notes of the upstream on which the distribution is bases.

The trick part is how to nicely populate the config.sh vars from the template. It is way easier to just dump a new config file from scratch.

In the other hand, making it from scratch makes trick to also provide an image template (or more than one).

Maybe we can clone from clr-distro-factory-config and fully replace 'config.sh' so we get the image template files but don't have a headache trying to populating the template with variable values.

A docker folder should be made in the config repository containing folders for each image name. Each image folder name should have the following files inside of it:

• Dockerfile
• bundles (contains one bundle name per line in the file)
• contact (one line in the format ",," without quotes

This data should be used as input to a process that builds a Docker image from scratch. This process should use the following swupd command to create the build context:

sudo -E swupd os-install --wait-for-scripts --path="$CHROOT_DIR" --version="$VER" --statedir="$STATE_DIR" --format="$(< "$SWUPDWEBDIR/$VER/format")" --url=file://"$SWUPDWEBDIR" --no-boot-update --bundles "$(cat "$TYPE_BUNDLE_FILE" | tr '\n' ',' | sed 's/,$//')"

Improve the way we use and maintain our workspace to avoid the extreme duplication of artifacts on the disk that we have today.

Also, improve the code in such a way that a "reusable" (BUILD_DIR) workspace isn't required while keep mixer to reuse the required artifacts (swupd key pair and update stream for the deltas) between builds.

This will have to be split in sub tasks, because the final solution isn't possible with current mixer requirements in place.

SimpleHTTPServer module has been merged into http.server in Python 3.0 which supports by Clear by default. I create a pull #30, please help to review :-)

SimpleHTTPServer was deprecated in favor of module http.server in Python 3.

Today when packages are published, all ~85GB of content is copied and synced for each build. Over time, this accumulates a high amount of disk space and requires a robust hardlinking strategy for proper management. Only new packages that have been added from the last build should be synced out to a common location, a cache, and existing packages should remain in that cache.

In order to create a wider array of images with special requirements, need to generalize post processing of image files. Rather than making assumptions about compression, make this step be a part of a finalize script. The finalize script may also be responsible for running qemu img-convert on the resulting image to prepare it for publishing into the marketplace of various cloud providers. Or it may post-process the content in the chroot to create a PXE image. Any number of changes are possible and can be associated with the config of the image. This allows images to be easily created in a standalone manner with the clr-installer outside of clr-distro-factory and aligns to an issue open with clr-installer: clearlinux/clr-installer#531. Once clr-installer closes this issue, this code will need removed from clr-distro-factory, but should be maintained until then.

Variables for these guys needs to be defined, but if they exist, they should be staged into the following directory structure:

repo/
    x86_64/
        os/
            Packages/
            repodata/
        debug/*
            repodata/
    source/*
            SRPMS/
                repodata/

* Indicates missing directories

Check the version of required tools meet the minimal requirements to work with a given clr-distro-factory release

You can use "mixer repo add " command to add new repos.
Add the repo url with file:// , http or https protocol

You can remove the "clear" repo from repo list if not needed using "mixer repo remove " command.
But "local" repo is auto added if removed with default url file:///local-yum.

You can set priorities to repos by editing ".yum-mix.conf file". We don't have mixer commands to set priority yet.

• Do not add all upstream bundles if an upstream in build_bundles function
• In the Building section, do not automatically perform/check for a format bump if an upstream.
• If an upstream, do not log what the mix is based on
• Make sure $clear_ver and $mix_ver are the same value if an upstream
• Consolidate usage of the Clear repo and the content repo if an upstream. Only one should be active.
• The --offline flag should be used everywhere

To help with conditionally performing work, create an IS_UPSTREAM flag in the config repository and key off this value.