fspin-k8s / fspin-infrastructure Goto Github PK

Automatically clean up:

• repo
• build-results
• gce images

Determine how many of each makes the most sense.

The "Re-Spin All" job needs to be discrete for each spin.

The job gets stuck when downloading the checksums and it errors:

++ wget -q -O - https://repo.fspin.org/latest_snapshot
+ export SNAPSHOT_ID=2018-12-20-1545271236
+ SNAPSHOT_ID=2018-12-20-1545271236
+ [[ ! -n x ]]
++ awk -F- -v OFS=- '{print $1,$2,$3}'
++ echo 2018-12-20-1545271236
+ export SPIN_ID=2018-12-20
+ SPIN_ID=2018-12-20
++ echo 2018-12-20
++ sed s/-//g
+ export RELEASE_STRING=20181220
+ RELEASE_STRING=20181220
+ export RESULTS_LOCATION=gs://build-results.fspin.org/releases/
+ RESULTS_LOCATION=gs://build-results.fspin.org/releases/
+ export TMP_LOCATION=gs://build-results.fspin.org/2018-12-20/
+ TMP_LOCATION=gs://build-results.fspin.org/2018-12-20/
+ export NUM_RELEASES=2
+ NUM_RELEASES=2
+ mkdir -p 2018-12-20
+ envsubst '${SPIN_ID}'
/opt/2018-12-20 /opt
Downloading checksums for 2018-12-20...
+ echo 'export SPIN_ID=2018-12-20'
+ pushd 2018-12-20
+ echo 'Downloading checksums for 2018-12-20...'
+ gsutil -m cp -c 'gs://build-results.fspin.org/live/*/CHECKSUM512-*20181220*' 'gs://build-results.fspin.org/source/*/CHECKSUM512-*20181220*' .
CommandException: No URLs matched: gs://build-results.fspin.org/live/*/CHECKSUM512-*20181220*
CommandException: No URLs matched: gs://build-results.fspin.org/source/*/CHECKSUM512-*20181220*
CommandException: 2 files/objects could not be transferred.

Create a standalone python 3 script to enumerate and download a published release for use on destination nodes.

If the packer task fails, the job does not report as failure.

For all jobs, return full logs for viewing via Jenkins.

#10 is resolved in a mock based compose.

Create jobs dynamically in the Jenkins pipeline vs checking into the repo.

We don't need some stuff that is in the repo, don't sync it.

This feature somehow broke when making the publish job properly fail if no build was available to publish.

For some reason, F28 upstream image import broke. Fix it.

• Add to import base image
• Add to repo

It seems that using a node type other than n1-standard-1 causes the jenkins agent to get oom killed by k8s. This seems to be a mismatch between what java decides is it's heap and what the cluster is going to allow.

https://dzone.com/articles/why-my-java-application-is-oomkilled

Automatically update firewall policy to only allow connections to ssh from the nodes in the cluster.

Get all of the supported languages working in the installer.

The publisher should combine everything the builders created into a single location and push to GCS where we consider the results to be RC. A maximum number of RCs should be managed by the publisher.

Added information:

• SHA512 generation
• Torrent generation
• README generation

Mirrors will then be able to pull from this location easily on an automated schedule.

create-snapshot failed (manually destroyed the gce that packer created) yet pipeline reported success.

Update the publish job to keep two releases around.

The official image is not importing due to a kernel issue initing on a cloud provider due to needing more entropy. The 4.16.6+ fedora kernels reverted the changes that caused this. Build a suitable image for import that includes a compatible kernel.

Present pungi node is a n1-standard16 (16 cpus and 60gb ram) which when running causes it to be a target for starving for other nodes to hit peak performance/throughput.

This proposed reduction would allow things on ALL nodes to be more streamlined and reduce the chance and impact of such starvation of extra resources.

Pulling down the snippet at runtime causes non-reproducible results and is a pain to develop.

In prep for f28, make all usage of the branch version variables that can be overridden at runtime.

If the releases path is missing, the publish job freaks out.

On the old infra our torrent script would create a series of torrents (one per ISO, minus source),

Using the k8s infra and then attempting to create the torrents from the non-k8s infra where final images are served from the path variations make for a failure of torrent generation.

The k8s infra should add as an operation of the publisher job the following (with relevant path changes)

repodir=/srv/Livecds/$2
for spin in CINN KDE LXDE LXQT MATE SOAS WORK XFCE; do
#for spin in CINN LXDE LXQT WORK SOAS XFCE; do
#for spin in SECURITY ; do 
#for spin in SOAS ; do
hash=$(grep $spin $repodir/CHECKSUM512-$2)
    transmission-create -c "ISO SHA512SUM: ${hash} " -s 2048 -p -t http://respins.fedorainfracloud.org:6969/announce -o $repodir/F$1-${spin}-x86_64-$2.torrent $repodir/F$1-${spin}-x86_64-$2.iso
    chmod +r $repodir/*.torrent

    echo "update opentracker whitelist for ${spin}"
    transmission-show $repodir/F$1-${spin}-x86_64-$2.torrent | awk '/Hash/{print $2 " - "}' >> $repodir/buildfile
    echo F$1-${spin}-x86_64-$2.iso >> $repodir/buildfile

These torrents and thier hashes (which are what presently fails when created on non k8s infra) are what is feed to the SIGs whitelisted tracker.

Setup a way to use BFO techniques to boot official images. Beta test for boot.fp.o but also setup infra for live booting spins.

Add more details to Jenkins jobs that describe what happens. Possibly order the jobs. Possibly create a master job that runs the other jobs in the right order.

Make build-results human and machine accessible.

Update the downstream jobs to not create a builder or spin for F27. Onward.

Right now the chart names pod with default. Contribute upstream to make that better.

https://github.com/kubernetes/charts/blob/master/stable/jenkins/templates/config.yaml#L35

Decrease build time by using locally attached SSD vs networked SSD to increase I/O performance.

Script out rebuilding TX and DE to be http hosting and automatic torrent seeding on much smaller nodes.

Update the name of the spins to match the following:

• Use an uppercase 4 char short name from the source target (i.e. WORK)
• Use a date vs epoch for resulting images (i.e. 2018-04-29)
• Update source ISO name

templates/f27-x86-64-template needs to be updated for live images.
pungi-create-install/fspin-27-x86-64-pungi needs to be update for source images.

The publish job needs to be able to fail some commands but not fail the job. Re-do the control flow.

Update the publisher to sign the CHECKSUM512 file with a project gpg key. This needs research so the security model is decent.