Build automation (RPM+ISO) about tigeros HOT 45 OPEN

@jflory7 tbh, git+jenkins/ci+bash (aka, what we have now) seems like the best option

from tigeros.

Maybe rework GH repo to funtion similar to the fedora lookaside cache

from tigeros.

I can help with this, hit me up another time and we can brainstorm

from tigeros.

From discussion:

1. Someone commits to GitHub
2. Pull on the build box (using git pull)
3. Make srpm + rpm
4. Copy to /var/builds-to-sign or somewhere that has "chmod g+w" and "chown build-signers:build-signers" w/ people allowed to sign in that group
5. (long-term) send a message via email and/or IRC to notify things need signing
6. Things get signed manually
7. Run after-signing script which does the following:
8. Copy items to mirrors folders
9. (long-term) Git commit to the GitHub repo w/ a RITlug-BuildBox account
10. (long-term) Post a message via email and/or IRC announcing new items (tigeros-announce?)

from tigeros.

@ct-martin also assigning you though I can't since you aren't on the repo yet. poke me if you want to be added and I will discuss with team and eboard.

from tigeros.

Here are a few option:
Self hosted COPR
Bash scripts and git hooks
mock-scm config options to make a tarball

from tigeros.

Self hosted COPR

For what it's worth, I'm 👎 to a self-hosted COPR because it will be difficult for someone to maintain later on. I'd be concerned about what would happen in four years if we hosted our own COPR.

from tigeros.

from tigeros.

It is likely though that the build system will be on our infra unless we can find a good hosted way to do RPM builds.

Well, if we're hosting it on our own, I'd figure it would have to be hosted on our infrastructure, right? 😉

If COPR is something we want to use, I have a strong preference for using Fedora's COPR and creating a group for the TigerOS team to maintain any packages there. Hosting on our own introduces a significant amount of work, and it would require all of the team to contribute documentation for how to manage that. I encourage all of you to think too about how this will be maintained in five years from now.

from tigeros.

@jflory7 Licensing might be an issue. Also COPR may want to have one repo per package.

from tigeros.

Status update:

• We have an Ansible playbook that installs all the needed programs and does part of the configuration.
• We (@axk4545) are rethinking CI, will continue once that is decided.

from tigeros.

@ct-martin Travis is a no go. This is gonna be interesting with COPR. Biggest concern is licensing. I would go with automating our self hosted stuff (maybe ala this substituting fedpkg for rpmbuild ) and the process there but be able to migrate to COPR with tito ala this

@jflory7 I would like to get your input on how to make either of those more sustainable

from tigeros.

for help with using tito as describe above in hosted COPR and getting COPR to work with our subdir based approach talk to clime in #fedora-buildsys on freenode

from tigeros.

ideas:

• Lightweight build system section here https://www.hogarthuk.com/?q=node/11
• https://wiki.jenkins-ci.org/display/JENKINS/RPM+Sign+Plugin
• https://github.com/jhrcz/jenkins-rpm-builder
• https://schneide.wordpress.com/2013/03/18/1-click-deployment-of-rpms-with-jenkins/
  @ct-martin

from tigeros.

ok. so we should probably use tito in some way to get around the subdir part. we can run tito with jenkins to trigger. signing can likely use the jenkins plugin and deployment is copying in some way from the output dir to the mirror/repo dir and fixing perms. tito will handle making the tarballs and bascially going from flat source files to RPMs and SRPMs.

from tigeros.

Test with standard web server directory since that should have proper selinux context

from tigeros.

New idea for RPMs, GitLab runner on buildbox that talks to the GitLab.com instance of the project and does per branch rebuilds. we can use the only option to restrict the branches and then jsut script the generation of tarballs and the building.

Process might be like the following:

1. git clone the branch or git checkout and pull
2. enter proper directory and create tarball with tar -cf
3. run fedpkg to rebuild the package and email the admin or a team member or emit a notification in slack that the packages are done
4. user goes in and signs the RPMs
5. RPMs are then copied to the download host and permissions and repo fixed and updated. (maybe script this part)

thoughts @jflory7 @RITlug/tigeros-team ?

References

• https://gitlab.com/gitlab-org/gitlab-runner
• https://docs.gitlab.com/ce/ci/yaml/README.html#only-and-except-simplified
• https://docs.gitlab.com/runner/install/linux-repository.html

from tigeros.

We briefly discussed in the TigerOS Slack channel.

I am +1 to creating an individual repo for each unique package that we create.

This follows best practice in the same way that Fedora does packaging with dist-git. Packages are easier to find and maintain individually – putting each unique package into a new branch is an uncommon practice and I think is liable to leave things forgotten over time. Following the dist-git model also helps us follow the upstream policy of packaging for potential inclusion of our packages in Fedora or other repositories, like RPMFusion.

Additionally, the tools available in Fedora that we are considering for our own use expect individual packages to be in their own repo – using unique branches for each package may create more future headaches since branches aren't well-designed for this type of workflow.

from tigeros.

So I am talking with @ritjoe a little and he was favoring the idea of one package per repo. We would still need to workout a way to hook our builder up to github or gitlab or w/e so that it can trigger tasks on a push. Maybe we also take this as an opportunity to establish a RITlug presence on gitlab.com

from tigeros.

Current Plan

Ok. so I know this is getting stupid. anyway I found http://frostyx.cz/posts/copr-stack-dockerized and https://pagure.io/copr/copr/blob/master/f/rpmbuild.
I talked to the maintainer and he is willing to add a feature to copr-rpmbuild to allow it to talk directly to the builder container and run a build. so we can use that to trigger a build on a push and then we just make sure that each package has it's own repo... @ct-martin @jflory7 input?

Upstream tracking issue for allowing copr-rpmbuild to talk directly to builder: https://pagure.io/copr/copr/issue/102

from tigeros.

@axk4545 I'm not informed enough to add much here. I'm still opposed to having us run our own Copr instance, but I trust you all to make the right decisions for the project.

from tigeros.

Playbook to setup a COPR builder can be found at: https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/copr/backend/files/provision/provision_builder_tasks.yml

from tigeros.

THIS IS NO LONGER ACCURATE. THE CHANGES HAVE BEEN ADDED TO THE MASTER BRANCH AND ARE NOW IN THE OFFICIAL FEDORA REPOS
Here are the links to the COPR for copr-rpm-build-cmdline and a git repo with the source:

COPR: https://copr.fedorainfracloud.org/coprs/clime/copr-rpmbuild-cmdline/

git: https://github.com/axk4545/copr-rpmbuild-cmdline

from tigeros.

Working on automating deployment of builder with Ansible. May revert to Docker if needed since that is premade.
Attaching relevant IRC logs
COPR_plan.txt

from tigeros.

Command example on builder: copr-rpmbuild scm --clone-url https://pagure.io/rpkg-client --chroot fedora-27-x86_64 the output RPM will be in /var/lib/copr-rpmbuild/results

from tigeros.

Having issues with Jenkins, looking into GitLab Runner

from tigeros.

We now have IP addresses and can provision the webhost

from tigeros.

It will trigger based on push to master

from tigeros.

I think this is going to take a lot longer than the Beta release to complete. Thoughts on moving this to the final release?

from tigeros.

@Tjzabel I'm not going to block the beta on it, but I would like to keep it on the beta milestone due to its importance

from tigeros.

@ct-martin sounds good.

Also, it is probably a good idea to split this issue into smaller issues so it is much easier to track.
It would be a good idea to have an issue for updating the builder + mirror Ansible playbooks, an issue for the GitLab ISO CI, and an issue for the RPM building CI

from tigeros.

@Tjzabel I actually have a draft issue for updating the playbooks right now with some other stuff as well. I have a couple thoughts on how to split up the issue, let's talk about them in the meeting in a couple minutes

from tigeros.

Next steps are to do the write-up on how TigerOS CI is going to work. Current draft is located on my GitLab repo.

The goal is to implement CI on a single RPM package to make sure the process goes smoothly. Then, a TigerOS mirror will be set up on the GitLab public site, where we will be hosting CI on the builder, and possibly the mirror.

The draft documentation will be added into the wiki at some point.

from tigeros.

Furthermore, I have an RPM CI script in the works to do the RPM building.

However, this is still a rough script, and needs to be ironed out in order to make the RPM building process fully automated. At the moment, the release version and package name need to be added as parameters to the script, which are not yet found automatically.

In order to build an RPM package, the specific release is needed. We need to figure out a way to grep for the release version without manually putting it in.

from tigeros.

In addition, RPM CI and ISO CI are very different from one another, and so I believe these two CI issues should be separated to create a more easy-to-follow workflow.

from tigeros.

@Tjzabel could you clarify the following? I have no idea what your mean, and my best guess doesn't get close to what I had planned.

Then, a TigerOS mirror will be set up on the GitLab public site, where we will be hosting CI on the builder, and possibly the mirror.

Also, I have access to the GitHub credentials now so I can proceed when I have time (probably next weekend)

from tigeros.

@ct-martin If it helps, I also have no idea what I was trying to write there 😄

I am going to update that comment now with what I actually meant to say.

from tigeros.

@ct-martin actually, I do know what I was saying there.

Essentially, there will be a GitLab instance that is mirroring what the GitHub repo has, so when there is a change to the repo, the GitLab CI will pull the change and do the automation. Correct me if this line of thinking isn't what you had in mind.

from tigeros.

@Tjzabel ok, yes, that is what I had in mind. I was confused by "mirror" having multiple meanings. What do you think about using terminology along the lines of "TigerOS mirror" or "the mirror" for mirrors.ritlug.com (to maintain backwards-compatibility in terminology) and "GitLab repo mirror" for the GitLab CI/CD for external repositories? Wording doesn't have to be exact.

Thanks for the clarification

from tigeros.

@ct-martin I got the same confusion when I re-read my own comment.

I definitely think it is a good idea to create uniform terminology when referring to the different parts. Even GitLab mirror is enough imo, and TigerOS mirror for the packages mirror.

from tigeros.

@Tjzabel I think it would be a good idea to use "GitLab repo mirror" (with "repo") since GitLab Runner will be automating things on the mirror soon. I'll file an issue to start a glossary and we can discuss on Wednesday

from tigeros.

As per meeting we have decided to do Continuous Delivery to a new development mirror based on the devel branches of the respective repos. This means that for devel builds instead of getting put into a queueing folder, they will be signed and put straight on a development mirror separate from the current one. This will also require standardizing branch protection, which I will figure out and document.

from tigeros.

I have installed GitLab Runner on the builder, will set up the pipelines another time.

from tigeros.

This is no longer in scope for a beta release.

from tigeros.

Update

Currently testing out the ISO-building Dockerfile in order to build the F29 ISO.
We will be one step closer to adding build automation if this succeeds 👍

from tigeros.