Provide alternative RHEL based agent Docker image about gocd HOT 16 CLOSED

Alright, let's go with Alma. Some purists feel the setup of the foundation behind Alma is a bit more solid as to risks of the project going in weird directions; but not difficult to switch in any case.

from gocd.

Thanks for the message. Just curious, what does your desire for a RHEL container stem from? Perhaps packaging RPMs in an agent or something like that?

from gocd.

The reason largely stems form my concern (and surprise, only really discovered lately) that CentOS Streams don't seem to get the security patch and package rebuild attention that more 'stable' ones do (Rocky, Alma etc).

from gocd.

Yep, I don't want CentOS stream either - I jumped ship to Rocky ;)

I've used CentOS since v3 was released ~2004 so now manage many servers running Rocky9 and is nice to have consistency with management tools accumulated over the years i.e. lots of shell, Perl, Ansible, etc which I'm "automating" with GoCD. But I'd like to be able to run all processes that are put into GoCD outside of GoCD as well in case "something bad happens"! So I'd like to target a specific environment inside and outside of GoCD.

I'm technically using GoCD more for a "task automation system" which is working well, rather than CI/CD, so maybe my use case is slightly different to other users?

It's just easier to have everything the same "flavour" (RedHat, yum/dnf, RPM, etc) rather than a mix of distros. But if I'm an outlier, I can work something else out... ;)

from gocd.

Do you build your own custom agent image with these tools pre-installed, using a GoCD agent image as base? If so, wondering if you have looked at whether the tools you need/want are available for Wolfi and could be layered on such that your GoCD jobs are largely OS agnostic and just need certainly binaries to be available.

from gocd.

Yep, I add tools to the GoCD base image. And yep, I looked into Wolfi but the apps I need aren't available, and even if they were, software packages are different between OS. e.g. Apache 2.4 on CentOS is very different to Apache 2.4 on Debian (config layout, management tools, etc) - I'm not actually using Apache in GoCD but hopefully that's a "familiar" example to illustrate differences in vendor packaged software.

If I was starting from scratch now, I might entertain the idea of something like Wolfi, but I have "accumulated" over 20 years of tools developed for "heavier" general purpose OS which is much easier to transition to the same "heavy" Linux OS in a GoCD image.

And I've already thought about switching to Debian or something when CentOS went "Stream" but even though Debian is a "heavy" OS like Rocky, it's still different enough to need "work" to transition. So yep, I know I could switch to Wolfi or Debian, but that's work that doesn't need to be done if I have a Rocky9 GoCD image to run existing workflows. The CentOS9 image has worked well so far.

I guess we'll see if anyone else is in the same boat soon, or if I'll be alone running Rocky9 in GoCD... Either way is fine ;)

from gocd.

Well, Apache is not a fair example here as that's not the type of software that is within the scope of what would be expected to run on a build/deploy agent.

Better examples would be in the realm of CLI tooling, or short-running stuff.

If you have a very sophisticated/opinionated set of tooling that has specific versions needed etc, I'd usually suggest people decouple that tooling from GoCD agents via use of docker-in-docker or podman-in-docker or something like that, such that ones GoCD tasks become docker run -v $(pwd):/my-scripts my-rocky-based-tools-image:v1.2.3 -- bash -cl /myscripts/do-work.sh or similar - and are thus less coupled with specific agents.

But docker-in-docker and similar have their own challenges/frictions of course.

from gocd.

Yep, Apache was just an obtuse example, but I didn't want to get too deep into implementation details. And yes, I could rework everything following your kind suggestions, but that's work I don't need to do if there's a Rocky9 agent.

Anyway, I was just posting as mentioned in the release notes to see if anyone else wants a RHEL flavoured agent image, but I guess more people use the Debian, Ubuntu, etc Docker agents?! And that's ok :)

from gocd.

Yup, it's still a fair request.

Just suggesting an approach you might not have considered, which might have some other benefits of decoupling - and be relatively straightforward.

Let's see if any others express interest though!

from gocd.

I am tracking this issue too for Rocky Linux gocd.

Alma linux does have a larger developer mindshare.

from gocd.

What’s your use case for needing a RHEL-esque agent @fire ?

If we are to do so at the moment I am leaning towards Rocky because

• the container images seem to have a lot more pulls (and the main interest here is a decent base image)
• Depending on your perspective, Rocky’s bit-for-bit RHEL compatibility might lead to a slightly smaller delta than Alma’s approach?

Having said that, Alma’s minimal images are a bit smaller, I guess I’d need to see if that makes a difference on a built image though after layering on what GoCD agent needs.

from gocd.

I am leaning towards Rocky too as I am trying to replace a centos based gocd agent.

I mentioned alma because my peers are trending towards it rather than rockylinux.

from gocd.

OK, but what's the reason for using a CentOS container for your agent rather than, say, any other GoCD container image?

Since the removal after 24.1.0 is of CentOS container images, the intent is to explore why people feel they need RHEL-like agent inages as opposed to the many other options that exist (debian, Ubuntu, Wolfi, alpine).

from gocd.

1. legacy of using Centos
   a. promise of more stability / effort funding the stabilization process
2. Caching of large binary toolchains in the image compared to stock images
   a. The alternative to this is to load the toolchains in dind, but it was a bit of work.
   b. Caching is important because the toolchains are time consuming to build and transfer
   b. a pipeline can be constructed to move the toolchains as an alternative
3. I wanted to revive the fedora image, but that's even more niche than the RHEL image.. for support software coded for the RHEL ecosystem
4. I wish for gocd to continue to support the enterprise ecosystems which are on RHEL ecosystem rather than like Wolfi, Alpine, Arch or Debian.

from gocd.

• legacy of using Centos
  a. promise of more stability / effort funding the stabilization process

Generally speaking I believe the "stability" requirements are rather different for a container than a virtual machine, given the reliance on the host kernel, absence of hardware driver relevance etc, however the stability and provision of standard packages is possibly relevant.

• Caching of large binary toolchains in the image compared to stock images

Which toolchains are you primarily concerned with? I've found most standard toolchains are easily available for wolfi since building in containers is such a common use case, but YMMV.

• I wish for gocd to continue to support the enterprise ecosystems which are on RHEL ecosystem rather than like Wolfi, Alpine, Arch or Debian.

Sadly "GoCD support" right now is really just me - i.e 1 person - so we have to keep it in perspective. Right now, there is limited adhoc funding of build/test environments from Thoughtworks, but that's it - and theoretically could go away any time as provided on a goodwill basis. The more there is to build, the more resources required :-)

Running GoCD on CentOS/RHEL is still supported (we continue to provide RPMs and such) - the question here is specifically about containers and agents, which are opinionated wrapping environments to make it a bit easier to layer on tools - and thus the wisdom of pre-building an image for rocky|alma centrally rather than having users with more specific requirements build and publish their own.

Back to the image choice

Personally I'd prefer to have one of alma|rocky as requested here and drop Debian images entirely (Ubuntu is usually a better choice from a security/patch response perspective IMHO), but sadly I don't have good data to make these decisions on.

Others

• gocd-agent-wolfi ~4000 pulls per year since release (new default elastic image) (vulns now: ZERO)
• gocd-agent-docker-dind (Alpine) ~300,000 pulls per since since release (default elastic image) (vulns now: ZERO)

RHEL variants

• gocd-agent-(rocky|alma)linux-9 N/A pulls (vulns now: ZERO)
• gocd-agent-centos-9 ~1500 pulls per year since release
• gocd-agent-centos-8 ~30000 pulls per year since release
• gocd-agent-centos-7 ~145000 pulls per year since release (this was a default at some point)

Debian variants

• gocd-agent-debian-12 ~2500 pulls per year since release (vulns now: 2 crit, 27 high)
• gocd-agent-debian-11 ~2200 pulls per year since release (vulns now: 3 crit, 41 high)
• gocd-agent-debian-10 ~11800 pulls per year since release
• gocd-agent-ubuntu-24 ~2500 pulls per year since release (vulns now: 7 med)
• gocd-agent-ubuntu-22 ~2500 pulls per year since release (vulns now: 8 med)
• gocd-agent-ubuntu-20 ~4240 pulls per year since release (vulns now: 7 med)

So I guess historically one can say the RHEL variants have been quite popular, but it's dropped pretty dramatically during the Stream transition years for one reason or another (GoCD container images didn't change repos with the stream transition, so the 8 numbers include CentOS and CentOS Stream variants). Unfortunately I don't have trend info from Docker Hub.

But I am softening to the idea of having an Alma or Rocky, as maybe I can kill off Debian images instead (which are full of vulnerability noise).

Alma or Rocky?

Pros for Alma?

1. At time of writing an alma image based on 9.3-minimal is about 30MB smaller (uncompresssed) than a Rocky image (391MB vs 422MB). This seems to do with Rocky

• including curl (Alma includes curl-minimal)
• including langpacks-en and some dependent fonts (Alma does not)

Not sure if this makes a huge difference, as may be something which could be changed upstream...

2. Rocky seems to be stuck with releasing their 9.4 versions at docker-library/official-images#16754

3. We currently already use Alma containers to test RPMs since they conveniently provide image variants with an init system (needed to validate installer services/systemd etc).

https://github.com/gocd/installer-testing/blob/c1f8686d71f86d1899b15de8d8f18460806770eb/Rakefile#L167-L175

from gocd.

Picking one of the variations in the RHEL ecosystem is probably better than picking both in terms of limited resources.

As you can tell I was pretty lost between the choices.

from gocd.