Scope and structure of this repository, godot-build-scripts and build-containers about godot-commit-artifacts HOT 2 OPEN

Okay, so there are a few things in here. And I'd like to mention beforehand that most of the building stuff is pretty niche, so it's on a backburner, so concreteness of any plans or their timeframe is not guaranteed at the moment.

With that said, I do see the merit of putting our build containers into some docker registry properly. registry.prehensile-tales.com is what we use, as Prehensile Tales, managed by @hpvb, are providing us with a build server. But I can see how trusting a 3rd party registry on an unrelated to Godot domain can be an issue for people. So yeah, if we can put them somewhere where people could easily fetch them through various docker-powered tools, let's do it. Adding a CI action to publish them directly from the repo makes sense to me.

As for providing pre-compiled binaries as containers ready for testing, this can also be useful, of course. For official public builds creation and deployment of those can be a part of our normal build routine, so the overhead should be pretty low. For the rolling version based on the dev branches it's trickier. Without a process to compile nightlies in place, we only have CI builds. And those CI builds, as I mentioned, are not suitable for publishing in an official capacity. We don't provide all the versions, and they are not compiled the same, so putting them on some latest tag wouldn't work. I'd wait for future development on nightlies or other more regular dev builds before considering this.

The reason build scripts don't have any CI is because we do not use GitHub to create the official builds, and I don't think we ever will. We have a build server which is dedicated to our needs and is powerful to handle these big builds with optimizations and other expensive steps. And it's isolated, so signing and whatnot can be done without any risk of exposure. So anything related to building Godot is not going to be a part of the public infrastructure, I'm afraid.

I also don't see much point in merging the repos, unless there is an issue with keeping them in sync. But we update them so rarely, it's probably not a big deal regardless.

And finally, if we have official nightlies or other sources that I can link in this repo, I surely will. For official releases we can add links to the download page on the main website too.

from godot-commit-artifacts.

But I can see how trusting a 3rd party registry on an unrelated to Godot domain can be an issue for people.

I believe that trusting a 3rd party is not much of a problem, as long as the process is public, thus auditable.

• The sources of the environments (containers).
• The scripts.
• The logs of how containers are built and how scripts are run, including the SHAs of the commits, and the sha256shums of the results.
• The public keys of the credentials used for signing.

I'm trusting GitHub or GitLab when I run workflows on the servers provided by them, so why not trust Prehensile Tales (particularly given their obvious interest in the good health of Godot). However, trust and faith are not the same. Whenever I run something on GitHub's or GitLab's runners I do have access to all the information I need to ensure it's correct. I can very easily reproduce everything on my own infrastructure and check that they are building the code I expect them to build and not something else. I don't need to do it every day, but I can do it whenever I need to debug something or just periodically for hygiene.

Without a process to compile nightlies in place, we only have CI builds. And those CI builds, as I mentioned, are not suitable for publishing in an official capacity. We don't provide all the versions, and they are not compiled the same, so putting them on some latest tag wouldn't work.

As commented, the proposal is to have a workflow in godot-build-scripts (or godot-builds) which builds the master branch the same way all regular releases are built and uploads the artifacts as assets of a pre-release in that secondary repository.
So, they would be compiled the same and the would not be in the main repository (to avoid misunderstandings).

For clarification:

• A workflow in build-containers builds the containers and pushes them to ghcr.io/godotengine/build and registry.prehensile-tales.com.
• Another workflow in godot-build-scripts:
  • Builds the master branch of godot and pushes the artifacts to a tip/nightly pre-release in godot-build-script.
  • And, builds the containers for users and pushes them to ghcr.io/godotengine/editor and ghcr.io/godotengine/export.
• The workflows or the artifacts from godot are unused/unrelated.

The reason build scripts don't have any CI is because we do not use GitHub to create the official builds, and I don't think we ever will. We have a build server which is dedicated to our needs and is powerful to handle these big builds with optimizations and other expensive steps. And it's isolated, so signing and whatnot can be done without any risk of exposure. So anything related to building Godot is not going to be a part of the public infrastructure, I'm afraid.

This is really unfortunate, because it requires faith. AFAIAA, the logs of that build server are not public. Hence, we cannot see which sources or optimisations are used. We need to believe that whatever is in the repos is not outdated/incomplete.

I think that nightlies should be built on GHA. Apart from improving the transparency, it would serve as continous testing of those resources.

Should performance be an issue, a GitHub runner (see https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners) might be executed in a sandbox on the machine provided by Prehensile Tales and attached to godot-build-scripts. Then, the scheduled workflow would use that runner instead of the ones provided by GitHub. As a result, logs would be shown in the Actions tabs and the same workflow used by Godot for official and nightly releases might be used by any other user on their own runners or using the ones provided by GitHub.

Alternatively, the logs of the runs on the machine provided by Prehensile Tales might be made public, but I'm afraid that might be more effort than setting up a runner.

I also don't see much point in merging the repos, unless there is an issue with keeping them in sync. But we update them so rarely, it's probably not a big deal regardless.

Since scripts are to be run on the containers and containers are to be used for running the scripts, issues and PRs are likely to be related. For instance, 30% of the closed PRs in godot-build-scripts are related to build-containers: https://github.com/godotengine/godot-build-scripts/pulls?q=is%3Apr+is%3Aclosed+build-containers. That's increased effort for both developers and readers with no obvious benefit (wrt requiring separated issues, projects, discussions, actions, etc.).

from godot-commit-artifacts.