paketo-community / explorations Goto Github PK

Context

Recently, a user found that in some instances, multiple builds on the same source code produce images with different digests (see this thread). The user expected that the build would've produced the same image. This issue appears to have cropped up without our knowledge since we did not have language-family level tests for reproducibility. This latest occurrence may be related to the SBOM work we recently added.

Issue

We should perform an investigation across all of our language family buildpacks to determine the status of build reproducibility. For any buildpacks that do not produce reproducible images, we should file an issue to flag that and (hopefully) resolve it down the line. The issues file should include an outcome about adding a test at the language-test level.
Buildpacks to investigate:

• .NET Core paketo-buildpacks/dotnet-core#756
• Go paketo-buildpacks/go#645
• NodeJS paketo-buildpacks/nodejs#631
• PHP paketo-buildpacks/php#568
• Python paketo-buildpacks/python#507
• Ruby paketo-buildpacks/ruby#811
• Web Servers
• Java Native Image (cc @dmikusa-pivotal)
• Java (cc @dmikusa-pivotal)

Summary

Many of the Paketo buildpacks contain references to dependencies that they will install during their build phase. These dependencies are often language runtimes like Ruby MRI or package managers like Poetry. The dependencies are tracked and built from their upstream source (dep-server) and updated in buildpacks (jam update-dependencies and dependency/update action) through a considerable amount of automation. This current architecture has outlived its utility and will likely present a significant technical headwind as we attempt to move buildpacks to new stacks.

Outcome

This exploration should focus on providing direction for a future effort to modernize the dependency-building infrastructure we depend upon in Paketo Buildpacks. In the process, this exploration should weigh the following goals and any others that may be discovered in the exploration process with the result being an RFC outlining a future direction for dependencies.

Goals

Remove Cloud Foundry Dependency

The dependency-building automation is tightly coupled to the legacy dependency-building infrastructure inherited from Cloud Foundry (buildpacks-ci and binary-builder). Making changes to these codebases to support new Paketo use cases and features is a convoluted and difficult process. Ideally, we could move, refactor, or rewrite this code into codebases that we maintain within the Paketo Buildpacks project.

Use Upstream References

Many of our dependencies may already be built in a form that is usable on top of our stacks. In these cases, we shouldn't be re-building them to no real benefit. Instead, we should just reference the upstream artifact download location. An example of this might be Go. The Go downloads page serves pre-built tarballs for Linux on a number of architectures. Ideally, we would just be able to reference these download URLs in our go-dist buildpack.toml file.

Adopt Federated Model

The current dependency-building automation is mostly centralized in the dep-server repository. While this was good when we had a dedicated team of folks with a strong working knowledge of these components, it has become more difficult to maintain with a more diffuse ownership model. It would likely be more advantageous for us to move much of the monitoring and building infrastructure into the repositories where these dependencies will ultimately reside. For instance, it would make sense to have the dependency-building infrastructure for the Node.js runtime within the node-engine repository and directly under the responsibility of the Node.js Maintainers team. It still would make sense for a Dependencies team to help maintain expertise and tooling for the workflows involved in dependency-building generally, but the particulars of each dependency could be distributed to their respective buildpack repository.

Consolidate with Java Workflows

The dependency-building infrastructure described above does not encompass any of the dependencies that contribute to the Java buildpacks. The Java buildpacks have their own system for managing dependencies. It is worth considering what a consolidation of these systems might look like.

Enable Multi-stack / Multi-architecture Support

The dependency-building infrastructure is tightly coupled to an Ubuntu Bionic-derived stack on a Linux AMD64 architecture. Ideally, we would propose a solution that would enable us to deliver dependencies on a more diverse set of operating system / architecture pairings.