Handle openshfit samples as declared in the imageset-config

download operators by semver when semver present in imageset-config

When I run create full with a config that has headsOnly: true for some catalog, or create diff, I expect that diff to be mirrored to disk as an image. I also expect publish to combine that diff with an existing catalog image (pulled from data in a CatalogSource) and pushed to the catalog's registry under a desired tag (defaulting to the tag from the existing catalog image).

Currently this functionality is broken/does not exist. Since opm diff libraries only generate an file-based catalog, a new image must be built containing this file then mirrored. The same needs to happen during publish, but with the output of opm render <existing catalog image> <mirrored diff catalog image>.

• If no metadata found, start new
• If metadata found, load last imageset and append new run to the existing imageset

This is a delicate procedure. We need to make sure that we nail the logic on this. The logic needs to be clarified, but this is a rough guess:

1. patch override operator hub
2. update icmp for ocp releases, images referenced in the catalogs, and other arbitrary images in the imageset.
3. check installed operator versions and verify upgrade path to new catalog. Error if operators will become orphaned.
4. once operators are upgradable to new catalog version, upgrade ocp
5. patch override operator hub
6. upgrade operators from new catalog version

For the record, I am not saying that we should be triggering cluster upgrades from this application. Also, some of this might be handled by CVO or OLM. So some of this might not be handled by this app, but some of these objects need to be managed by us.

Disk space optimization:

• Remove files as they being processed.
• For create this would be as they are being archived
• For publish this would be as the images are being pushed to the registry one at a time and files are pulled from the archives as needed
  Note: Limitation include symlink tracking and unarchiving. We need to capture symlink information in the associations during the create process.

Improve auth ux by allowing user defined alternate credentials location. (cli flag or in imagespec)

The File type needs to be changed to include a type so we can determine which are manifests so we can scan them and unmarshal them into Schema2 types from this library.

Allow for more granular control when declaring operators to be downloaded. Allow operator downloads to be optionally selected by individual name and optionally selected by semver. If no semver, the selected named download should behave as expected with headsonly.

All mirrored images need to include icsp refs.

Help us feel better about closing tickets that we wont work on.

I want to add a new command to do a describe on a particular imageset.
It will parse the metadata and output into a human readable format so users can verify their bundles before they publish if needed to troubleshoot any bundle sequence issues.

The size of the generated tar-ball is immense -- 100+GB. Duplicating it to disk before running the publish seems a big waste. Suggest processing the file without extracting to disk first, or extract small portions, process, and delete the temporary files, and continue with the next set of files.

• clarify and cleanup comments
• comment exported items
• Clean up logging and get better log level categorization of certain messages. (Need more info messages and need to fix log file output)

To ensure integrity of metadata stored in the disconnected environments, I want to create or use a merkle tree implementation to represent all publishing transactions.

The path that operators are downloaded to includes a duplication of paths as seen here:

file://redhat/redhat-operator-index/redhat/redhat-operator-index/rhscl/postgresql-10-rhel7

The path needs to be deduplicated.

In certain situations, it would be helpful to mirror OCP release content from alternative locations. Can an optional mirror key be added to the release struct that provides an alternate location to retrieve release content?

When the pull secret is set in the imageset config, the value is not being passed when generating the diff with the heads only option. The 401 error occurs when action.Run trys to render the news refs.

https://github.com/estroz/operator-registry/blob/1787329c541ff5c4533d680df5f208805eee93ea/pkg/action/diff.go#L52

It might make more sense to combine create full and create diff into a single function called create. The behavior of the app would be determined by the state of the metadata. If a new imageset (create full) is desired, then a new workspace would be specified. This would reduce cli complexity and provide a similar experience to publish.

The publishing command needs to verify the sequence number (should be one) in the first imageset processed to avoid processing a differential imageset without ever processing a full imageset.

@estroz , not sure if this is expected behavior. I get this warning at the beginning of the operator download during the create phase.

WARNING base64 decode CSV icon for bundle "verticalpodautoscaler.4.8.0-202107291502"  error=illegal base64 data at input byte 0

• Load registry pull secrets from imageset-config.
• Handle multiple credential strings (multiple registries)

background: Allow users to add separate registry credentials instead of editing serialized json in the red hat pull secret.

I want to add support for a streamlined process with the create and publish functionality to support customers with partially disconnected environments.

Add a flag that allows a user to specify a CA cert bundle used while interacting with registries.

Release, operator, and sample images must allow for architecture declaration

1. Improve error identification (currently not failing on oc errors)
2. add release/additional image testing (currently only tests operators)

We wanted to use fake implementations to unit test our interfaces

https://github.com/maxbrunsfeld/counterfeiter

Download additionalImages as annotated in the imageset-config

I think go install is a pretty well-known/standard approach. but currently, if I do that I get:

$ go install github.com/RedHatGov/bundle@latest
go: downloading github.com/RedHatGov/bundle v0.1.0-alpha.1
go install github.com/RedHatGov/bundle@latest: module github.com/RedHatGov/bundle@latest found (v0.1.0-alpha.1), but does not contain package github.com/RedHatGov/bundle

When publish is called, check for existing directory structure and create a new one if needed

The current architecture of bundle does not support CI workflows.
Decouple the imageset-config from the root of the managed directory and pass it to bundle with a --config flag

I want the ability to block and image, ex. alpine, and all images that depend on a blocked image's layer(s) with a warning log.

During publishing/catalogsource creation, write catalogsource image value by tag of declared version from the imageset-config

Add support for helm charts and associated images:

• as a imageset-config key under mirror.
• Default to latest for charts if version is not specified.
• Pack the chart in the imageset with it's associated images.
• Output mirrored charts in sub-directories with the icsp/catalog source manifests
• include chart image references in icsps

Download operator latest when declared by name (minus semver)

Prevent blocked images from being downloaded when declared in additionalImages and operators

Allow for user defined remote backends for managed data

During the first "oc-bundle create full" run, when a given version of OCP is downloaded, the text/prompt is "uploading": uploading:

file://openshift/release sha256:2e41ca5fb22b44b82ce7d08e7436d0965776843a45bcdcf5fc2a38fafe4f027b 9.534MiB
uploading: file://openshift/release sha256:ae411670fc26c36fee9fd1232139e051b31ebbed7b411f22dc21fd153ff06456 85.64MiB
uploading: file://openshift/release sha256:8795e1eff4a91cb42c085fd44ce7724e42b5f536da218a88bcbe5d23b3376eda 103.5MiB
uploading: file://openshift/release sha256:917fc69dd02cce554468e1bce0969c5dc133ba8a0427d08626fd7a3cd461faa1 28.61MiB
uploading: file://openshift/release sha256:fdae22ebfdbf3ab549eeabe88fcf4d14df92758a6d0080989af85810afdbff00 515.6KiB

I would expect downloading and not uploading in this context?

When running create full, the output imageset name is bundle_000000.tar

When running create diff in the same workspace, the output imageset name is the same (bundle_000000.tar).

As a user, I would expect the name of the imageset to reflect the sequence number of the creation.

• We need to validate the checksum of each image layer
• We need to validate the checksum of the image
• We need to validate the the manifests of the image

Use Case:

ImageSourceContentPolicies need to be generated for releases in the event that a cluster would go from connected to disconnected.

Problen:

Currently oc release mirror only produces a mapping to the tagged symlinks on disk.

Solution:

To allow ICSP to be generate we need to get the link target information and manipulate the mapping before the Associations are gathered.

Generate ICMP/CatalogSource files for the following:

• Release images
• operator images as per catalog version
• additional images as needed

• Create basic CI for linting/validation
• Create contributing page

If oc-bundle exits during an image download, a partially downloaded layer/image may cause a crash in a subsequent oc-bundle run that attempts to start from where the previous run exited. The command should be "smart" enough to clean up these images and restart the download from the previous run's exit point, or at the very least not crash.

Instead of tracking past files, we need to track past layer digests in the metadata and have the archiver only pack new layers and all manifests.

bundle inherits a lot of logic from oc for handling releases and operators. It also uses the registry client from oc mirror for general container image mirroring. The most of the pieces that we needed from oc were unexported, so we resorted to calling Run() from each applicable oc package. By using the Run function from the oc imports, we lose control over logging output, stdout/err, and any other behavior that we want to program into the process.

Remove the Run() calls to oc imports and either use internal logic for or PR oc to export the needed pieces for reuse here.

Download entire catalog latest

Requirements:

Full

• Must create a full archive that is segmented per user define MB size
• Upon writing a file to the archive it must write an empty file to bundle/

Diff

• Must create on archive and pull in existing bundle data
• Must be set to NOT overwrite
• Pull in each file under source
• Segment from there

Optional, but nice to have:

• Bundle verification stronger than GZIP CRC

Running an initial "create full" that includes ubi8, I get a "FATAL error creating image resolver: /config.json" where "" is the content of the pull secret in the imageset-config file. I have the following option in the additionalimages:

  additionalimages:
    - name: registry.redhat.io/ubi8/ubi:latest

Since the pull secret is long and shouldn't be included here, I'll just enclose the few lines before the error happens:

info: Planning completed in 2.59s
sha256:038b03c6db9a6484c7328c33eac4cb7b4d9f64a5a74c3849add4b000222f20c4 file://registry.redhat.io/ubi8/ubi
sha256:ba6956f6fabe47ecee824470fa3e3886e563c7b0e4a5240fe96effe559396234 file://registry.redhat.io/ubi8/ubi
sha256:4ce314c597d03077198d51cdf0421d567234bb559e912b89fbbbfa87ed67dc1f file://registry.redhat.io/ubi8/ubi
sha256:08117490cf2a61a85d560f87a5dac3516c043dc092319880f3874b6b6982e71b file://registry.redhat.io/ubi8/ubi
sha256:7254254afafd44c850940445c0480e449fea73a169aa067389252fef35bd7a3f file://registry.redhat.io/ubi8/ubi:latest
info: Mirroring completed in 0s (0B/s)
FATAL error creating image resolver: {"auths":{"cloud.openshift.com"......................

Changes need to be made to additional_test.go to use a mirror or a different image. We are currently using an alpine image from docker.io and frequently see CI failures do to rate limiting.