hawtio / hawtio-operator Goto Github PK

When creating a Hawtio instance and setting the metadataPropagation to add labels and annotations to the created resources doesn't do anything, there's no error in the operator log and the resources are created as if no metadataPropagation was set at all.

Example CR spec:

spec:
  auth: {}
  config:
    about: {}
    branding: {}
    online:
      consoleLink: {}
  metadataPropagation:
    annotations:
      - key=value
    labels:
      - asdf=aaaa
  nginx: {}
  rbac: {}
  resources: {}
  route:
    caCert:
      key: ''
    certSecret: {}
  type: Namespace

Following #25, it would be useful that the operator watches for the generated client certificate secret, and issue / rotate a new certificate when it is about to expire.

There is version field that allows us to change it from latest in the Hawtio CRD, but for dev purposes sometimes it's also convenient to change the docker image url from docker.io/hawtio/online to somewhere else.

Overview

There are a couple of recommended best practices for how operators handle the version of the operand as detailed here, namely:

1. Operator fan-out - where the Operator allows the user to specify the version in the custom resource
2. single version - where the Operator is tied to the version of the operand.
3. hybrid approach - where the Operator is tied to a range of versions, and the user can select some level of the version

Currently, hawtio uses the first approach, specifying a default version in the CR yet allowing the user to change this, as appropriate. In my personal view, I think the upgrade to Hawtio-Online 2 would be a good time to switch the operator to the 2nd approach.

The 2nd approach is, to my mind, simpler in that it maintains a 1:1 relationship between operator and operand. It ensures that the upgrade process is controlled by the OLM's operator versioning pattern. As the operator is responsible for installing a number of resources, including the deployment, the current design allows the possibility that the user could install a much later operand in an old operator, resulting in a situation where all the resources in the operator no longer correctly handle features of the new operand.

The current approach only seems to specify the version number as a default. Therefore, it is possible for the user to change this to any version / tag, without any perceived limits (I cannot find any checks on the version in the operator so it may just be my ignorance :-) ).

Suggestion for Change

Remove the version from the CR and instead handle it with an environment variable and golang constant in the same manner as the image repository. This would allow the operator to be compiled with a 1:1 relationship to the operand (the golang constant) but would maintain a degree of customization by having the environment variable be available to override if necessary, eg. development / testing.

The removal of the version from the CR would be useful to remove prior to the upgrading of the CRD to 1.0.0. After that, if we chose to make this change, it would be more problematic to include such a change in a micro-version update.

• hawtio/hawtio-online#138

Errors being produced:

{"level":"error","ts":1701096292.863362,"logger":"controller_hawtio","msg":"Error reconciling ConfigMap","Request.Namespace":"hawtio-dev","Request.Name":"hawtio-online","error":"error AddResources: configmaps \"hawtio-online\" already exists","stacktrace":"github.com/hawtio/hawtio-operator/pkg/controller/hawtio.(*ReconcileHawtio).Reconcile\n\tgithub.com/hawtio/hawtio-operator/pkg/controller/hawtio/hawtio_controller.go:411\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:90"}

{"level":"error","ts":1701096292.863478,"logger":"controller","msg":"Reconciler error","controller":"hawtio-controller","name":"hawtio-online","namespace":"hawtio-dev","error":"error AddResources: configmaps \"hawtio-online\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:237\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:90"}

As hawtio-online now provides RBAC support, it should be enabled when it's deployed with the operator.

It should be possible to provide a custom ACL file.

Please can you add an appropriate LICENSE file to the top-level directory.

• Add support for installing and executing the operator on kubernetes;
  • #99;
• Upgrade operator dependencies to support ingress resources
  • #101;
• Remove deprecated v1/betav1 Cronjob API
  • #102;

If the provided tag is a sha256 checksum then its format is sha256:[a-fA-F]+ (256 bytes). The container deployment construction is not able to support this format due to the hardcoded use of the :.

We need to cut 0.4.0 to address hawtio/hawtio#2695.

There should be a way to add custom certificates for the Route created by the operator. Alternatively there should be an option to disable the Route creation entirely, so we can add a custom Route.

It apears that the finalizers are installed for cleaning up some cluster-wide resources like ConsoleLink and OAuthClient:

Probably the issue is that it doesn't work as originally expected and instead blocks the clean up and reinstallation in some way. Instead of removing the finalizer altogether, we can revisit and improve it to not block the clean up anymore.

This was a useful update for maintenance purpose in syndesis. It ensures that clienttools is passed around and the same clients are used throughout and always know where to get them from.

When trying to set some non-number value (e.g. dev) to the version in Hawtio CRD, hawtio-operator fails to deploy it with the following error:

{"level":"error","ts":1605829871.2426846,"logger":"controller","msg":"Reconciler error","controller":"hawtio-controller","name":"hawtio-online","namespace":"hawtio","error":"Invalid Semantic Version","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:237\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90"}

Considering it specifies a tag for the docker image, it should accept any arbitrary values.

Receiving odd golang errors with the building of the operator in build systems. Identified that the go.mod file was generated with golang 1.13 while using later version (1.19) for the golang image building the operator. Internal build systems, in turn, use golang 1.20.

So to mitigate and hopefully remove any errors, the go.mod file will be re-generated using golang 1.20, aligning with the build systems.

https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#subresources

When the main.ImageVersion var is assigned using a -X parameter in go's LDFlags, the tag of the hawtio-online deployment remains latest, eg.

CGO_ENABLED=0 go build -ldflags "-X main.ImageVersion=2.0.0-202311281710 -X main.ImageRepository=quay.io/phantomjinx/hawtio-online"

The operator should perform the steps described in https://github.com/hawtio/hawtio-online#openshift-4 for users.

See also: https://issues.redhat.com/browse/ENTESB-11132

The ConsoleLink objects are created with the same name of the Hawtio object. Due to not being created in the hawtio namespace, those resources are changed by multiple installations of the hawtio-operator.
The ConsoleLink objects need to include the hawtio-operator's namespace in their name to avoid conflicts.

Following #34, the ClusterRole permission required to read the service signing certificate secret should be restricted to the openshift-service-ca namespace.

The minimal RBAC can be achieved with:

ca_role.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: hawtio-operator-ca
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  resourceNames:
  - signing-key
  verbs:
  - get

ca_role_binding.yaml:

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: hawtio-operator
  namespace: openshift-service-ca
subjects:
- kind: ServiceAccount
  name: hawtio-operator
  namespace: hawtio
roleRef:
  kind: ClusterRole
  name: hawtio-operator-ca
  apiGroup: rbac.authorization.k8s.io

That gives:

$ kubectl auth can-i get secrets/signing-key -n openshift-service-ca --as system:serviceaccount:hawtio:hawtio-operator
yes

The problem is to make is work with kustomize. We have to find a way to avoid it changes the openshift-service-ca namespace.

It seems it is possible to work-around kustomize aggresive namespace replacement, by using patches, e.g.:

deploy/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- cluster_role_binding.yaml
- cluster_role.yaml
- ca_role.yaml
- ca_role_binding.yaml
- role.yaml
- role_binding.yaml
- service_account.yaml
- operator.yaml

patches:
- path: patch.json
  target:
    kind: Deployment
- path: patch.json
  target:
    kind: Role
- path: patch.json
  target:
    kind: ServiceAccount
- path: patch.json
  target:
    kind: RoleBinding
    namespace: ^$
- patch: |-
    - op: replace
      path: /subjects/0/namespace
      value: TEST
  target:
    kind: RoleBinding
    namespace: openshift-service-ca

deploy/patch.json:

[
  {"op": "replace", "path": "/metadata/namespace", "value": "TEST"}
]

The new problem is that OLM bundle generation with operator-sdk generate bundle does not seem to managed to take the RoleBinding into account. Actual, that seems impossible to describe such a RoleBinding (from a ClusterRole to a namespace) in the CSV resource. The way to do it seems to include the RoleBinding into the OLM bundle directory, but that has to be tested.

We already enjoyed hawt.io/v1alpha1 for long time, and I think the CRD is already stabilised to some extent. It's good time to move to v1beta1 or v1 depending on how much we expect to add new features or change the API.

https://github.com/hawtio/hawtio-operator/blob/0.5.0/deploy/crd/hawtio_v1alpha1_hawtio_crd.yaml

What are the recommended steps to install the operator from source with OLM? Is it already documented somewhere? If not, it's good idea to document it.

When specifying disabledRoutes to ['camel', 'Camel', 'Quartz'] the respective routes are still accessible in UI and served by Hawtio.

Since one version of a CRD api can be stored in the etcd, other versions are converted into this version. Without a conversion hook, the only migration that takes place is the api version number. In order to facilitate the conversion of properties then a conversion hook should be added and broadcast to the cluster.

Reference: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#before-you-begin

Once we adopt hawtio-online v2 #68, we should release it as 1.0.0.

• #68
• #67

There are some breaking changes since hawtio-operator 0.3.0 (#46, #48). We need to document for <=0.3.0 users the migration steps of their hawtio applications to 0.4.0. Notable changes include:

• CRD apiVersion upgraded from apiextensions.k8s.io/v1beta to apiextensions.k8s.io/v1
• spec.rbac.enabled field is removed from CR and types API. RBAC support is now always on and cannot be turned off.

@astefanutti I think what we really need to document is for users with the following Hawtio CR yaml (either with true or false) to just remove the field from the spec. Is my understanding correct?

spec:
    rbac:
        enabled: {true|false}

Especially in combination with Artemis plugin which has been introduced to hawtio-online recently.

Do

diff pkg/apis/hawtio/v1/hawtio_types.go pkg/apis/hawtio/v1alpha1/hawtio_types.go

and see what diffs there are. Please update v1 types so that it's up to date with the latest type definitions.

Important section of the console config map could be reconcile from the console spec.

Using kubebuilder on the project displayed the following message:

⌁ [: … hawtio-operator] 1.x+* ± kubebuilder version
[Deprecation Notice] This version is deprecated and is no longer scaffolded by default since `28 Apr 2021`.The `go/v2` plugin cannot scaffold projects in which CRDs and/or Webhooks have a `v1` API version.Be aware that v1beta1 API for CRDs and Webhooks was deprecated on Kubernetes 1.16 and areremoved as of the Kubernetes 1.22 release. Therefore, since this plugin cannot produce projects thatwork on Kubernetes versions >= 1.22, it is recommended to upgrade your project to the latest versions available.

Version: main.version{KubeBuilderVersion:"3.13.0", KubernetesVendor:"1.27.1", GitCommit:"c8a7cc58eeb56586c019cf8845dad37286d077ff", BuildDate:"2023-11-02T20:43:44Z", GoOs:"linux", GoArch:"amd64"}

This highlights that the CRD API is using a deprecated version and may well stop working at some point. We should aim to upgrade the project to remove the deprecations.

Since Docker is sunsetting Free Team organisations:
https://web.docker.com/rs/790-SSB-375/images/privatereposfaq.pdf

Currently, the Go version we use is very old. We should bump the version, if there is no problem, to the latest 1.21.0.