Description:
As per spinnaker/spinnaker#5106 (comment), Spinnaker versions from 1.19.0 support Helm version 3 as a rendering engine. Thus, this needs to be evaluated.

Sub Tasks:

• Evaluate the Helm 3 rendering engine in Spinnaker official chart
• Use Helm version 3 client in the Jenkins Docker image
• Code/Peer review

Affected Product Version:
Helm Resources for WSO2 Kubernetes Pipeline for version 1.1.0 and above

Related Issues:
#82

Description:
Include ELK stack for centralized logging from WSO2 product deployments.
In the CI/CD setup we would have a single ELK(elasticsearch and Kibana only) stack. Each of the products being deployed will be having their own log-stash instance that would be pushing logs to this single elasticsearch instance each having a unique node name. This way it is possible to create indexes for each of these nodes and monitor these logs in Kibana.

Description:
jenkins-casc-conf.yaml which contains the configurations for Jenkins Configuration as code has a reference to the shared library. This needs to be removed since it's no longer required.

Fix
Remove the following from jenkins-casc-conf.yaml

unclassified:
      globalLibraries:
        libraries:
          - name: "wso2-jenkins-shared-lib"
            defaultVersion: master
            retriever:
              modernSCM:
                scm:
                  git:
                    remote: "https://github.com/Aaquiff/jenkins-shared-lib"

Description:
Currently, used Spinnaker Helm chart version is 1.22.0. This can be upgraded to the latest stable release version at Helm Hub.

Affected Product Version:
Helm Resources for WSO2 Product Kubernetes Pipeline version 1.1.0 and above

Description:
Spinnaker is a multi-cloud continuous delivery platform that supports Kubernetes deployments. It encapsulates the best practices of cloud deployments and can be used as the deployment tool for better visibility and control.
The following needs to be done

• Add spinnaker chart from stable as a dependency
• Add overridable configurations to configure spinnaker. Including docker registries.
• Use Spin CLI to create pre-configured pipelines for the WSO2 product.
• Integrate with Jenkins which handles the integration.

Description:
It is required to introduce the K8s CI/CD pipeline for Micro Integrator.

Description:
It is required to fix the incorrect Docker repository in the sample MI value YAML.

Description:
In general, Google Kubernetes Engine (GKE) is used as the development Kubernetes environment for Kubernetes/Helm resources.

Kubernetes/Helm resources for WSO2 Kubernetes Pipeline need to be tested with AWS EKS managed service.

Description:
It was identified that once a particular Spinnaker Pipeline deployment fails (i.e. in the given case, a particular deployment environment for a WSO2 product deployment such as, development, staging and production), the product deployment remains in the broken state. No automated rollbacks take place.

Affected Product Version:
Helm resources for WSO2 Product Kubernetes Pipeline version 1.0.0

Description:
Currently, the pipeline deployment does not allow test scripts stored in private GitHub repositories to be used in the deployment. No option has been provided to set the GitHub credentials.

Affected Product Version:
Helm Resources for WSO2 Kubernetes Pipeline version 1.0.0

Description:
Instead of using pipeline stages to trigger subsequent pipelines, automated triggers could be used. This would reduce the number of stages in the pipelines.

Description:
The current hostnames need to be changed to reflect an actual domain.
e.g- jenkins.example.com, spinnaker.example.com

The hostname for Jenkins should also be parameterized to allow for change by end-users.

Description:
As of now, when using WUM based custom images in the pipeline, it was understood that the pulled base images are stored in the Docker local cache of the VM upon which the Jenkins instance is running.

The drawback of this approach is that during the following scenarios, the Jenkins instance may fail to use the base images stored in the local cache during past installations.

• If the underlying Kubernetes cluster VM hosting the Jenkins Pod instance is replaced with a new VM instance
• If the Jenkins Pod instance is updated via Helm and when the new instance spawns up in a new VM instance

Affected Product Version:
Helm resources for WSO2 Kubernetes Pipeline version 1.0.0

Description:

The helm version should be mentioned in the docs. eg. [1]. The given commands don't work with helm v3. It gives this error.

Error: unknown flag: --name

And crd-install hooks are no longer supported in v3.

[1] https://github.com/wso2/kubernetes-pipeline/blob/master/docs/getting-started-mgw.md

Description:
In the current Dockerfile for the Jenkins Docker image doesn't contain specific versions for the tools that are being used. Therefore, it is difficult to maintain a constant state for Jenkins in Docker image builds.
Affected Product Version:
1.0.0

Description:
The Kibana Helm chart dependency can be upgraded to version 7.8.1.

Affected Product Version:
Helm Resources for WSO2 Product Kubernetes Pipeline version 1.1.0 and above

Description:
The Elasticsearch Helm chart dependency can be upgraded to version 7.8.1.

Affected Product Version:
Helm Resources for WSO2 Product Kubernetes Pipeline version 1.1.0 and above

Description:
The newly created images for API Microgateway should be tagged with the current timestamp.

For example, the image savidude/petstore:v1 should be tagged with something like savidude/petstore:v1-20190904092619 when being pushed to the Docker registry through Jenkins.

Affected Product Version:
Helm resources for API Microgateway 3.0.1

Description:
Currently, WSO2 Kubernetes Pipeline Helm chart supports only DockerHub for image storage. This is not ideal for any user who intends to use any other container registry solution (e.g. cloud based container registries, public third party registries or self hosted registries).

Thus, support for using any target container registry for image storage.

Affected Product Version:
Helm Resources for WSO2 Kubernetes Pipeline version 1.0.0

Description:
To maintain consistency with the names of other pods, the elastic search master should be renamed to contain the release name.

Planned Fix:

• Prefix the name with the release name using {{ .Release.Name }}.
• Override values in the Spinnaker pipeline create configs should be changed accordingly.

Description:
It was experienced that the IAM CI/CD Pipeline official website documentation has the following issues:

In the Overview page [1],

• Replicated content: Section Continuous integration and continuous delivery explained is replicated
• The Pipeline's architecture diagram does not display the artifact source repository

In the Getting Started page [2],

• Steps Prepare, Experiment and Customize do not direct to the appropriate page links. It would be better, if the relevant step (or verbs) link to the relevant pages.

[1]: Overview page
[2]: Get Started With CICD Pipeline

Description:
Prometheus fails to start because it is unable to load configurations.

Affected Product Version:
1.0.0

Description:
Jenkins configuration as code loads configuration only at startup. When changing the configmap, the pod needs to restart so that the updated configurations are reflected in Jenkins. This could be done using AUTOMATICALLY ROLL DEPLOYMENTS [1]

[1] https://helm.sh/docs/howto/charts_tips_and_tricks/

Description:
Currently, the daily image update Jenkins Job (i.e. using the updated base image) is supposed to execute as a daily cron job. It has been identified that the frequency should be configurable for the user, as per his/her needs.

Description:
In the current implementation, the Jenkins job to build the Helm chart runs a function to remove all folders named tests inside the dependent charts. This logic fails when there are no dependencies. There should be a check added to the function to make sure that it is not empty before looping over the directory.

Description:
It is required to have a Micro integrator endpoint at the Docker image build stage. As per the current implementation, it is not possible. The given Micro integrator endpoint is used to run remote unit testing at the build stage.

Affected Product Version:
1.1.0

Description:
Jenkins could be used as the integration tool to build docker images.
The following needs to be done,

• To preconfigure Jenkins at startup Jenkins configuration as code could be used.
• JobsDSL could be used to preconfigure the pipeline jobs.

Description:
Deployment spec for API version v1 requires the field selector. This needs to be added.

Description:
The three Grafana dashboards provided by WSO2 to monitor deployments have inconsistent tags. This needs to be corrected.

Description:
When two applications with the same Helm chart are configured together, pipelines in both applications are triggered when the chart is pushed to the webhook endpoint. This can be solved by adding an artifact constraint with the application name and adding the same to the webhook content.

Description:

Dockerfile for the Jenkins image adds additional binaries on top of the Jenkins LTS image. Some of these could be removed to reduce the size of the image.

Description:
When installing the pipeline, the spinnaker-pipeline-creator job fails repeatedly before completing it. This is because the front50 takes a long time to initialize. Adding an init container to wait for the front50 service to initialize would fix the issue.

Description:
It was experienced that currently we need to maintain multiple source repositories (i.e. GitHub repositories) for maintaining the Docker resource source in a pipeline deployment involving multiple WSO2 servers (i.e. one source repository for the customized Docker resources of each WSO2 server).

It was identified that a user may want to host the customized container image source for multiple WSO2 servers in the same source repository.

Affected Product Version:
Helm Resources for WSO2 Kubernetes Pipeline version 1.0.0

Description:
It is required to test the WSO2 product Kubernetes Pipeline Helm Chart with Helm version 3. Currently, it has only been tested with version 2 of Helm.

Affected Product Version:
Helm Resources for WSO2 Kubernetes Pipeline version 1.0.0

Description:
Change Jenkins deployment strategy to Recreate

Description:
Pushing the latest tag of an image does not trigger the pipelines in Spinnaker. Therefore we can always push the latest tag of the image. This solves the issue of manually pushing the latest tag when installing the pipeline.

Description:
Jenkins update strategy should be recreate since the volume does not support read write many.

Description:
It was noticed that the base Jenkins Docker Image used in the Jenkins Docker image for WSO2 Kubernetes Pipeline is outdated, as of 2020/05/25.

Further, the Jenkins plugin versions are outdated, as well. The following error logs are observed during the Jenkins Server startup.

INFO: Listed all plugins
May 24, 2020 9:29:38 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading plugin Lockable Resources plugin v2.8 (lockable-resources)
java.io.IOException: Lockable Resources plugin v2.8 failed to load.
 - Structs Plugin v1.17 is older than required. To fix, install v1.20 or later.
 - Script Security Plugin v1.53 is older than required. To fix, install v1.62 or later.
 - Pipeline: Supporting APIs v3.2 is older than required. To fix, install v3.4 or later.
 - Matrix Project Plugin v1.13 is older than required. To fix, install v1.14 or later.
	at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:652)
	at hudson.PluginManager$2$1$1.run(PluginManager.java:517)
	at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
	at jenkins.model.Jenkins$5.runTask(Jenkins.java:1083)
	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

May 24, 2020 9:29:39 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading plugin Pipeline v2.6 (workflow-aggregator)
java.io.IOException: Pipeline v2.6 failed to load.
 - Lockable Resources plugin v2.8 failed to load. Fix this plugin first.
	at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:652)
	at hudson.PluginManager$2$1$1.run(PluginManager.java:517)
	at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
	at jenkins.model.Jenkins$5.runTask(Jenkins.java:1083)
	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

These need to be updated.

Affected Product Version:
WSO2 Product Kubernetes Pipeline version 1.0.0

Description:
It was identified that some Helm charts are implemented with no Kubernetes Namespace defined under metadata (e.g. reference API). By default, Spinnaker does not set the Namespace on the specification, thus leading to deployment of these Kubernetes resources in undesired Namespaces.

This can be avoided by spinnaker/deck#7016 in Spinnaker.

Affected Product Version:
Helm Resources For WSO2 Product Kubernetes Pipeline version 1.0.0

Description:
The default values.yaml needs to be documented through comments.

Description:
Upgrade Prometheus operator and Spinnaker charts - 2020/08/18

Description:
$subject instruction can be added to the Jenkins CI Docker image rather than requesting the user to copy the Microgateway toolkit pack, manually to the build context.

The default URL could be set to the latest version of Microgateway Toolkit available in GitHub.

Affected Product Version:
WSO2 Kubernetes Pipeline version 1.1.0 and above

Description:
The docker registry, and organization name should be obtained from the deployment config file such that a tag could be created in order to push to the Docker registry.

Affected Product Version:
Helm resources for API Microgateway 3.0.1

Description:
It was noticed that the automatic rolling update upon pipeline related Kubernetes Secrets changes does not work (e.g. updates to GitHub credentials, WSO2 Subscription credentials or target container registry credentials).

This needs to be fixed.

Affected Product Version:
Helm Resources for WSO2 Kubernetes Pipeline version 1.0.0

Description:
A change log should be introduced to this repository to indicate the changes that occur across each release performed.

The format suggested at [1] can be used in the change log.

[1]: Change log format

Description:
Use Prometheus operator to setup Prometheus and Kibana for monitoring WSO2 product deployments. prometheus chart can be used as a dependency to set up these tools.
The following needs to be done,

• prometheus-blackbox-exporter could be used to retrieve the login status of the deployments.
• preconfigured grafana dashboards will be packaged with the pipeline that could read from the serviceMonitors that are deployed by the product helm chart installations.

Description:
Current, Jenkins CI Pipeline Docker image packages WSO2 Microgateway version 3.0.1 toolkit.

As of 2020-08-31, the version 3.2.0 is available.

Affected Product Version:
WSO2 Kubernetes Pipeline version 1.1.0 and above

Description:

I'm getting below error at helm install step.

[16:51:27] bhathiya@MacBookPro:/data/techs/kube-pipeline/darwin-amd64$ ./helm install --name apim3v1 wso2/kubernetes-pipeline -f ../values-mgw.yaml --namespace wso2
2019/12/03 17:56:20 Warning: Merging destination map for chart 'prometheus-node-exporter'. Overwriting table item 'extraArgs', with non table value: [--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/) --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$]
2019/12/03 17:56:20 Warning: Merging destination map for chart 'prometheus-node-exporter'. Overwriting table item 'extraArgs', with non table value: [--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/) --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$]
2019/12/03 17:56:20 Warning: Merging destination map for chart 'prometheus-node-exporter'. Overwriting table item 'extraArgs', with non table value: [--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/) --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$]
2019/12/03 17:56:20 Warning: Merging destination map for chart 'prometheus-node-exporter'. Overwriting table item 'extraArgs', with non table value: [--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/) --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$]
Error: timed out waiting for the condition

Description:
It was noted that the current Helm version 2 update used in Jenkins is v2.14.1.

As of now (2020/07/23), the latest Helm version 2 update is v2.16.7. The deployment needs to be evaluated by upgrading to this version.

Affected Product Version:
Helm Resources for WSO2 Kubernetes Pipeline for version 1.1.0 and above

Description:
It should be possible to configure TLS for the ingress resource in the Helm chart and sub-charts.
Some ingress endpoints are,

• Jenkins
• Spinnaker Deck
• Spinnaker Gate
• Kibana
• Grafana