Giant Swarm Testing Infrastructure
Getting started
Interacting with our test-infra setup is currently done exclusively through the releases repository. On this repository you can trigger new test runs through comments on Pull Requests.
Please be aware that all interactions with test-infra are designed with the idea in mind that a pull request will either modify or add a single release in the releases repository! Commands which are currently available are as follows:
/test cncfruns thecncfsuite against the release./test cisruns thecissuite against the release./test awsrunsawscnfmagainst the release - this only works for AWS.
There are multiple dashboards which can be useful to track the progress of tests:
- prow gives a rough overview of the running tests.
- tekton gives a more in depth overview of the test progress,
- kibana gives full insight of the logs emitted during your test.
Each test will generally perform the same flow currently which can be summarized as this:
- Determine release CR from PR diff.
- Create release CR on provider CP (
gaia,godsmack,gorgoth). - Create tenant cluster with created release on target CP.
- Run tests against created tenant cluster.
- Report test results.
- Delete tenant cluster.
- Delete release on CP.
Feel free to reach out to team ludacris if you have further questions.
Developing test-infra
You can find guides for the creation of new tests in the docs subfolder.
Local development
A lot of functionality of tekton is available locally - the biggest exceptions being anything requiring an Ingress of webhooks.
Tools required for local development:
- KIND
- Kustomize
- Kubectl
- Giant Swarm internal tooling for secrets
To setup tekton in a KIND cluster you can simply do the following:
- Create a cluster with
kind create cluster - Apply all necessary files with
./tekton-kind.sh
Hack away!
In our ops cluster
Deploying test-infra is currently a bit cumbersome as it is running as an app inside a tenant cluster.
The tenant cluster is rfjh2 on gorilla.
To update test-infra you will need to manually edit the app CR on the gorilla CP in the namespace rfjh2.
There is currently no automatic rollout by merging to master!
The roles and serviceaccounts on the target CPs are currently created manually.
You can find the resources here.
Kubeconfigs to the target CPs are currently generated with a script using opsctl and then manually updated to rfjh2.
You can find the necessary scripts here.
Included components
This repository contains manifests for our individual test pipelines as well as the components we run to build up our testing infrastructure. The two main open source projects used are Prow and Tekton
Prow
Prow is a CI/CD system running on Kubernetes. We utilize prow to trigger tests and have a nice Github integration to run high cost conformance tests.
Components
This is a list of components which we run:
Deck- UI for prow jobs.Sinker- Clean up stale prow jobs.Hook- Handle GitHub events dispatching them to plugins.Horologium- Start periodic jobs.Plank- Start prow jobs.ghProxy- Designed to reduce Github API token usage.Tide- Automatically retest PRs & merge them if tests go green.Crier- Reports your prowjob status changes.Pipelines- Automatically creates Tekton pipelines for each Prow job.
Tekton
Tekton is a CI/CD system which is highly integrated with Kubernetes. We utilize it to actually run tests and to structure our test stages.
Components
This is a list of components which we run:
Pipelines- This enables the meat of functionalities by Tekton.Dashboard- Gives and overview of active Tekton tasks and pipelines.prow-log-aggregator- Collects logs from Tekton tasks and makes them available to Deck.
Reference
- How to write
ProwJobs
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "github-actions[bot]")
![renovate[bot] avatar](https://avatars.githubusercontent.com/in/2740?v=4 "renovate[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent