Giter Club home page Giter Club logo

hcunit's Introduction

hcunit

CircleCI

Helm Chart Unit: helps to unit test rendering of your templates using policies

Usage as a Helm Plugin:

$> echo "install the latest version of the plugin"
$> helm plugin install https://github.com/xchapter7x/hcunit/releases/latest/download/hcunit_plugin.tgz
Installed plugin: unit

$> echo "you might have have to make the plugin binaries executable"
$> helm env | grep "HELM_PLUGIN" | awk -F"=" '{print $2}' | awk -F\" '{print "chmod +x "$2"/hcunit_plugin/hcunit*"}' |sh

$> echo "lets run some tests of our templates' logic"
$> helm unit -t templates -c policy/values_toggle_on.yaml -p policy/testing_toggle_on.rego
PASS: data.main.expect["another passing case 123"]
PASS: data.main.expect["force passing"]
PASS: data.main.expect["another passing case"]
PASS: data.main.expect["force passing abc"]
[SUCCESS] Your Helm Chart complies with all policies!

$> echo "lets explore the available flags for the plugin call"
$> helm unit --help
Usage:
  hcunit_osx [OPTIONS] eval [eval-OPTIONS]

given a OPA/Rego Policy one can evaluate if the rendered templates of a chart using a given values file meet the defined rules of the policy or not

Help Options:
  -h, --help           Show this help message

[eval command options]
      -t, --template=  path to yaml template you would like to render
      -c, --values=    path to values file you would like to use for rendering
      -p, --policy=    path to rego policies to evaluate against rendered templates
      -n, --namespace= policy namespace to query for rules
      -v, --verbose    prints tracing output to stdout
      

Usage as a Standalone CLI... Download Binaries

https://github.com/xchapter7x/hcunit/releases/latest

Notes on Syntax and Rego

Rego is a Policy Language for the Open Policy Agent eco system. We use rego here as our testing DSL. Any rego rule which is an assert or expect will get executed and must evaluated to true. The gist is that everything between the {} is a rule. Everything between {} should evaluate to true. Assignments yield true, and if any statement in the {} block is false then the entire rule will return false and therfore fail our test case.

For more information you can try: https://www.openpolicyagent.org/docs/latest/#rego

Or

for a Online playground: https://play.openpolicyagent.org/

Options

-> % hcunit --help
Usage:
  hcunit [OPTIONS] <eval | render | version>

Help Options:
  -h, --help  Show this help message

Available commands:
  eval     evaluate a policy on a chart + values
  render   Render a template yaml
  version  display version info

Sample usage

000@000-000 [00:00:00] [helm-charts/concourse] [master *]
-> % cat policy/testing.rego
───────┬───────────────────────────────────────────────────────────────
       │ File: policy/testing.rego
───────┼───────────────────────────────────────────────────────────────
   1   │ package main
   2   │
   3   │ assert ["this should always be true b/c its true"] {
   4   │   true
   5   │ }
   6   │
   7   │ assert ["when web is enabled then namespace is toggled on"] {
   8   |     "true" == input["values"].web.enabled
   9   │     "Namespace" == input["namespace.yaml"].kind
   10  │ }
───────┴───────────────────────────────────────────────────────────────

000@000-000 [00:00:00] [helm-charts/concourse] [master *]
-> % hcunit eval -t templates/ -c values.yaml -p policy/testing.rego
PASS: data.main.assert["this should always be true b/c its true"]
PASS: data.main.assert["when web is enabled then namespace is toggled on"]
[SUCCESS] Your Helm Chart complies with all policies!

000@000-000 [00:00:00] [helm-charts/concourse] [master *]
-> % cat policy/testing_fail.rego
───────┬───────────────────────────────────────────────────────────────
       │ File: policy/testing_fail.rego
───────┼───────────────────────────────────────────────────────────────
   1   │ package main
   2   │
   3   │ assert ["this should always be true b/c its true"] {
   4   │   false
   5   │ }
   6   │
   7   │ assert ["when web is enabled then namespace is toggled on"] {
   8   |     "true" == input["values"].web.enabled
   9   │     "NamespaceWrongKind" == input["namespace.yaml"].kind
   10  │ }
───────┴───────────────────────────────────────────────────────────────

000@000-000 [00:00:00] [helm-charts/concourse] [master *]
-> % hcunit eval -t templates/ -c values.yaml -p policy/testing_fail.rego
FAIL: data.main.assert["this should always be true b/c its true"]
FAIL: data.main.assert["when web is enabled then namespace is toggled on"]
[FAILURE] Policy violations found on the Helm Chart!

About hcunit

  • Uses OPA and Rego to evaluate the yaml to see if it meets your expectations
  • By convention hcunit will run any rules in your given rego file or recursively in a given directory as long as that rule takes the form assert ["some behavior"] { ... } or expect ["some other behavior"] { ... } .
  • using variables or duplicate values in the hash for your tests is prohibited by hcunit. Reason being duplicate hashes opens up the potential for inconsistent/confusing results.
  • Your policy rules will have access to a input object. This object will be a hashmap of your rendered templates, with the hash being the filename, and the value being an object representation of the rendered yaml. It will also contain a hash for the NOTES file, which will be a string.
  • uses helm's packages to render the templates so, it should yield identical output as the helm template command
  • supports multiple values.yml file inputs, does not yet support values set as flags in the cli call.

hcunit's People

Contributors

xchapter7x avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Forkers

syllogy

hcunit's Issues

Installation procedure fails with helm v3.4.0

Describe the bug

As a user

  • in order to use hcunit
  • I need to install it

To Reproduce

Steps to reproduce the behavior:

  1. Follow procedure at https://github.com/xchapter7x/hcunit#usage-as-a-helm-plugin
  2. Execute helm plugin install https://github.com/xchapter7x/hcunit/releases/latest/download/hcunit_plugin.tgz

Expected behavior

A clear and concise description of what you expected to happen.

Installed plugin: unit

Observed behavior

$ helm plugin install https://github.com/xchapter7x/hcunit/releases/latest/download/hcunit_plugin.tgz
Error: Unable to get repository: Cloning into '/data/shared/gberche/.cache/helm/plugins/https-github.com-xchapter7x-hcunit-releases-latest-download-hcunit_plugin.tgz'...
remote: Please upgrade your git client.
remote: GitHub.com no longer supports git over dumb-http: https://github.com/blog/809-git-dumb-http-transport-to-be-turned-off-in-90-days
fatal: unable to access 'https://github.com/xchapter7x/hcunit/releases/latest/download/hcunit_plugin.tgz/': The requested URL returned error: 403
: exit status 128

I also tried to download the plugin and install it locally

$  curl -LO https://github.com/xchapter7x/hcunit/releases/latest/download/hcunit_plugin.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0    368      0 --:--:-- --:--:-- --:--:--   368
100   643  100   643    0     0   1030      0 --:--:-- --:--:-- --:--:--  1030
100 22.9M  100 22.9M    0     0   343k      0  0:01:08  0:01:08 --:--:--  277k

$ helm plugin install --debug ./hcunit_plugin.tgz

Error: plugin metadata (plugin.yaml) missing
helm.go:81: [debug] plugin metadata (plugin.yaml) missing
helm.sh/helm/v3/pkg/plugin/installer.init
	/home/circleci/helm.sh/helm/pkg/plugin/installer/installer.go:32
runtime.doInit
	/usr/local/go/src/runtime/proc.go:5453
runtime.doInit
	/usr/local/go/src/runtime/proc.go:5448
runtime.main
	/usr/local/go/src/runtime/proc.go:190
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1373

I also tried the git repo syntax

$ helm plugin install https://github.com/xchapter7x/hcunit --version v0.7.5
Error: plugin metadata (plugin.yaml) missing

workaround

$  curl -LO https://github.com/xchapter7x/hcunit/releases/latest/download/hcunit_plugin.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0    368      0 --:--:-- --:--:-- --:--:--   368
100   643  100   643    0     0   1030      0 --:--:-- --:--:-- --:--:--  1030
100 22.9M  100 22.9M    0     0   343k      0  0:01:08  0:01:08 --:--:--  277k

$ mkdir hcunit_plugin
$ (cd hcunit_plugin;  tar xvfz  ./hcunit_plugin.tgz)
$ helm plugin install --debug ./hcunit_plugin
Installed plugin: unit

Desktop (please complete the following information):

  • OS: "Ubuntu 18.04.5 LTS" (bionic)
  • Browser [e.g. chrome, safari]
  • helm version: version.BuildInfo{Version:"v3.4.0", GitCommit:"7090a89efc8a18f3d8178bf47d2462450349a004", GitTreeState:"clean", GoVersion:"go1.14.10"}

Additional context
Add any other context about the problem here.

suggested suite implementation

Future Suite Functionality

hcunit will traverse the test directory and for each _test.yml & _test.rego pair will use the .yml file as a values input for rendering and run the corresponding rego policy against the rendered templates. the input object made available in the rego policy will be a hashmap with keys using the paths of the rendered templates and the values file. The corresponding value in the hashmap being the object representation of the files.

Chart Dir Convention

chart
│   README.md    
│
└───templates
│   │   NOTES.txt
│   │   _helpers.tpl
│   │   web-deployment.yaml
│   │   web-ingress.yaml
│   
└───test
    │   values_invalid_test.yml
    │   values_invalid_test.rego
    │   values_valid_test.yml
    │   values_valid_test.rego 
    │   values_scenariob_test.yml
    │   values_scenariob_test.rego

when array index is a variable it breaks failure check

Describe the bug
when array index is a variable it breaks failure check

To Reproduce
Steps to reproduce the behavior:

  1. create a failing rule
  2. use a variable for the index

Expected behavior
a clear error on that rule

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context
The reason for this is that the array index will not necessarily be unique, so the query to run the tests will match several rules, and therefore we will see a match for the query in the result set. We have previously assumed a 1-1 on each query - result. in the event of duplicate indexes there will be a conflict and inconsistent failure results.

All failing policies show no policies or output

Describe the bug
If there are no policies which pass, then the output should show all failing policies, however it shows the below error.

To Reproduce
Steps to reproduce the behavior:

  1. Run any set of policies which all fail
  2. you will see output that "your given query did not yield any matches"

Expected behavior
All failing tests should be listed as failing and status should be out put to the screen.

Screenshots

-> % helm unit -t templates/ -c values.yaml -c policies/values_test.yaml -p policies -v
your given query did not yield any matches
Error: plugin "unit" exited with error

Additional context
Add any other context about the problem here.

Support asserting helm rendering errors

Is your feature request related to a problem? Please describe.

As a helm chart author

  • in order to assert input validation code that trigger rendering error (e.g. using the required function
  • I need hcunit to support assertions on rendering error output

Describe the solution you'd like

Given the following template

apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceInstance
metadata:
  namespace: {{ required "A valid namespace is required" .Values.namespace }} # Mandatory since ServiceInstance is a namespaced resource

When helm unit runs with a value missing namespace,

then a render_error hash in the input such as the following is available for assertions to check:

template: "hcunit/templates/serviceinstance.yaml"
message: "error calling required: A valid namespace is required"
statement: "required \"A valid namespace is required\" .Values.namespace>"
line: 5
column: 16

Note: statement, line and column are "nice-to-have". Policies would really need to assert on template and message.

Describe alternatives you've considered

Ignore rendering errors in tests, see for instance test launcher in sample project and associated output below

----------------
Test suite ./unit-tests/mandatory-namespace with user inputs: ./unit-tests/mandatory-namespace/user-inputs/empty-user-inputs.yml  ./unit-tests/mandatory-namespace/user-inputs/mocked-lookup-values.yml 

Checking expected rendering failure in ./unit-tests/mandatory-namespace/policy/service_instance_should_fail_to_render.rego
error while rendering: render error in "hcunit/templates/serviceinstance.yaml": template: hcunit/templates/serviceinstance.yaml:5:16: executing "hcunit/templates/serviceinstance.yaml" at <required "A valid namespace is required" .Values.namespace>: error calling required: A valid namespace is required
Error: plugin "unit" exited with error
helm.go:81: [debug] plugin "unit" exited with error
Ok rendering was rejected as expected

Additional context
Add any other context or screenshots about the feature request here.

Nested templates do not seem to be available to tests

I have a chart that has the following directory structure:

templates/
-- configmaps/
---- configmap1.yaml
---- configmap2.yaml
-- routes/
---- route1.yaml
---- route2.yaml
-- secrets/
---- secrets1.yaml
---- secrets2.yaml
---- secrets3.yaml

If I was to run the test in the following format:
helm unit -t templates/ -c values.yaml -p tests/policies/

how can i reference route1.yaml in my test? I tried both of the following and neither was successful:

`package main

assert ["test 1"] {
input["route1.yaml"]
}`

and

`package main

assert ["test 1"] {
input["routes/route1.yaml"]
}`

If I move route1.yaml into the root of the templates directory the test will pass by referencing it as "route1.yaml"

Potential import collision: import path should be "k8s.io/helm", not "github.com/helm/helm".

Background

I find that k8s.io/helm and github.com/helm/helm coexist in this repo:
https://github.com/xchapter7x/hcunit/blob/master/go.mod (Line 15 & 31)

github.com/helm/helm v2.14.3+incompatible 
k8s.io/helm v2.14.3+incompatible 

The helm/helm has already renamed it’s import path from "github.com/helm/helm" to "k8s.io/helm".
https://github.com/helm/helm/blob/v2.14.3/pkg/chartutil/values_test.go#L31

package chartutil
import (
	…
	"k8s.io/helm/pkg/proto/hapi/chart"
	"k8s.io/helm/pkg/timeconv"
	"k8s.io/helm/pkg/version"
)
…

The "k8s.io/helm" and "github.com/helm/helm" are the same repos. This will work in isolation, bring about potential risks and problems.

So, why not get rid of the old import path "github.com/helm/helm", use "k8s.io/helm" instead.

Solution

Replace all the old import paths, change "github.com/helm/helm" to "k8s.io/helm".
Where did you import it: https://github.com/xchapter7x/hcunit/search?q=github.com%2Fhelm%2Fhelm&unscoped_q=github.com%2Fhelm%2Fhelm

templates using the lookup function fail to render

Describe the bug

As a hcunit user

  • in order to assert templates making use of the lookup function
  • I need hcunit to support this keyword

To Reproduce

With the following serviceinstance.yml template

apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceInstance
  {{ $serviceInstance:= ( lookup "servicecatalog.k8s.io/v1beta1" "ServiceInstance" .Release.Namespace  $fullName) -}}
metadata:
  namespace: {{ .Release.Namespace }}
  name: {{ include "p-mysql.fullname" . }}
  labels:
  {{- include "p-mysql.labels" . | nindent 4 }}
spec:

  #Let svcat reference service class and service plans from external names (i.e. OSB names)
  #See https://github.com/kubernetes-sigs/service-catalog/blob/a204c0d26c60b42121aa608c39a179680e499d2a/contrib/examples/walkthrough/mini-instance.yaml#L1-L11
  clusterServiceClassExternalName: {{ .Values.serviceClassName }}
  clusterServicePlanExternalName: {{ .Values.servicePlanName }}

when running

 helm unit eval -t templates/ -c values.yaml -p my_assert.rego

then

error while rendering: parse error in "hcunit/templates/serviceinstance.yaml": template: hcunit/templates/serviceinstance.yaml:3: function "lookup" not defined
Error: plugin "unit" exited with error
helm.go:81: [debug] plugin "unit" exited with error

on version 0.7.5

-v on failing tests doesnt actually print verbose output

Describe the bug
the -v flag should print the verbose output and all the trace information.
This is true when all of the tests are passing, but when one fails, the verbose output is suppressed.

To Reproduce
Steps to reproduce the behavior:

  1. hcunit eval -v
  2. on a failing tests
  3. no verbose output will be printed

Expected behavior

  1. hcunit eval -v
  2. on a failing tests
  3. verbose output will be printed (just as it is when all tests pass w/ -v flag)

Dump input policy on failure in debug mode

Is your feature request related to a problem? Please describe.

As a hcunit user

  • in order to troubleshoot failed assertion
  • I need an easy way to inspect the content of the input object against which policies are executed

Describe the solution you'd like

when --debug flag is set (possibly with an additional --dump-input), then upon a policy failure, then the input structure is dumped to stdout as yaml (which includes all hashes, i.e. rendered templates, values and potential others in the future)

Describe alternatives you've considered

Additional context
Add any other context or screenshots about the feature request here.

Support rendering default templates relying on helpers

Is your feature request related to a problem? Please describe.

As a hcunit user

  • in order to unit tests helm charts generated by the helm create command which include a dependency to _helpers.tpl template helper
  • I need hcunit to support specifying a template helper path

Observed behavior

at <include "my-chart.fullname" .>: error calling include: template: no template "my-chart.fullname" associated with template "gotpl"

where fullname is defined in the _helpers.tpl template helper

{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "p-mysql.fullname" -}}
{{- if .Values.fullnameOverride }}
[...]

Describe the solution you'd like

I'd like to be able to render templates in unit tests for debugging purposes, without having access to a full-fledge kubernetes instance (e.g. as part of circle ci unit tests where a KIND is not yet available)

Describe alternatives you've considered

I've worked around the issue using helm install my-render-test --dry-run --debug ./my-chart/ --values ./my-chart/values.yaml on an environment with access to a K8S server

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.