openfaas / certifier Goto Github PK

faasd now has memory limit setting and reading, could we add a test for that?

cc @LucasRoesler @nitishkumar71

Refactor the test cases in the secretCRUD_test.go to use the same table driven test pattern introduced in #66 so that we can run the tests in the default "empty" namespace and a non-default namespace when it is configured.

Part of #61

Add instructions to the readme for how to run this with faas-netes and faas-swarm locally

We should verify the log provider behavior is implemented certified openfaas provides and covered by the certifier

Switch to faas-cli SDK where possible for invoking endpoints.

We could migrate from dep to Go modules like we are doing with many of the other projects, e.g. openfaas/faas-netes#585

The certifier currently does not know about multi-namespace support in faas-netes. Additionally, we would like to eventually bring namespace support to faasd. We should update the certifier to validate that multi-namespace support works as expected. It is likely to find bugs for us and help verify when faasd is done with its own version of namespace support.

I think the ideal flow for this is to add a new config or env variable that is a list of namespaces, for exmaple

CERTIFIER_NAMESPACES=testa,testb,testc

when this value is set, the tests (where it makes sense) would run multiple times, one per namespace in the list.

When the value is empty, the certifier should default to its current behavior.

The production use-case is to use auth, so can we move to using auth?

From the README:

The tests assume a local environment with basic authentication turned off.

There are no tests for http redirection use cases using the golang-middleware template

Description

Add support for running with a limited feature-set

Context

Not all providers support all advanced features such as: logs, secrets and namespaces.

We should have a way of being able to segregate or select the suites of tests that are run based upon some simple criteria or configuration.

Case in point

faasd support function CRUD, and could be tested with the certifier, but will fail due to lack of autoscaling and secrets support at present. It is still useful to run the tests on CRUD.

https://github.com/alexellis/faasd/

We should run the tests with both Swarm and Kubernetes, that way we can tell if changes made to this project are valid or not before merging.

For Swarm install Docker as a travis package and run docker swarm init, clone openfaas/faas, then run deploy_stack.sh --no-auth

For Kubernetes k3sup can install k3s locally, then openfaas after that.

You'll need to remove Swarm in between the two tests docker swarm leave --force, should clean up everything.

Write a bash script and have it called by .travis.yml - https://github.com/openfaas/certifier/blob/master/.travis.yml

Description

Remove references and code paths for Swarm which has been deprecated for around half a year now.

The README and/or scripts may need an update.

Secret CRUD test was added to the test suite, but it is missing a test that shows function is rejected when secret is missing. See more information in PR #22 comments.

After reviewing the gateway api spec https://github.com/openfaas/faas/blob/master/api-docs/swagger.yml

I think the info endpoint is one of the only remaining endpoints that is not minimally tested

We should add a test that the provider returns non-empty info values, at the least.

We should add test to require the "image" field in the function status endpoint /system/function/NAME

Currently, it's missing in faasd and went undetected: openfaas/faas-cli#914

Each provider that supports multiple namespaces needs to support list available namespaces, the endpoint is /system/namespaces. This should be the first test case we add for multinamespace support as part of #61

https://github.com/openfaas/faas-provider/blob/bc0f26cae70c1b3ba2a595cea8bfb8c04925d5c6/serve.go#L68

For faas-netes the implementation is here https://github.com/openfaas/faas-netes/blob/e5cf64dcf4d01f04636d2ffe736be110981b83c5/pkg/handlers/namespaces.go#L22-L48 It requires deployment with a ClusterRole and annotations added to the permitted namespaces.

We need to be able to confirm both of these cases:

1. when deployed with multi-namespace support disabled, the endpoint should only return a single namespace, the default namesapce (usually openfaas-fn)
2. when deployed with multi-namespace support enabled, it should return it should return each of the configured namespaces and it should also include the default namespace.

I don't think it requires writing multiple test cases, rather the ideal situation is that the one test can work to run against deployments/configurations.

Description

Test faasd with the certifier

Details

Some tests won't work such as scaling above 1 replica, however scaling from zero will work.

Whoever takes up the task will need to recommend a way of selectively running certain tests, or selectively not running certain tests.

Context

This work will help ensure faasd has as much parity as possible with faas-netes (OpenFaaS on Kubernetes)

A GitHub Action should work here, like it did with Swarm in the past.

Continuing with #61, let's add tests that verify that deploying a function works when multiple namespaces are enabled. A couple things we would really like to have

1. we still want to test when the namespace value is omitted, this is the implied "deploy to default namespace"
2. we probably don't need to test all of the non-default namespaces, so we are probably ok with just testing in the first value of config.Namespaces
3. ideally, these additional tests should be controlled by the config.Namespaces so that simply providing the CERTIFIER_NAMESPACES is enough to enable the additional tests. Similarly, if CERTIFIER_NAMESPACES is empty or does not exist, then it only tests the implied default namespace, see (1) above. Ideally, this means we do not need any additional configuration to control this.
4. the test Test_Deploy_PassingCustomEnvVars_AndQueryString actually looks like it should be in the invoke_test.go let's move and rename that appropriately. It isn't actually testing the deployment or configuration of a Function, rather it is testing the support for GET parameters during invoke.
5. Similarly, this since we have a separate file/flow for testing invoking, we don't need to have any function invocation in the deploy tests. For example, i think we can omit this line

   certifier/tests/deploy_test.go

   Line 91 in bad219a

   _ = invoke(t, functionRequest.FunctionName, emptyQueryString, http.StatusOK)

6. Finally, one last improvement we could make to the tests is to consolidate the Test_Deploy_Stronghash, Test_Deploy_WithLabels, and Test_Deploy_WithLabels into a single table driven test. We can improve the tests in general by just always checking the name, labels, annotations, etc. For example, it is valid to verify that if the test case has an empty label set, then the deploy response should still be empty.

Putting all of that together, i think we end up with a single table-driven test TestDeploy where we have a sub-test for (at the minimum)

• the minimal deployment, default and non-default namespace
• deployment with non-empty labels, default and non-default namespace
• deployment with non-empty annotations, default and non-default namespace

We can probably create this by creating the initial list of test cases

cases := []struct{
   name string
   req sdk.DeployFunctionSpec
}{
   // cases here 
}

and then we can just loop over it to add a copy of each test with the non-default namespace

defaultCases := len(cases)
for i := 0; i < defaultCases; i++ {
    newCase := cases[i]
    // rename the case
    // set the req Namespace
    cases = append(cases, newCase)
}

Refactor the test cases in the invoke_test.go to use the same table driven test pattern introduced in #66 so that we can run the tests in the default "empty" namespace and a non-default namespace when it is configured.

Part of #61

Description

Add support to test faasd in the certifier

Some tests may not work like scaling to zero or above 1 replica

Refactor the test cases in the scaling_test.go to use the same table driven test pattern introduced in #66 so that we can run the tests in the default "empty" namespace and a non-default namespace when it is configured.

Part of #61

Kubernetes tests are failing in master

https://github.com/openfaas/certifier/runs/3856442402?check_suite_focus=true

For both Swarm and Kubernetes (via Makefile)

Create a secret and then deploy a function which consumes that secret.

Make sure that the function can echo the value back.

Make this re-runnable (i.e. it will manage the secrets)

We just need to add basic auth if let's say env vars like basic_auth_user and basic_auth_password are present

Use /system/scale-function for scaling .

Test cases:
1x without limits
1x with limits
1x autoscaling disabled
1x 0 to 1 replicas

With the successful refactor of the travis ci flow, the run-e2e-test-*.sh are no longer used and can safely be removed. Removing these scripts helps keep the repo tidy and removes any future questions/distractions about what needs to be maintained

Testing a function that can access a secret is now covered by first step of Test_SecretCRUD. We should remove Test_Access_Secret and all of the related setup/cleanup code to help simplify the certifier code base

At the same time, it might be good to break the Test_SecretCRUD into subtests using t.Run to make it clearer in the test logs and in the codebase itself that this Test_Access_Secret case is covered.

It would be really great to have versioned binaries that we can then use in CI systems instead of needing to clone and run this repo. Having a built binary means we also don't need to worry about the go environment the user has.

we can use

go test -c ./tests -o faas-certifier

and this will produce a binary called faas-certifier for the tests.

We should add a scenario to test the new secret CRUD behaviour added in gateway 0.9.14 and the corresponding faas-swarm / faas-netes / openfaas-operator versions.

Suggested test scenario:

• Create
• Invoke (function that prints the secret)
• List
• Update
• Invoke (function that prints the secret)
• Delete

Description

Auto-scaling e2e tests are required to check for compliance and to test regression.

Each test case can be submitted in a separate PR starting with the simplest (1st) test-case.

Test case: min scale

When using the com.openfaas.scale.min label, there should be a minimum number of replicas available via the API.

Test case: scale from zero

1. Deploy function
2. Scale to zero
3. Wait until no ready replicas
4. Invoke function and make sure result is returned as 200

Test case: auto-scaling up > min, then returning down to min

Test case may take up to 30s or 1m to complete, but can be run easily.

1. Deploy a function with min = 1, max = 2
2. Invoke function in a loop
3. Wait until replicas > min
4. Stop invoking function
5. Wait until function replicas returns to min

Refactor the test cases in the logs_test.go to use the same table driven test pattern introduced in #66 so that we can run the tests in the default "empty" namespace and a non-default namespace when it is configured.

Part of #61

Description:
As we are moving all the repo to dep vendoring tool. Here is the proposal openfaas/faas/issue#462. This repo also should use dep.

For help: openfaas/faas-cli#282. please check this.