Description

If during setup suite we get unexpected failure (e.g. fatal is happen), we don't process it properly.
In this case, we may not have a test name, and this event is skipped now.

Motivation

Cloudtest tool can schedule only single tests. We need to add support for testify suites.

Usecase

Input:

1. We have cloudtest configured to work with two clusters.
2. We have a test based on testify suites(suite contains 8 subtests).

Steps:

1. Cloudtest finds tests
2. Cloudtest started work
3. ...
4. Cloudtest realizes that the next task to run is testify suite.
5. Cloudtest splits the testify suite to two tasks for two clusters(the first task contains 4 subtests and the other task contains 4 subtests)
6. cluster1 runs testifysuite with 4 tests
7. cluster2 runs testifysuite with 4 tests

Solution

Example of run single test from the suite

go test -testify.m TestA -v

List of all tests in the suite

go test -testify.m “.*” -test.list “.*”

Currently, nothing happens if a test specified on 'only-run' section of config file (or on command line) is not found during test search.

An error should be produced.

I want to create two kind clusters one after another and interacting with each other, is it possible to run go tests after that.
Based on my understanding the clusters are created/deleted as well as the tests are executed independently.
I want to run multiple cluster tests.
Please suggest any tags/fields.
Thanks.

Since we do not need CircleCI for this kind of build, let's try github actions.

Expected

If cloudtest config has only-run tests then cloudtest will run only these tests.

Actual

Cloudtest ignores only-run tests

Commit

networkservicemesh/networkservicemesh@09166bf
networkservicemesh/networkservicemesh@8837ac2 (take a look at CI result)

Right now CloudTest validation is only based on Kubernetes APIs, we need to make it generic to be able to use with any kind of cloud.

.cloudtest.yaml config could contain kubernetes-env keys to pass KUBECONFIG variables to individual integration tests, we need to name it cloud-config like to make generic.

Motivation

Some stuff like this https://github.com/networkservicemesh/cloudtest/blob/master/pkg/utils/shell_utils.go#L41 can be reworked/simplified with using https://github.com/edwarnicke/exechelper OR https://github.com/commander-cli/cmd

This might prevent proper artifact saving for failed tests

Issue

Currently packet provider is not covered with test because there are no stubs for the packet API.

Solution

Create packet_test package with stubs. Ex:

type Option func(*packngo.Client)

func NewClient(options ...Option) *packngo.Client {
    client := new(packngo.Client)

    for _, option := range options {
        option(client)
    }

    return client
}

func WithDevices(conf *Config) Option {
    deviceService := newDeviceService(conf)
    return func(client *packngo.Client) {
        client.Devices = deviceService
    }
}

...

Motivation

The option to break testing after several tests fail could reduce CI time and give more fast feedback.

Usecase

1. The user pushes commit which destroys one of the general functionality of product.
2. Cloudtest starts testing.

Actual: Cloudtest will finish after all tests fail.
Expected: Cloudtest will finish after several tests fail if set option to break testing.