marckhouzam / cobra-completion-testing Goto Github PK

If one in the series of tests has a failure (a series of test is run in a container), the error code generated does not propagate up and is not shown at the end of the entire test run.

It seems an error code is only shown if the last run of all the tests has an error.

This makes it very easy to miss an error.

Currently the test cases are defined for each shell. Most of them are duplicates. This makes it unnecessary hard to maintain. I think it would make more sense to store them in a single file and run the same cases for each shell. This would make it clear what cases are failing for each shell.

I think it would make sense to store the test cases in a table like structure. In the podman project we have a function for such tests https://github.com/containers/podman/blob/142b4ac966e12559c534be380093a44d0a1d2959/test/system/helpers.bash#L438-L465
You can see it in use here: https://github.com/containers/podman/blob/142b4ac966e12559c534be380093a44d0a1d2959/test/system/065-cp.bats#L25-L51

I think a good structure could be

Name of the test | test command | expected output | (optional) Name of the shells where the test is expected to fail

And possible more depending on what information the tests require.

@marckhouzam shall an step or job be added to the cobra CI for running this test suite? Alternatively, shall a CI workflow be added to this repo?