[Tracking] `devcontainer features test` improvements about cli HOT 5 OPEN

Hello.
I developed a feature using the feature template and noticed that the test for option selection was not covered.

Is it possible to run multiple installations by referencing a pre-created options matrix (json file) when running an installation as a test?
Or the user may want to give options as CLI arguments to parallelize using the matrix on GitHub Actions.

Similarly, it would be useful to be able to specify the platform (CPU architecture).

edit: I understand that we can add scenarios.json to the folder for each feature, but it seems useless because when I set up the GitHub matrix, the tests in scenarios.json are run under all matrix conditions --skip-scenarios seems work fine.

from cli.

Can we add another task? Currently, there isn't a way to supply --docker-path/--docker-compose-path to any of the features commands, which causes issues if you're using podman in ci scenarios.

from cli.

@Clockwork-Muse Do you mean a way to specify the path to the docker/docker-compose CLIs? Yes, I'll convert your request into an issue 👍 (If you're interested, we'd also accept a PR on this)

from cli.

Yes, that's what I meant.

from cli.

If there is a scenario.json file and no test.sh, the test will not run unless --skip-autogenerated is set. We should be smarter about this

👍 id very much like to see this so that i can do this in something like gh actions:

devcontainer features test -f ${{ matrix.id }}

and NOT fail like this:

as a workaround I'm currently doing this:

if [[ -e test/${{ matrix.id }}/test.sh ]]; then
  devcontainer features test -f ${{ matrix.id }}
else
  devcontainer features test -f ${{ matrix.id }} --skip-autogenerated
fi

accidentally made devcontainers/spec#294 🙄

from cli.