Comments (5)
dylanhitt
commented on June 23, 2024
1
@SimonBaeumer I think this can be closed. As it will be addressed in #77. I can add a check to not print Test suite failed, use --verbose for more detailed output in #126.
from commander.
dylanhitt
commented on June 23, 2024
Hey @billychow68,
Can you send me a little more info. An error message or your dir structure would suffice. I currently cannot recreate the issue. Here is my output - I am able to run commander with --dir or --verbose in either order:
commander ((HEAD detached at v2.1.0)) $ ./commander test --dir --verbose examples/
Starting test against directory: examples/...
2020/07/03 12:32:24 title: 'it should validate exit code' Command: exit 1
2020/07/03 12:32:24 title: 'it should validate exit code' Directory:
2020/07/03 12:32:24 title: 'it should validate exit code' Env: []
2020/07/03 12:32:24 title: 'it should validate exit code' ExitCode: 1
2020/07/03 12:32:24 title: 'it should validate exit code' Stdout:
2020/07/03 12:32:24 title: 'it should validate exit code' Stderr:
2020/07/03 12:32:24 title: 'it should validate exit code' Stdout-Expected: {[] map[] 0 [] map[] map[]}
2020/07/03 12:32:24 title: 'it should validate exit code' Stdout-Result: true
2020/07/03 12:32:24 title: 'it should validate exit code' Stderr-Expected: {[] map[] 0 [] map[] map[]}
2020/07/03 12:32:24 title: 'it should validate exit code' Stderr-Result: true
2020/07/03 12:32:24 title: 'it should validate exit code' Exit-Expected: 1
2020/07/03 12:32:24 title: 'it should validate exit code' Exit-Result: true
✓ [commander.yaml] [local] it should validate exit code
2020/07/03 12:32:24 title: 'echo hello' Command: echo hello
2020/07/03 12:32:24 title: 'echo hello' Directory:
2020/07/03 12:32:24 title: 'echo hello' Env: []
2020/07/03 12:32:24 title: 'echo hello' ExitCode: 0
2020/07/03 12:32:24 title: 'echo hello' Stdout: hello
2020/07/03 12:32:24 title: 'echo hello' Stderr:
2020/07/03 12:32:24 title: 'echo hello' Stdout-Expected: {[] map[1:hello] hello 1 [] map[] map[]}
2020/07/03 12:32:24 title: 'echo hello' Stdout-Result: true
2020/07/03 12:32:24 title: 'echo hello' Stderr-Expected: {[] map[] 0 [] map[] map[]}
2020/07/03 12:32:24 title: 'echo hello' Stderr-Result: true
2020/07/03 12:32:24 title: 'echo hello' Exit-Expected: 0
2020/07/03 12:32:24 title: 'echo hello' Exit-Result: true
✓ [commander.yaml] [local] echo hello
2020/07/03 12:32:24 title: 'it should fail' Command: invalid
2020/07/03 12:32:24 title: 'it should fail' Directory:
2020/07/03 12:32:24 title: 'it should fail' Env: []
2020/07/03 12:32:24 title: 'it should fail' ExitCode: 127
2020/07/03 12:32:24 title: 'it should fail' Stdout:
2020/07/03 12:32:24 title: 'it should fail' Stderr: /bin/sh: invalid: command not found
2020/07/03 12:32:24 title: 'it should fail' Stdout-Expected: {[] map[] 0 [] map[] map[]}
2020/07/03 12:32:24 title: 'it should fail' Stdout-Result: true
2020/07/03 12:32:24 title: 'it should fail' Stderr-Expected: {[/bin/sh: 1: command invalid: not found] map[] 0 [] map[] map[]}
2020/07/03 12:32:24 title: 'it should fail' Stderr-Result: false
✗ [commander.yaml] [local] it should fail
2020/07/03 12:32:24 title: 'it should print hello world' Command: echo hello world
2020/07/03 12:32:24 title: 'it should print hello world' Directory:
2020/07/03 12:32:24 title: 'it should print hello world' Env: []
2020/07/03 12:32:24 title: 'it should print hello world' ExitCode: 0
2020/07/03 12:32:24 title: 'it should print hello world' Stdout: hello world
2020/07/03 12:32:24 title: 'it should print hello world' Stderr:
2020/07/03 12:32:24 title: 'it should print hello world' Stdout-Expected: {[hello world] map[] 0 [] map[] map[]}
2020/07/03 12:32:24 title: 'it should print hello world' Stdout-Result: true
2020/07/03 12:32:24 title: 'it should print hello world' Stderr-Expected: {[] map[] 0 [] map[] map[]}
2020/07/03 12:32:24 title: 'it should print hello world' Stderr-Result: true
2020/07/03 12:32:24 title: 'it should print hello world' Exit-Expected: 0
2020/07/03 12:32:24 title: 'it should print hello world' Exit-Result: true
✓ [commander.yaml] [local] it should print hello world
2020/07/03 12:32:24 Use docker executor
2020/07/03 12:32:24 Pulling image docker.io/simonbaeumer/test-private:latest
✗ [docker.yaml] [ubuntu] cat /etc/lsb-release
2020/07/03 12:32:25 title: 'it should overwrite env' Command: echo $KEY
2020/07/03 12:32:25 title: 'it should overwrite env' Directory:
2020/07/03 12:32:25 title: 'it should overwrite env' Env: [KEY=local]
2020/07/03 12:32:25 title: 'it should overwrite env' ExitCode: 0
2020/07/03 12:32:25 title: 'it should overwrite env' Stdout: local
2020/07/03 12:32:25 title: 'it should overwrite env' Stderr:
2020/07/03 12:32:25 title: 'it should overwrite env' Stdout-Expected: {[local] map[] 0 [] map[] map[]}
2020/07/03 12:32:25 title: 'it should overwrite env' Stdout-Result: true
2020/07/03 12:32:25 title: 'it should overwrite env' Stderr-Expected: {[] map[] 0 [] map[] map[]}
2020/07/03 12:32:25 title: 'it should overwrite env' Stderr-Result: true
2020/07/03 12:32:25 title: 'it should overwrite env' Exit-Expected: 0
2020/07/03 12:32:25 title: 'it should overwrite env' Exit-Result: true
✓ [environment.yaml] [local] it should overwrite env
2020/07/03 12:32:25 title: 'echo $KEY' Command: echo $KEY
2020/07/03 12:32:25 title: 'echo $KEY' Directory:
2020/07/03 12:32:25 title: 'echo $KEY' Env: [KEY=global]
2020/07/03 12:32:25 title: 'echo $KEY' ExitCode: 0
2020/07/03 12:32:25 title: 'echo $KEY' Stdout: global
2020/07/03 12:32:25 title: 'echo $KEY' Stderr:
2020/07/03 12:32:25 title: 'echo $KEY' Stdout-Expected: {[global] map[] 0 [] map[] map[]}
2020/07/03 12:32:25 title: 'echo $KEY' Stdout-Result: true
2020/07/03 12:32:25 title: 'echo $KEY' Stderr-Expected: {[] map[] 0 [] map[] map[]}
2020/07/03 12:32:25 title: 'echo $KEY' Stderr-Result: true
2020/07/03 12:32:25 title: 'echo $KEY' Exit-Expected: 0
2020/07/03 12:32:25 title: 'echo $KEY' Exit-Result: true
✓ [environment.yaml] [local] echo $KEY
2020/07/03 12:32:25 title: 'echo hello' Command: echo hello
2020/07/03 12:32:25 title: 'echo hello' Directory:
2020/07/03 12:32:25 title: 'echo hello' Env: []
2020/07/03 12:32:25 title: 'echo hello' ExitCode: 0
2020/07/03 12:32:25 title: 'echo hello' Stdout: hello
2020/07/03 12:32:25 title: 'echo hello' Stderr:
2020/07/03 12:32:25 title: 'echo hello' Stdout-Expected: {[] map[] 0 [] map[] map[]}
2020/07/03 12:32:25 title: 'echo hello' Stdout-Result: true
2020/07/03 12:32:25 title: 'echo hello' Stderr-Expected: {[] map[] 0 [] map[] map[]}
2020/07/03 12:32:25 title: 'echo hello' Stderr-Result: true
2020/07/03 12:32:25 title: 'echo hello' Exit-Expected: 0
2020/07/03 12:32:25 title: 'echo hello' Exit-Result: true
✓ [minimal_test.yaml] [local] echo hello
Results
✗ [commander.yaml] [local] 'it should fail', on property 'Stderr'
Expected
/bin/sh: invalid: command not found
to contain
/bin/sh: 1: command invalid: not found
✗ [docker.yaml] [ubuntu] 'cat /etc/lsb-release' could not be executed with error message:
could not pull image 'docker.io/simonbaeumer/test-private:latest' with error: 'Error response from daemon: pull access denied for simonbaeumer/test-private, repository does not exist or may require 'docker login': denied: requested access to the resource is denied'
Duration: 0.373s
Count: 8, Failed: 2
Test suite failed, use --verbose for more detailed output
from commander.
billychow68
commented on June 23, 2024
Hello @dylanhitt,
$ tree
.
├── README.md
├── commander
├── media
│ ├── test1.wav
│ ├── test2.wav
│ └── test3.wav
├── run.sh
└── tests
├── adjust_audio_volume.yaml
├── animate_builtin.yaml
├── exclude
│ └── post_logs.yaml
├── install_audio.yaml
├── play_builtin.yaml
└── scripts
└── gq.sh
4 directories, 12 files
Run Commander with --dir flag:
$ ./commander test --dir tests/
Starting test against directory: tests/...
✓ [adjust_audio_volume.yaml] [local] GQ0200 - Test method=adjust_audio_volume, volume=0
✓ [adjust_audio_volume.yaml] [local] GQ0201 - Test method=adjust_audio_volume, volume=1
✓ [adjust_audio_volume.yaml] [local] GQ0202 - Test method=adjust_audio_volume, volume=25
...
(I purposely limited the long output)
...
Duration: 11.743s
Count: 60, Failed: 8
Test suite failed, use --verbose for more detailed output
$
Run Commander with both, --dir and --verbose, flags in that order:
$ ./commander test --dir tests/ --verbose
Test may not be filtered when --dir is enabled
I just learned that if I reverse the order of the flags it works.
$ ./commander test --verbose --dir tests/
Starting test against directory: tests/...
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25' Command: ./tests/scripts/gq.sh 202
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25' Directory:
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25' Env:
...
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25' ExitCode: 0
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25' Stdout: {"success":true}
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25' Stderr:
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25' Stdout-Expected: {[{"success":true}] map[] 0 [] map[] map[]}
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25' Stdout-Result: true
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25' Stderr-Expected: {[] map[] 0 [] map[] map[]}
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25' Stderr-Result: true
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25' Exit-Expected: 0
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25' Exit-Result: true
✓ [adjust_audio_volume.yaml] [local] GQ0202 - Test method=adjust_audio_volume, volume=25
...
Duration: 12.220s
Count: 60, Failed: 8
Test suite failed, use --verbose for more detailed output
Minor bug: noticed it prints "Test suite failed, use --verbose for more detailed output" even though I ran it with --verbose?
from commander.
SimonBaeumer
commented on June 23, 2024
The --verbose is interpreted as an argument which is used to filter for a specific test in your test suite.
An error message is displayed because filtering is not enabled with the --dir option.
from commander.
SimonBaeumer
commented on June 23, 2024
Added a draft for the filter implementation. Closes this issue because of #77 as @dylanhitt mentioned above.
from commander.
Related Issues (20)
- Test Suite aggregation should occur in runtime HOT 2
- Create a custom GitHub action for Commander HOT 1
- Properly mock file and dirs in app tests to not read from the filesystem
- config value dir, should work for file assertion
- user should be able to set the working dir with `--workdir` HOT 1
- Add junit ouput support HOT 1
- Move away from travisCI HOT 5
- Add yaml linter in CI
- documented install command fails with go 1.16 HOT 5
- Upgrade docker version
- Change []runtime.TestCase in Suite to an array of pointers
- Unable to run docker via kind cluster HOT 11
- SSH public key authentication broken in commander 2.4.0 on CentOS 8.4.2105 HOT 1
- Enable regular expression for stdout, stderr assertion HOT 3
- Add proper error handling and coroutine cancellation
- Rename TestCommandContext to TestCommandConfig
- Support bash and awk verifiers HOT 1
- Commands in Installation section try to download non-existing version HOT 1
- dotfiles in test dir cause a panic HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from commander.