marvinjwendt / testza Goto Github PK

Currently, it looks like this. We can add more colors and describe the objects better.

Remove the diff and only show one object, like in AssertNotEqual.

Current state

Some methods don't output msg.

Testza should be able to create a snapshot of any object and save it to a file. Later, in other test runs, that snapshot should be compared with the object and validate the values.

There is a missing comment for AssertRegexp and AssertNotRegexp.

CaptureStdoutAndStderr should return two strings, Stdout and Stderr.

Currently, AssertNoError prints the whole error object if it fails. We should focus on the actual error text and maybe append the whole object at the end.

Testza should support some kind of configuration.
The exact implementation has to be figured out (new scope [SetXxx()], settings struct, variables, etc.).

Settings to include:

• disable color
• disable line numbers

Add go test flags for settings.

Changing MockInput to Fuzz or Fuzzy would make the name fit better, as fuzzy testing describes using random input values for unit tests.

Add GetXxx functions for settings.

Currently, the CONTRIBUTING.md file is empty.
We should put some guidelines in there.

Must haves:

• Function naming
• File naming
• Writing tests
• Documenting functions
• Adding examples to functions

Current

Expected

I've just recently discovered this library and am finding it great. However, one issue I've got is that the output is really hard to read in VSCode instead of in a terminal.

In VSCode it looks like this:

And the exact same in the terminal looks like this:

Cheers

Color the object names magenta

Current

Expected

The output objects should be renamed. Currently, it says that it expects something to be a kind, but actually it expects the object to not be of that kind.

Error output starts with line number 0. It should start with 1.

Add functions to mock user input.

Hi~Could you share your terminal theme and config?

Assertions should return a boolean, that indicates if the assertion completed.

Currently if you are comparing strings with testza, it only detects if they are different, but not what is different within them as it does with structs. When comparing multiline strings, it gets even more confusing as finding the issue on large strings requires using external tools.

Here's an example of current behavior:

It would be great if it was possible to have a rich git-like diff on a string. There are even libraries that have specialized for this within Go: https://github.com/sergi/go-diff

If this is a feature that would be accepted by the maintainers, I would be willing to implement it via PR.

When using the following:

err = testza.SnapshotCreateOrValidate(t, t.Name(), string(result))
testza.AssertNoError(t, err)

The first run succeeds and creates a snapshot. The second test run fails with the error:

"stat /[path redacted]/testdata/snapshots/TestReports/test_name.testza: permission denied"

Running ls -l on the parent directory shows drw------- for the testdata directory.

AFAICT, there are two paths for creating directories in snapshot.go: via SnapshotCreate, which ends up calling os.MkdirAll(dir, 0755).

The second, via SnapshotCreateOrValidate, calls os.MkdirAll(path.Dir(snapshotPath), 0600):

If I understand this correctly, the issue is that the permissions bits on the second call should match the first.

I think, removing the Input part in the Fuzz functions would make them smaller. I don't know, if we add anything else, to fuzz, than inputs. If we do, we would need this seperation, but nothing else comes to my mind, that could benefit of fuzzing.

Expected object should be green in AssertLen.

pointers, etc will not work properly with AssertNil . The assert will fail for nil values and print:

1| An object that should be nil is not nil.
2| 
3| Expected:
4| (interface {}) <nil>
5| 
6| Actual:
7| (*main.record)(<nil>)

Here's a GoPlay that illustrates it.

The solution would be to use reflect package and check for nil using reflect.ValueOf(v).IsNil() method.

Since I was surprised myself, I have found this article that does a good job, IMHO, to explain why.

func TestHello(t *testing.T) {
testza.AssertDirEmpty(t, "FolderName")
}

output:

# Running tests with Testza
## Using seed "1647841197635983500" for random operations
## System info: OS=windows | arch=amd64 | cpu=Intel(R) Core(TM) i5-8400 CPU @ 2.80GH
z | go=go1.18

--- FAIL: TestHello (0.00s)
    assertion_helper.go:273: 
        
           0| Error opening directory specified
           1| 
           2| dir:
           3| (string) (len=10) "FolderName"
        
        
    main_test.go:9: 
        
           0| The directory is not empty.
           1| 
           2| Directory:
           3| (string) (len=10) "FolderName"
        
        
FAIL

I have see a case where I had to test time.Now(), it could be done here.

Add a subdirectory with a test file that fails every test to see the output.

Color the object names and make the output more understandable.

We should print a status report after all tests ran.
We have to figure out what we can display. Furthermore, we can definitely display the succeeded and failed assertions count, and the count of how many tests use testza.

We should include more assertions:

• #34
• #33
• #36
• #37
• #31
• #28
• #29
• #30
• #35
• #32

Rename AssertDirExist to AssertDirExists, as the second one is grammatically right.

In favor of feedback and clean code, the structure will be refactored.
testza.Use.Assert.True will be testza.AssertTrue.
This structure change will apply to every existing function, and it's breaking.

The CI system needs to be rewritten too.

Currently all snapshots get passed through a spew.Sdump which then get saved to a file. This ensures that even if the object type/name changes but content doesn't, snapshot would still fail.

An alternative option could be that upon taking a snapshot, the type is tested, and if it casts to a string, it could be stored as-is into the snapshot file, except with a header that would be similar in structure to the current spew.Sdump to ensure the type-check is still in place.

It currently shows Expected and Actual. In reality, they are meant to be the same. Actual == Expected when the test fails. This doesn't look right and we should consider a new format for this.

Currently, AssertImplements does not show any objects.

Currently, it shows the object, which is redundant, as the object is expected to not be nil. So if the test fails, it's always nil.