testy / testyts Goto Github PK

The code climate report should be A.

I've had a problem where tests focused using the ftest decorator are not ran in the class context. Therefore, this is set to undefined.

The code coverage is (71% as of February 8th). This is pretty sad and ironic for a testing framework. It would be good to bring this up at least to 85%.

Is it correct that you will receive a "Cannot read property [x] of undefined" error if referencing an enum in a test file, unless that enum is imported from a file with nothing else in it but that enum?

Consider the following fails:

import { expect, TestSuite, Test } from 'testyts';
import { Log, LogLevel } from "../../services/logger";

@TestSuite()
export class Enum {
 
    @Test()
    CreateDB() {

        Log("This will fail with error 'Cannot read property LogLevel.Trace of undefined'", LogLevel.Trace);
    }
}

However this works:

import { expect, TestSuite, Test } from 'testyts';
import { Log } from "../../services/logger";
import { LogLevel } from "../../services/logger_util";

@TestSuite()
export class Enum {
 
    @Test()
    CreateDB() {

        Log("Once split into 2 files, this will run fine.", LogLevel.Trace);
    }
}

I had been searching for a fix for this for some time (since it appears to affect other TS test frameworks as well, including Mocha at least). After trying to copy the .tsconfig and testys.json configs with no success, I finally stumbled upon the answer by looking here.

It seems that this is a limitation of TS itself (Please tell me if it's otherwise), so I'm not sure if this is something you can 'fix'. Perhaps a note in the documentation somewhere? Otherwise perhaps this issue itself will help others if they do a search for 'enum' like I tried.

Anyway, really like what you've done. Keep up the good work :-)

Hi,

I'm currently using a beta feature from TypeScript 4.1 beta, which has a new flag in tsconfig.json called noUncheckedIndexedAccess.

ts-node doesn't understand this yet, which is evident when I try to run testyts:

Error: "⨯ Unable to compile TypeScript:
error TS5023: Unknown compiler option 'noUncheckedIndexedAccess'."

Is it possible to use an external tsc --watch process and just point testyts at the compiled .js files?

I'm having trouble figuring out what to import to use this library. The examples show impoting locally form the package. Also, where is expect coming from, used by the assertions.

Hello Aboisier,
Is there a way to, for example, run a single test file, like:

testyts ./tests/01.test.spec.ts

Thanks in advance for your help,

This problem stems from the loadTests method. We need to throw an error if there is a name conflict instead of blindly overrinding the previous test suite.

public async loadTests(root: string, patterns: string[], tsconfig: tsnode.Options): Promise<TestSuiteInstance> {
    this.registerTranspiler(tsconfig);

    const testSuites = new RootTestSuite();
    const files = await this.getTestFiles(root, patterns);
    for (const file of files) {
        const testInstances = await this.getTestInstancesFromFile(file);
        testInstances.forEach(x => testSuites.set(x.name, x));
    }
    
    return testSuites;
}

Recently, the logging was improved by printing the stacktrace of failed tests.

It would be interesting to see a summary of the failed tests to prevent having to scroll to find failed tests.

Current behaviour

Root
  Expect ArraysToBeEqual Test Suite
    Arrays to be equal
      √ arrays with undefined values
      √ empty arrays

[cropped]

  Test Runner Visitor Error Messages Tests
    x Asynchronous tests failures  - Expected test to succeed.
      Error: Expected test to succeed.
          at Expect.toBeEqual (C:\testyts\TestyTs\src\lib\assertion\expect.ts:34:13)
          at Function.expectReportsToBeEqual (C:\testyts\TestyTs\src\spec\utils\testUtils.ts:16:12)
          at Function.expectReportsToBeEqual (C:\testyts\TestyTs\src\spec\utils\testUtils.ts:28:12)
          at TestRunnerVisitorTests.<anonymous> (C:\testyts\TestyTs\src\spec\tests\testRunnerVisitor.spec.ts:44:15)
          at Generator.next (<anonymous>)
          at fulfilled (C:\testyts\TestyTs\src\spec\tests\testRunnerVisitor.spec.ts:11:58)
    √ Tests failures
  
  Test Runner Visitor Tests
    x Simple test suite - Expected B to equal C.
      Error: Expected B to equal C.
          at Expect.toBeEqual (C:\testyts\TestyTs\src\lib\assertion\expect.ts:34:13)
          at Function.expectReportsToBeEqual (C:\testyts\TestyTs\src\spec\utils\testUtils.ts:16:12)
          at Function.expectReportsToBeEqual (C:\testyts\TestyTs\src\spec\utils\testUtils.ts:28:12)
          at TestRunnerVisitorTests.<anonymous> (C:\testyts\TestyTs\src\spec\tests\testRunnerVisitor.spec.ts:44:15)
          at Generator.next (<anonymous>)
          at fulfilled (C:\testyts\TestyTs\src\spec\tests\testRunnerVisitor.spec.ts:11:58)
    √ Test suite with failure
    √ Test suite with skipped tests
    √ Skipped and focused

  Tests Suite Testszz
    √ get tests, all normal tests


Summary: 182/183 passed, 1/183 failed, 0/183 skipped. (0.202s)

Suggested behaviour

Root
  Expect ArraysToBeEqual Test Suite
    Arrays to be equal
      √ arrays with undefined values
      √ empty arrays

[cropped]

  Test Runner Visitor Error Messages Tests
    x Asynchronous tests failures  - Expected test to succeed.
      Error: Expected test to succeed.
          at Expect.toBeEqual (C:\testyts\TestyTs\src\lib\assertion\expect.ts:34:13)
          at Function.expectReportsToBeEqual (C:\testyts\TestyTs\src\spec\utils\testUtils.ts:16:12)
          at Function.expectReportsToBeEqual (C:\testyts\TestyTs\src\spec\utils\testUtils.ts:28:12)
          at TestRunnerVisitorTests.<anonymous> (C:\testyts\TestyTs\src\spec\tests\testRunnerVisitor.spec.ts:44:15)
          at Generator.next (<anonymous>)
          at fulfilled (C:\testyts\TestyTs\src\spec\tests\testRunnerVisitor.spec.ts:11:58)
    √ Tests failures
  
  Test Runner Visitor Tests
    x Simple test suite - Expected B to equal C.
      Error: Expected B to equal C.
          at Expect.toBeEqual (C:\testyts\TestyTs\src\lib\assertion\expect.ts:34:13)
          at Function.expectReportsToBeEqual (C:\testyts\TestyTs\src\spec\utils\testUtils.ts:16:12)
          at Function.expectReportsToBeEqual (C:\testyts\TestyTs\src\spec\utils\testUtils.ts:28:12)
          at TestRunnerVisitorTests.<anonymous> (C:\testyts\TestyTs\src\spec\tests\testRunnerVisitor.spec.ts:44:15)
          at Generator.next (<anonymous>)
          at fulfilled (C:\testyts\TestyTs\src\spec\tests\testRunnerVisitor.spec.ts:11:58)
    √ Test suite with failure
    √ Test suite with skipped tests
    √ Skipped and focused

  Tests Suite Testszz
    √ get tests, all normal tests

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Summary: 182/183 passed, 1/183 failed, 0/183 skipped. (0.202s)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

x Asynchronous tests failures  - Expected test to succeed.
  Error: Expected test to succeed.
      at Expect.toBeEqual (C:\testyts\TestyTs\src\lib\assertion\expect.ts:34:13)
      at Function.expectReportsToBeEqual (C:\testyts\TestyTs\src\spec\utils\testUtils.ts:16:12)
      at Function.expectReportsToBeEqual (C:\testyts\TestyTs\src\spec\utils\testUtils.ts:28:12)
      at TestRunnerVisitorTests.<anonymous> (C:\testyts\TestyTs\src\spec\tests\testRunnerVisitor.spec.ts:44:15)
      at Generator.next (<anonymous>)
      at fulfilled (C:\testyts\TestyTs\src\spec\tests\testRunnerVisitor.spec.ts:11:58)

x Simple test suite - Expected B to equal C.
  Error: Expected B to equal C.
      at Expect.toBeEqual (C:\testyts\TestyTs\src\lib\assertion\expect.ts:34:13)
      at Function.expectReportsToBeEqual (C:\testyts\TestyTs\src\spec\utils\testUtils.ts:16:12)
      at Function.expectReportsToBeEqual (C:\testyts\TestyTs\src\spec\utils\testUtils.ts:28:12)
      at TestRunnerVisitorTests.<anonymous> (C:\testyts\TestyTs\src\spec\tests\testRunnerVisitor.spec.ts:44:15)
      at Generator.next (<anonymous>)
      at fulfilled (C:\testyts\TestyTs\src\spec\tests\testRunnerVisitor.spec.ts:11:58)

It would be interesting to be able to configure testy using a .ts file. This would make testy more extensible. For instance, it would be easy to use an external reporter.

I do not see it in the comamnd line options -- is it possible to get code coverage with the testyts runner? I am considering migrating from Jest, where I can do test jest --coverage and get a coverage report.

The error message for timed out tests is undefined.

When a test is started, a timeout is also started concurrently.

setTimeout(() => reject('Test has timed out.'), timeout);

When this error is caught, the error is a string and not an error. The problem is that the error message is set like this by the test runner:

report = new FailedTestReport(test.name, err.message, 0);

We should check whether the error is a string or an object.

The following test will also fail with an undefined error message:

@Test()
async test() {
    await new Promise((resolve, reject) => reject('Failure!'));
}

Currently, if you want to set a custom timeout but do not have cases or a custom test name, you need to pass two empty arguments.

@Test(undefined, undefined, 10000) // Really slow test
async slowTest() {
    // Some test
}

As suggested by @bryanmenard in #25, it would be interesting to specify the custom timeout using a @Timeout decorator instead.

@Test()
@Timeout(7000) // Really slow test
async slowTest() {
    // Some test
}

I intend to use the tests as part of a build pipeline. but even when some tests fail the exit code is 0.

There's a known issue in ts-node that occurred due to breaking API changes that ts-node did not account for.

The fix is in version 10.6 of ts-node

I have a testy.json configured as:

  "tsconfig": "./tsconfig.spec.json",
  "include":[
    "test/**/*.spec.ts"
  ]

Testyts version 1.5.0

When I run testyts, there are errors:

An error occured while executing the following command: /usr/local/Cellar/node@14/14.19.1/bin/node /usr/local/bin/testyts. Error: "⨯ Unable to compile TypeScript:
error TS5023: Unknown compiler option 'noUncheckedIndexedAccess'.
error TS5023: Unknown compiler option 'noImplicitOverride'.
error TS5023: Unknown compiler option 'noPropertyAccessFromIndexSignature'.
"

But these config parameters are removed in tsconfig.spec.json, they only exist in the tsconfig.json. If I rename tsconfig.json to tsconfig.app.json, then the tests run as expected.

Testy seems to be still using tsconfig.json if it exists regardless of the config parameters.

I currently can't use path mapping resolution i.e #/src/somewhere/deeper I was wondering if this was already possible but missing in the documentation, or if there was something else I had to do.

The error is:

error TS2307: Cannot find module

I've had a problem where two test classes inheriting the same base class are merged.

class Base { 
    @beforeEach() beforeEach() { }
}

@testSuite()
export class A extends Base {
    @test() testA() { }
}

@testSuite()
export class B extends Base {
    @test() testB() { }
}

A and B will both have tests testA and testB. A's name will also be overwritten with B.

Shouldn't it be 0/3 failed?

I structured my TypeScript NodeJS app as ES modules ("type": "module" in package.json). This is the output of testyts:

An error occured while executing the following command: /usr/bin/node <REDACTED>/node_modules/.bin/testyts. Error: "Must use import to load ES Module: <REDACTED>/src/app.spec.ts
require() of ES modules is not supported.
require() of <REDACTED>/src/app.spec.ts from <REDACTED>/node_modules/testyts/build/lib/utils/testsLoader.js is an ES module file as it is a .ts file whose nearest parent package.json contains "type": "module" which defines all .ts files in that package scope as ES modules.
Instead change the requiring code to use import(), or remove "type": "module" from <REDACTED>/package.json.
"

The current test case syntax is pretty ugly in my opinion.

@Test('My test', [
      new TestCase('Two plus two is four', 2, 2, 4),
      new TestCase(`Minus one that's three`, 4, -1, 3)
])
quickMaths(a: number, b: number, result: number) {
    expect.toBeEqual(a + b, result);
}

I think we can improve this. Here are a few examples of how we could go about it:

I like this syntax, but I'm not sure where we would put the timeout parameter, and we lose the opportunity to have a custom root-test name.

@TestCase('Two plus two is four', 2, 2, 4)
@TestCase(`Minus one that's three`, 4, -1, 3)
quickMaths(a: number, b: number, result: number) {
    expect.toBeEqual(a + b, result);
}

This option solves the name and timeout parameters problem and is still more readable than the current implementation.

@Test()
@TestCase('Two plus two is four', 2, 2, 4)
@TestCase(`Minus one that's three`, 4, -1, 3)
quickMaths(a: number, b: number, result: number) {
    expect.toBeEqual(a + b, result);
}

Comments in i.e. tsconfig.json not supported

testyts An error occured while executing the following command: C:\Program Files\nodejs\node.exe C:\Users\mail\AppData\Roaming\npm\node_modules\testyts\build\testy.js. Error: "client\tsconfig.json: Unexpected token / in JSON at position 169"

--
Cool project, I like your approach!

lg.g.

For example:

@TestSuite()
export class MyTestSuite {
  @Test()
  onePlusOne() {
    // Act
    const result = {x: 2};

    // Assert
    expect.toBeEqual(result, {x: 2});
  }
}

This will fail, with a stack overflow, because the inputs to toBeEqual are objects and toBeEqual doesn't know how to handle objects.

When I try to follow the implementation of toBeEqual, I end up in some non-TypeScript hell of .d.ts files, instead of actual TypeScript code, which seems weird for a framework that advertises itself as using the "full power of TypeScript". Since there's no implementation code, I'm not sure what's wrong with the implementation or how I could make a pull request to try and fix it.

Error on the following tsconfig.json file.

testyts An error occured while executing the following command: C:\Program Files\nodejs\node.exe C:\Users\mail\AppData\Roaming\npm\node_modules\testyts\build\testy.js. Error: "client\tsconfig.json: Unexpected token } in JSON at position 229"

{
    "compilerOptions": {
        "experimentalDecorators": true,
        "emitDecoratorMetadata": true,
        "module": "commonjs",
        "target": "ES2016",
        "outDir": "dist",
        "sourceMap": true,
    },
    "include": [
        "src/**/*",
        "tests/**/*",
        "typings-custom/**/*.d.ts",
    ],
    "exclude": [
        "dist"
    ]
}

lg.g.

Add .d.ts definitions to work with @types

I ask if the framework support integration with chai?

Thank you for TestyTs!

There are some nice features in TypeScript 4 (4.2.4 as I use) when declaring a name together with the return type, which I would really like TestyTs to be compatible with.

Example:

// Does NOT work
unpackField(image: Buffer, pointer: number): [field: Field, value: any, newPointer: number] {}
// Does work
unpackField(image: Buffer, pointer: number): [Field, any, number] {}

This code requires a newer version of TypeScript than TestyTs uses.

I can hack this to work with TestyTS if I manually update the version of ./node_modules/testyts/package.json to TypeScript 4.2.4 and run npm i from within that directory.

Thank you!

It's missing this line:

import { BeforeEach, expect, Test, TestSuite } from 'testyts';

OS: ubuntu 20.04 LTS
npm: 7.11.1

alain@aegnor:~/devel_syno/NodeJS/svg-be-draw$ npm i testyts --save-dev

added 24 packages, and audited 144 packages in 5s

15 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

alain@aegnor:~/devel_syno/NodeJS/svg-be-draw$ npm ll
[email protected]
│ /mnt/ubuntu18.04/home/alain/devel_syno/NodeJS/svg-be-draw
│ Backend drawing of SVG files (experimental)
├── @svgdotjs/[email protected]
│   
├── @types/[email protected]
│   
├── [email protected]
│   
├── [email protected]
│   
├── [email protected]
│   
└── [email protected]

alain@aegnor:~/devel_syno/NodeJS/svg-be-draw$ testyts init
testyts: command not found

Global install does not work any better :-(.

Package is well present under node_modules. What can be wrong?

alain@aegnor:~/devel_syno/NodeJS/svg-be-draw$ ls -ld node_modules/test*
drwxrwxr-x 1 alain alain 110 mai 18 22:28 node_modules/testyts

It would be nice to know where the errors comes from. Print at least the line number. Possibly the entire stack trace.

Hey,
would be cool if we can define a global timeout. This would be great especially testing with puppeteer or playwright.

Hi,

Please add error handling for @BeforeAll, @AfterAll, @BeforeEach, @AfterEach methods

Version: 2.0.0-beta.7

ex:

@TestSuite()
export class TestSuit {
  @BeforeAll()
  async beforeAll() {
    throw new Error('Some error');
  }

  @Test()
  public test() {
    ok(true);
  }
}

The output is:

Root
TestSuit
Summary: 0/1 passed, 1/1 failed, 0/1 skipped. (0s)

There are any info about the exception

testy.json was created even though this error is printed in the console:

» yarn add testyts -D
...
✨  Done in 3.13s.
» npx testyts init
/path/to/project/node_modules/testyts/build/lib/utils/jsonLoader.service.js:21
                throw new Error(`The specified configuration file could not be found: ${path}`);
                      ^

Error: The specified configuration file could not be found: /path/to/project/testy.json
    at JsonLoader.<anonymous> (/path/to/project/node_modules/testyts/build/lib/utils/jsonLoader.service.js:21:23)
    at Generator.next (<anonymous>)
    at /path/to/project/node_modules/testyts/build/lib/utils/jsonLoader.service.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (/path/to/project/node_modules/testyts/build/lib/utils/jsonLoader.service.js:4:12)
    at JsonLoader.load (/path/to/project/node_modules/testyts/build/lib/utils/jsonLoader.service.js:18:16)
    at TestyCli.<anonymous> (/path/to/project/node_modules/testyts/build/lib/cli/testyCli.js:50:55)
    at Generator.next (<anonymous>)
    at /path/to/project/node_modules/testyts/build/lib/cli/testyCli.js:8:71
    at new Promise (<anonymous>)

Node.js v17.0.1
» ls testy.json
testy.json
» npx testyts init # no output when run a second time
»

This is on macos

Can we support "tsconfig" option via TestyConfig as well? Currently I see it being supported only as an option to CLI. Apologies if I am missing something here, as today only I came across your library, and since it looks very lightweight I am think to try it out for the good of my thin little packages that I write. Let me know your thoughts. Appreciate the idea.

I reckon the standard reporter colors probably do not display properly in every terminal. It should be possible to disable them.

I see multiple ways of going about this.

1. Implementing a new StandardColorless reporter
2. Making it possible to configure reporters, and add a noColors configuration to the standard reporter
3. Adding a colors: boolean configuration in the testy.json, and/or a command-line argument

Currently, I have to deploy pre-versions manually. It would be great if the publish action could be a bit smarter and automatically tag versions with the proper tag. For example:

The current test runner is not extensible. For example, the test reporting is embedded in the test runner class. Implementing the Visual Studio Code adapter involved a full re-implementation of the test runner.

I believe using a decorator pattern would allow extending the test runner (and eventually other test visitors) more easily.

It would be interesting to be able to export a test run as a TAP report.

The name parameter should be optional in both ftest and xtest.

This is related with the new global "timeout" setting in testy.json.

Problem: The yarn task "testyts" is not exiting after the test completion. The task is running at least the timeout time specified in the testy.json file.

I.E. if there is a long timeout defined "timeout": 100000, the task will only exit after this timeout.

This issue brings some disadvantages:

• VSCode Test debugger doesn't stop for a long time.
• Long execution time for tests

Workarounds:

• using a global install of testyts und execute the tests with testyts (why is it working here?) (no VSCode debugging support).
• lower the timeout as low you can... (but a deallocated container instance can take some time to come up...)

The E2E test section examples is mentioned several times but the links to the documentation are broken.

Version 1.1.0

Test Code:

@Test()
test() {
    expect.toBeEqual(NaN, 0);
}

Expected outcome:

x test - Expected NaN to equal 0.

Actual outcome:

x test - Expected null  to equal 0.

In the project sgvizler2, I try to use TestyTs but my project use also JQuery.

I have the error:

ERROR An error occured while executing the following command: /usr/bin/node /home/karima/git/sgvizler2/node_modules/.bin/testyts. Error: "jQuery is not defined"

You can reproduce the error:

1. clone https://github.com/BorderCloud/sgvizler2.git
2. remove the comments in the test "doParseOptionsRawWithoutOption" (file Chart.spec.ts)
3. testyts

May be, I forgot an option in testy.json ?

The current test and testSuite interfaces often lead to redundancy. For example

@test('OnePlusOneTest')
onePlusOneTest() { 
    // The test
}

I suggest making the name parameter optional and inferring the test/test suite name from the method/class name.

@test()
onePlusOneTest() { 
    // The test
}

This could potentially cause a problem with the test code is minified or uglified.