testomatio / check-tests Goto Github PK

Is your feature request related to a problem? Please describe.
Synchronized the test suite. But then I want to exclude one test from the set of tests that are synchronized.

Describe the solution you'd like
Enter a command in the console with the name of the test that I want to exclude from synchronization.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

TOTAL 0 TESTS FOUND


 🚀 Sending data to testomat.io

 🎉 Data sent to Testomat.io

0 tests - there is nothing to send to testomatio.

The message should be like this:

Can't find any tests in this folder.
Change file pattern or directory to scan to find test files:

Usage: npx check-tests <pattern> -d [directory]

should also be addded to reporter

Instead of considering the folder as one , it is taking and splitting them into different folders

If you try to use latest check-test command in playwright-demo project the error = "TypeError: Cannot read properties of undefined (reading 'name')" was shown.

Add to check-tests CLI a command which will generate documentation in Markdown format:

npx check-tests CodeceptJS **.test --doc tests.md

This will create tests.md file and write into it all found tests.

This feature is already available as GitHub Action, it also pushes generated markdown to GitHub Wiki. But we need that this option to exist in CLI and generate a simple md file which can be used as documentation in GitHub repo or on statice website, etc.

Documentation Generator: https://github.com/testomatio/check-tests/blob/master/src/document.js

With - uses: testomatio/check-tests@stable, as documented in the README.md, GitHub issues a deprecation warning:

"Node.js 12 actions are deprecated. Please update the following actions to use Node.js 16: testomatio/check-tests@stable. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/."

1. What is the current recommendation for using this action? - uses: testomatio/check-tests@???
2. Will this JavaScript action be migrated to use node16?

Describe the bug
tests with dynamic parameters are not uploaded correctly

To Reproduce
Steps to reproduce the behavior:
Create a test of below format

test.describe('New Loan Application', () => {
    const provinces = ['AB', 'BC', 'MB', 'NB', 'NL', 'NS', 'ON', 'SK'];
    provinces.forEach((state) => {
        test(`Validate New Loan Application for Specific Province ${state} @canpre`, async ({

Expected behavior
Should read the dynamic parameters and upload the tests

Screenshots

• test collection editing
• implement update-ids
• clean ids
• unsafe clean
• docs
• unit tests
• update examples project (to use folders)

When I use typescript path aliases, updatei-ids doesn't work as expected. It inserts the suiteId on the first line that matches the describe title. In this case it's "Shift"

import {expect, Page, test} from '@playwright/test'
import Shift @Sff4de22bModel from '@src/models/ShiftModel'

test.describe('Shift', () => {

In Import instructions on website we provide a very wide pattern for files to scan and it appears that tests from "node_modules" folder are added to output.

Please check the path of scanned tests.
If path contains node_modules directory - ignore that test.

In my current project, I have quite complex functionality that requires many tests with complex preconditions and actions. In order not to have test files for several thousand lines and 100 tests, I break such tests into files, but leave the same "describe/title" for all describes in each file.

They are normally imported to testomat. The problem occurs when I want to call update-ids.

Let's say I have test files shifts1.spec.ts, shifts2.spec.ts, shifts3.spec.ts. I have imported suites and it has the following structure in Testomat

• Shifts @S1 shifts1.spec.ts
  • With detailed capacity @S1a
• Shifts @S2 shifts2.spec.ts
  • Without detailed capacity @S2a
• Shifts @S3 shifts3.spec.ts
  • Extra @S3a

When I call npx check-tests@latest Playwright ....... --update-ids
@S3 suiteId will be inserted into each file's root describe and child suites will not be mapped/inserted.

• auto assign ids from testomat
• manual ids assignment
• sync with testomat.io

Support following actions:

• Importing Into Project
• Disable Detached Tests
• Synchronous Import
• Auto-assign Test IDs in Source Code
• Keep Test IDs Between Projects
• Clean Test IDs
• Import Into a Branch
• Keep Structure of Source Code
• Delete Empty Suites
• Import Into a Specific Suite

Support for data driven tests in Jest.

Jest has test.each syntax which can generate multiple test cases at once. We need to support it in our parser to handle such cases.

Jest parser should be updated to handle test.each as tests.
As a reference take similar syntax from CodeceptJS

// in Jest
test.each([1,2])()

// in CodeceptJS
Data([1,2]).Scenario()

Here is the implementation from CodeceptJS
https://github.com/testomatio/check-tests/blob/master/lib/frameworks/codeceptjs.js#L72

Implement a similar thing for Jest parser as well.
https://github.com/testomatio/check-tests/blob/master/lib/frameworks/jest.js

Unit test for test.each case should also be added here https://github.com/testomatio/check-tests/blob/master/tests/jest_test.js

Trace below-

Generating script.
Script contents:
TESTOMATIO_PREPEND_DIR="Cash Money" TESTOMATIO=* npx check-tests@latest Playwright "**/*{.,_}{test,spec,cy}.ts" --typescript  --purge --no-detached --keep-structure --update-ids --sync --create
========================== Starting Command Output ===========================
/usr/bin/bash --noprofile --norc /usr/local/azuredevopsagent/_work/_temp/9f550b77-aa0e-4b26-89b7-9feeed724d36.sh
 🤩 Tests checker by Testomat.io v0.8.19
Error: ENOENT: no such file or directory, open '/usr/local/azuredevopsagent/_work/1/s/tests/Capo.UI.Testing/Cash Money/tests/acquisition/loan-application.spec.ts'
Error: ENOENT: no such file or directory, open '/usr/local/azuredevopsagent/_work/1/s/tests/Capo.UI.Testing/Cash Money/tests/acquisition/loan-application.spec.ts'
    at Object.openSync (node:fs:589:3)
    at Object.readFileSync (node:fs:457:35)
    at cleanIdsCommon (/home/azureagent/.npm/_npx/876480043fc67df8/node_modules/check-tests/src/updateIds/updateIds.js:121:26)
    at cleanIds (/home/azureagent/.npm/_npx/876480043fc67df8/node_modules/check-tests/src/updateIds/index.js:11:10)
    at Command.<anonymous> (/home/azureagent/.npm/_npx/876480043fc67df8/node_modules/check-tests/bin/check.js:89:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
##[error]Bash exited with code '1'.
Finishing: Upload Playwright tests

Add "--line-numbers" option to check-test npm command.
Example:
[without mode]

 Scenario('Edited todo is saved', async (I, TodosPage) => {
  I.say('Given I have some todos');\n" +
});

[with mode]

10: Scenario('Edited todo is saved', async (I, TodosPage) => {
11:  I.say('Given I have some todos');\n" +
12: });

When release is created in check-tests repo, new package version should be published to npm.

done for check-tests and check-cucumber

I'm running the command TESTOMATIO={the_project_key} npx check-tests@latest Jest "**/*{.,_}{test,spec}.js" -d src/tests.

The command manages to find only one valid test.

All other tests fail, apparently because the command does not understand JSX.

Here are a few examples where the checker throws the error:

	it('has import now button', () => {
		const button_text = 'Push me!';
		render(<App buttonText={button_text}/>);
		expect(button()).not.toBeNull();
		expect(button().textContent).toEqual(button_text);
	});

describe('Items to scan UI', () => {
	const items = ['plugin1', 'plugin2'];
	let onSelected;
	const render = () => shallow(<ItemsToScan heading={'heading'} items={items} onSelected={onSelected}/>);

	beforeEach(() => {
		onSelected = jest.fn();
	});

describe('MO Scan on Localization page', () => {

	it('creates app', () => {
		const fileCount = 10;
		const filesToScan = {
			count: fileCount,
			plugins: ['plugin1'],
			themes: [],
			other: []
		};
		const expectedTitle = "WPML has detected changed or new MO files";

		const wrapper = shallow(
			<MOScanUILocalizationPage filesToScan={filesToScan}/>
		);

In all cases, it fails when it encounters some JSX (<App buttonText={button_text}/>, <ItemsToScan, <MOScanUILocalizationPage).

I don't know if that's a known limitation (from the docs, it doesn't look like it is) and how I can get around it.

Running Jest, of course, works without problems, as these are tests we have for a few years already.

I can share the source code if needed, but with other more private means, as it's from a commercial product.

After test.each support was added (#35) the new problem appeared. If the file has a test.each construction it will upload as one test with template string as name. This situation makes problems with runs because such type of test can't be matched on the platform.
The test parser should add every test from test.each construction as a separate test with the right ID

Is your feature request related to a problem? Please describe.
when we try to upload the test XML report then all test suits will be uploaded and an appropriate folder will be created even if there are no test suits tag references in the test suits name

Describe the solution you'd like
would be great to ignore test suits without @ tag references in the name

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Source code example Playwrigh typescript:

Testomat shows:

Auto-assign Test IDs with --typescript option reformat code and cut out typings in test.

ENV: TS, Jest

STR:

1. In __tests__ folder create new file eg.test.ts
2. Add code:

const isEven = (n: number): boolean => n % 2 === 0;

const cases: Array<[number, boolean]> = [
    [2, true],
    [0, true],
    [1, false],
    [-2, true],
    [-1, false],
    [Math.PI, false],
    [50, true],
    [75, false]
]

describe('Check if number is even', () => {
    it.each(cases)('The %d is even %p', (number, expected ) => {
        expect(isEven(number)).toBe(expected)
    })
})

3. Run update ids command

check-tests --typescript Jest "__tests__/*{.,_}{test,spec}.ts" --update-ids

AR:

1. Code had reformatted
2. Typings had removed

const isEven = (n) => n % 2 === 0;
const cases = [[2, true], [0, true], [1, false], [-2, true], [-1, false], [Math.PI, false], [50, true], [75, false]];
describe('Check if number is even @S298ffaa1', () => {
  it.each(cases)('The %d is even %p', (number, expected) => {
    expect(isEven(number)).toBe(expected);
  });
});

ER:

const isEven = (n: number): boolean => n % 2 === 0;

const cases: Array<[number, boolean]> = [
    [2, true],
    [0, true],
    [1, false],
    [-2, true],
    [-1, false],
    [Math.PI, false],
    [50, true],
    [75, false]
]

describe('Check if number is even @S298ffaa1', () => {
    it.each(cases)('The %d is even %p', (number, expected ) => {
        expect(isEven(number)).toBe(expected)
    })
})

When I have skipped describes, I get an error like on the screenshot. "Skip" at the test level works properly

test.describe.skip('Email notifications', () => {
	test.describe('Some', () => {

To Reproduce

1. import tests with check-tests v 0.8.5 with --update-ids to Testomat project
2. import tests with check-tests v 0.8.5 with --update-ids again to the project
3. see warning in terminal:
   ! Previously set Test IDs detected, new IDs ignored
   ! Clean previously set Test IDs to override them
   ! Run script with DEBUG="testomatio:*" flag to get more info of affected tests
   0 files updated.

Users should use string literals instead of + in test titles:

BAD                                           GOOD
'test' + id + ' very important' => `test ${id} important`

• Docs to use string literals instead (Most top, in Requirements)
• Try to detect concatenation (or any extra things except string or string literal)

If use parallelization of tests in one file - tests with test.concurrent (Jest doc) do not immediately import into the testomat.io.
Workaround: first import simply test to assign an id, and then add concurrent

Describe the bug
There seems to be a no way to get rid of empty test suites

To Reproduce
I have tried to import test with both options

After import - empty suite remains available

Expected behavior
When noempty is sent as true - Testomatio removes empty suites

Describe the bug
Upon importing automated tests from Playwright to the project with manual tests with the same names as auto tests, it can be noticed that the manual and automated tests do not match the corresponding manual tests.

To Reproduce
Steps to reproduce the behavior:

1. Create a classic project with a manual test with a set name
2. Import Playwright test with the same name

Expected behavior
Manual and automated test cases merge

Actual behavior
Tests stay separated

Desktop (please complete the following information):

• OS: macOS 13.5.2 (22G91)
• Browser Chrome
• Browser Version 116.0.5845.187
• Application: production

I've tried to add the test ID so to keep it in sync, then ran the check-test command again and got that:

I'm probably doing something wrong, but I'm not sure what this is.

Describe the bug
Testomatio API incorrectly parse parameterised test value

To Reproduce
Execute parameterised test
Ex. request to API

{
  'artifacts': [],
   'code': None,
   'example': None,
   'message': None,
   'run_time': 0.7919288370176218,
   'stack': None,
   'status': None,
   'steps': None,
   'suite_id': None,
   'suite_title': 'test_local_file_operations_via_cli.py',
   'test_id': 'f30bd3b0',
   'title': 'Version 1.0.0 @Tf30bd3b0'
}

Expected behavior
Test is displayed in test run with 1.0.0 parameter

Actual behaviour
Test is displayed in test run with 1 parameter

Screenshots

https://app.crisp.chat/website/4192ab4e-8b76-4b7e-a8c4-ea66dea249e7/inbox/session_247e0042-8301-4ce9-9ac1-6dab21470700/

If I provide non-lowercased framework name check-tests reports to find 0 test cases:

➜  TESTOMATIO={xxx} npx check-tests codeceptJS "**/**_test.js" -d todomvc-tests
 🤩 Tests checker by testomat.io

SHOWING CODECEPTJS TESTS FROM **/**_test.js:



TOTAL 0 TESTS FOUND


 🚀 Sending data to testomat.io

Once I change this to codeceptjs it is fine.

• It should lowercase the framework name and search for it
• If 0 tests is found it should show proper message and probably exit with 1 code

The script that we use to update the test ids does not add new test ids for added test scenarios

Script: TESTOMATIO=b0f2514004q7 npx check-tests@latest CodeceptJS "**/*{.,_}{test,spec}.ts" --typescript --update-ids

Example file

More info on this slack thread.

I tried to run a CodeceptJS scenario as a Manual Run on Testomat.io but it failed, as it tried to use the default configuration in ./codecept.conf.js which defines localhost.

Expected behavior:

The test should use codecept.ci.conf.js which defines a publicly available live URL of my production server.

When nested suites have tests with the same titles - they should be taken into account in the final count of all tests.

Link to thread https://testomatio.slack.com/archives/C051DR0CK6W/p1684921657414229?thread_ts=1680431321.338879&cid=C051DR0CK6W

State is a parameterized parameter

• create a jest project
• import tests
• update one test with ID manually in source doe
• import tests
• delete test with ID in testomat
• do import again

Expected failure to be shown however import process should be finished for other cases.

This follows on from issue #79 "stable branch gives Node.js 12 deprecation warning".

The stable branch recommended in the README documentation is unchanged since commit 0ea638f of @DavertMik on Sep 25, 2021.

See stable...master

If fixes and features are added to the default master branch, they will not be used by GitHub workflows such as .github/workflows/main.yml in this repository.

This can be seen in the "Set up job" job for instance in job 8219074452

Download action repository 'testomatio/check-tests@stable' (SHA:0ea638fcec1820cf2e7b9854fdbdd04128a55bd4)

from the full job log

Current runner version: '2.303.0'
Operating System
Runner Image
Runner Image Provisioner
GITHUB_TOKEN Permissions
Secret source: None
Prepare workflow directory
Prepare all required actions
Getting action download info
Download action repository 'actions/checkout@v1' (SHA:50fbc622fc4ef5163becd7fab6573eac35f8462e)
Download action repository 'actions/setup-node@v2' (SHA:1f8c6b94b26d0feae1e387ca63ccbdc44d27b561)
Download action repository 'testomatio/check-tests@stable' (SHA:0ea638fcec1820cf2e7b9854fdbdd04128a55bd4)
Complete job name: Check Tests

Suggestion

Merge the latest release 0.8.12 into the stable branch.

• auto assign ids from testomat
• manual ids assignment
• sync with testomat.io

Support following actions:

• Importing Into Project
• Disable Detached Tests
• Synchronous Import
• Auto-assign Test IDs in Source Code
• Keep Test IDs Between Projects
• Clean Test IDs
• Import Into a Branch
• Keep Structure of Source Code
• Delete Empty Suites
• Import Into a Specific Suite

Is your feature request related to a problem? Please describe.
Some of our automation tests fulfill the requirements of several test case ID's from our test management system. Currently it seems only one-to-one mapping exists between a single automation test and single test case that is imported or created in testomatio

Describe the solution you'd like
The ability to map a single automation test to multiple test cases.

Describe alternatives you've considered
Rewriting automation tests to be one to one, however with 1k+ tests, it could take considerable time.

Additional context
In a framework like Playwright, we have the ability to create a test which includes several test.step methods. It might be helpful if we can identify which portion of the test applies to individual test cases. From my understanding, most frameworks offer the ability have soft assertions which continue on the test if one step fails. Currently, we have multiple test cases tied to a single automation test setup as follows however:

        test.describe("Add/Edit user (parallel)", () => {
            for (const role of roles1) {
                baseTest.describe(`[C88, C81641, C8209, C8199, C43080, C45741]: ${role.groupName}`, () => {
                    baseTest.use({ storageState: `${role.storageStatePath}` });
    
                    baseTest(`Add new Worker`, async ({ userPage, random, apiRequest, common, request }) => {
                        await apiRequest.createAgency(request, random.agencyName);
                        const officeID = await apiRequest.createBillingOffice(request, random.officeName);
                        const campusID = await apiRequest.createCampus(request, random.campusName);
                        await apiRequest.createFacility(request, random.facilityName, random.rand, officeID, campusID);
                        await Promise.all([
                            userPage.page.goto(`${process.env.BASE_URL}admin`),
                            userPage.page.waitForResponse(resp => resp.url().includes('api/v1/users/table') && resp.status() == 200)
                        ])
                        await userPage.usersPO.openAddUserPage();
                        await userPage.addEditUserPO.addWorker(random.userEmail, random.userPhoneNumber, random.facilityName, random.agencyName, role.groupName);
                        await common.goToUsersPage();
                        await userPage.usersPO.verifyTheUserExist(random.userEmail);
                        await userPage.usersPO.openFirstUserPage();
                        await expect(userPage.addEditUserPO.submittionStatusButton).toContainText('accepted');
                        await userPage.addEditUserPO.verifyWorkerInfo(random.userEmail, random.userPhoneNumber, random.facilityName, random.agencyName, role.groupName);
                    })
                })
           }

This approach is less ideal but what we're currently using.

Users get error Validation failed: Title cannot have only a blank title with only tags when there are empty Suite or Test titles.

here is a link for slack thread https://testomatio.slack.com/archives/C013RBWCXJA/p1690447596532989

https://github.com/testomatio/check-tests/pull/83/checks

Error: Resource not accessible by integration
RequestError [HttpError]: Resource not accessible by integration
    at /home/runner/work/_actions/testomatio/check-tests/stable/node_modules/@octokit/request/dist-node/index.js:66:23
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async run (/home/runner/work/_actions/testomatio/check-tests/stable/index.js:119:7) {
  status: 403,

codeceptjs/CodeceptJS#2138 (comment)

Is your feature request related to a problem? Please describe.

Playwrigh has new tags format. Need to update check-tests to import tags correctly to Testomat.

• auto assign ids from testomat
• manual ids assignment
• sync with testomat.io

Support following actions:

• Importing Into Project
• Disable Detached Tests
• Synchronous Import
• Auto-assign Test IDs in Source Code
• Keep Test IDs Between Projects
• Clean Test IDs
• Import Into a Branch
• Keep Structure of Source Code
• Delete Empty Suites
• Import Into a Specific Suite