Giter Club home page Giter Club logo

wdio-rerun-service's Introduction

WebdriverIO Re-run Service

wdio-rerun-service CI npm npm bundle size GitHub issues

This service tracks failing Mocha or Jasmine tests and Cucumber scenarios executed within the WebdriverIO test framework. It will allow failing or unstable tests or scenarios to be re-run.

NOTE: Users running WebdriverIO 7.x.x, use version 1.7.x version of this service. If you are on the latest major version of 8.x.x, use the latest 2.x.x version of this service.

Re-run vs. Retry

The retry logic built into WebdriverIO for Cucumber and Mocha/Jasmine is helpful for handling flaky steps in Cucumber and Mocha/Jasmine. Retrying in each framework has caveats:

  • Cucumber: It does not take into account that some steps may not be able to be retried in the middle of a test. Running a step twice may break the rest of the Scenario or it may not be possible in the test context.
  • Mocha/Jasmine: The retry logic may be applied to an individual test, however, this is still done in real-time and perhaps does not account for temporal issues or network connectivity problems.

The main distinctions of the re-run:

  • Will re-run an entire individual Cucumber Scenario and not just a single step
  • Enables an entire spec file to be re-run after a main test execution is complete
  • May be copied and executed locally (retry cannot)
  • Can still be used in conjunction with retry methods
  • Does not require any code change to apply retry logic to flaky or problematic tests

It is recommended to take some time to evaluate the options available. A hybrid solution may be the best solution to provide the best real and actionable test results.

Installation

Using npm:

npm install wdio-rerun-service

or using yarn:

yarn add wdio-rerun-service

After package installation is complete, add it to services array in wdio.conf.js:

// wdio.conf.js
import RerunService from 'wdio-rerun-service';
export const config = {
    // ...
    services: [RerunService, {
        // ...
    }]
};

Instructions on how to install WebdriverIO can be found here.

Usage

By design, this service does not automatically re-run failed tests.

After WebDriver.IO has completed execution, if failures or unstable tests/scenarios are found a file will be at rerunScriptPath or by default ./rerun.sh (see Configuration).

Conditional Re-run

Every teams re-run needs will differ execution could be based on any number of factors, this is an example of how to accomplish a conditional re-run based on # of failures.

attemptRerun.sh

Executes ./rerun.sh if less than 25 failures have been found in last execution of WebDriver.IO.

#!/usr/bin/env bash
MAX_TO_RERUN=${MAX_TO_RERUN:=25}
if [ -f "rerun.sh" ]; then
  echo "[rerun.sh] file exits, checking total failures."
  NUMBER_OF_FAILURES=$(grep "\-\-spec" -o rerun.sh | wc -l | xargs)
	if [ "$MAX_TO_RERUN" -gt "$NUMBER_OF_FAILURES" ]; then
    echo "Re-running $NUMBER_OF_FAILURES failed scenarios!"
    . ./rerun.sh
	else
    echo "Skipping re-run, expected < $MAX_TO_RERUN failures to qualify execution for re-run, got $NUMBER_OF_FAILURES failures."
	fi
else
  echo "No [rerun.sh] file exits, skipping re-run!"
fi
Bash Re-run Command

Execute in shell

. ./attemptRerun.sh
Integrate with NPM

Add task in package.json

"attempt-rerun": ". ./attemptRerun.sh"
NPM Re-run Command

Execute in shell

npm run attempt-rerun

Configuration

The following options may be added to the wdio.conf.js file. To define options for the service you need to add the service to the services list in the following way:

// wdio.conf.js
import RerunService from 'wdio-rerun-service';
export const config = {
    // ...
    services: [
        [RerunService, {
            // Re-run service options here...
        }]
    ],
    // ...
};

rerunDataDir

Directory where all the re-run JSON data will be kept during execution.

Type: String

Default: ./results/rerun

Example:

import RerunService from 'wdio-rerun-service';
export const config = {
    // ...
    services: [
        [RerunService, {
            rerunDataDir: './custom-rerun-directory'
        }]
    ],
    // ...
}

rerunScriptPath

Path to write re-run Bash script.

Type: String

Default: ./rerun.sh

Example:

import RerunService from 'wdio-rerun-service';
export const config = {
    // ...
    services: [
        [RerunService, {
            rerunScriptPath: './custom-path-for-rerun.sh'
        }]
    ],
    // ...
}

ignoredTags

(Cucumber-only) Set of Cucumber tags to exclude. If scenario contains a tag, the re-run service will skip analysis.

Type: Array

Default: []

Example:

import RerunService from 'wdio-rerun-service';
export const config = {
    // ...
    services: [
        [RerunService, {
            ignoredTags: ['@known_bug']
        }]
    ],
    // ...
}

commandPrefix

Prefix which will be added to the re-run command that is generated.

Type: String

Default: ''

Example:

import RerunService from 'wdio-rerun-service';
export const config = {
    // ...
    services: [
        [RerunService, {
            commandPrefix: "VARIABLE=true"
        }]
    ],
    // ...
}

customParameters

Parameters which will be added to the re-run command that is generated. Can be used with commandPrefix.

Type: String

Default: ''

Example:

import RerunService from 'wdio-rerun-service';
export const config = {
    // ...
    services: [
        [RerunService, {
            customParameters: "--foobar"
        }]
    ],
    // ...
}

wdio-rerun-service's People

Contributors

aizaztoppa avatar cesar-rivera avatar esaari avatar jansmitmans avatar lamkovod avatar nav-2d avatar scg82 avatar seanpoulter avatar sw-carlos-cristobal avatar wdio-bot avatar yopasa94 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

wdio-rerun-service's Issues

Option to add tests manually to rerun queue

I think it might be a good option to enable adding tests to the re-run queue rather adding all the failed tests by default.

Flow:
Test failed - Check conditions - if pass, add to re-run - if not skip adding to re-run

Generate line number at Example data line of the failing case rather than the Scenario Outline line

1 Scenario Outline: This is a scenario outline 
2    When I start the test
3    And I use the <testData> for <testCase>
4
5     Examples:
6     | testCase           | testData    |
7     | case1              | value1      |
8     | case2              | value2      |
9     | case3              | value3      |

Currently, for the above Scenario Outline, if the test scenario with say, the second row of Example data fails, then rerun service will generate the rerun scenaroisLineNumber using the line number of the whole Scenario Outline (line number 1 here) instead of the line number of the Example data of the failing scenario (line number 8 here). Consequently, all three scenarios will be rerun instead of the only failing scenario.

It would be more efficient if we could improve this behaviour.

Re-run fails when using Cucumber

When using cucumber framework, in a scenario of a failure in one of the steps, the rerun is not performed and I get an error regarding missing suite name:
image
image
image

rerun script is invalid on Windows

wdio: 7.0.7
rerun: 1.7.2

Hello,

The generated rerun.bat file is not a valid command for Windows ( at least on my machine)

DISABLE_RERUN=true node_modules/.bin/wdio run --spec=C:/Apps/test/features/01_test.feature:5

DISABLE_RERUN=true. this is not recognised

should be set DISABLE_RERUN= true if it is an env variable

Working line would be:

set DISABLE_RERUN=true && npx wide run ./wdio.conf.ts --spec=C:/Apps/test/features/01_test.feature:5

Do I miss something ?

Thanks

Attila

Re-run service does not work if the before hook fails when using Mocha

Here is an example of how my spec file is structured.

describe(Log into my google account, () => {
before(() => {
openAndLogin('https://www.google.com', username, password);
});

after(() => {
logout();
});

it('Should be able to update my profile pic', () => {
....
});

it(Should update my address, () => {
....
});

Versions:

"@wdio/cli": "7.7.3",
"@wdio/mocha-framework": "7.7.3",
"wdio-rerun-service": "^1.7.0"

If my openAndLogin() function fails in the before hook, the re-run service doesn't seem to recognise that and doesn't generate the rerun.sh

Possible to rerun Cucumber Scenario based on condition/exception

I would like to rerun failed Cucumber Scenarios if there is specific condition or exception comes. Let's assume that Scenario failed due to the ongoing deployment of the service or Cloud Device restarted or interrupted in between the execution.

In all of these cases we get specific exceptions and if they come, retry the scenario else just marked failed without retry.

wdio-rerun-service appears to be cucumber specific

Unless I'm missing something this service seems to be cucumber specific? But your read me suggests it would work for other frameworks.

When running a suite with failing tests I see "Re-run service did not detect any non-passing scenarios." seems to only be checking for non-passing scenarios not non-passing tests.

I've done the setup outlined in the readme, please let me know if I might be missing something.

wdio-rerun-service not working with cucumber

I am trying wdio-rerun-service with cucumber framework.Following are the logs.
In logs I can see that it has generated rerun.sh file. But looks like it is not getting executed.
I am using latest version of service 0.0.9.
###############################################################################
Execution of 1 spec files started at 2020-09-17T10:31:49.785Z

[0-0] RUNNING in chrome - C:########\cucumber-boilerplate-master\cucumber-boilerplate-master\src\features\MakeMyTrip2.feature
[0-0] Re-run service is activated. Data directory: ./results/rerun
[0-0] Error in "Test MakeMyTrip Login: Test if user is able to serach hotels on MakeMyTrip portal - MakeMyTrip2 Scenario1: And I click on the element "div.selectHtlCity123""
expect(received).toBeGreaterThanOrEqual(expected)

Expected: >= 1
Received: 0
[0-0] Re-run service is inspecting non-passing scenario.
[0-0] Scenario location: src\features\MakeMyTrip2.feature:9
Scenario tags:
[0-0] FAILED in chrome - C:############\cucumber-boilerplate-master\cucumber-boilerplate-master\src\features\MakeMyTrip2.feature
Re-run script has been generated @ ./rerun.sh

"spec" Reporter:

[Chrome 85.0.4183.102 win32 #0-0] Spec: C:###############\cucumber-boilerplate-master\cucumber-boilerplate-master\src\features\MakeMyTrip2.feature
[Chrome 85.0.4183.102 win32 #0-0] Running: Chrome (v85.0.4183.102) on win32
[Chrome 85.0.4183.102 win32 #0-0] Session ID: 48850ec7-ed94-40db-bbb7-4930b9b48562
[Chrome 85.0.4183.102 win32 #0-0]
[Chrome 85.0.4183.102 win32 #0-0] Test MakeMyTrip Login
[Chrome 85.0.4183.102 win32 #0-0] As a developer
[Chrome 85.0.4183.102 win32 #0-0] I want to be able to test if user is able to search for hotels Scenario
[Chrome 85.0.4183.102 win32 #0-0] on MakeMyTrip site
[Chrome 85.0.4183.102 win32 #0-0]
[Chrome 85.0.4183.102 win32 #0-0] Test if user is able to serach hotels on MakeMyTrip portal - MakeMyTrip2 Scenario1
[Chrome 85.0.4183.102 win32 #0-0] As a developer
[Chrome 85.0.4183.102 win32 #0-0] I want to be able to test if user is able to search for hotels Scenario
[Chrome 85.0.4183.102 win32 #0-0] on MakeMyTrip site
[Chrome 85.0.4183.102 win32 #0-0]
[Chrome 85.0.4183.102 win32 #0-0] ✓ Given I open the site "/"
[Chrome 85.0.4183.102 win32 #0-0] ✓ When I doubleclick on the element "li.menu_Hotels a"
[Chrome 85.0.4183.102 win32 #0-0] ✖ And I click on the element "div.selectHtlCity123"
[Chrome 85.0.4183.102 win32 #0-0] - And I wait on element "div[role='combobox'] > .react-autosuggest__input" for 5000ms to exist
[Chrome 85.0.4183.102 win32 #0-0] - And I set "Pune" to the inputfield "div[role='combobox'] > .react-autosuggest__input"
[Chrome 85.0.4183.102 win32 #0-0]
[Chrome 85.0.4183.102 win32 #0-0] 2 passing (19.6s)
[Chrome 85.0.4183.102 win32 #0-0] 1 failing
[Chrome 85.0.4183.102 win32 #0-0] 2 skipped
[Chrome 85.0.4183.102 win32 #0-0]
[Chrome 85.0.4183.102 win32 #0-0] 1) Test if user is able to serach hotels on MakeMyTrip portal - MakeMyTrip2 Scenario1 And I click on the element "div.selectHtlCity123"
[Chrome 85.0.4183.102 win32 #0-0] expect(received).toBeGreaterThanOrEqual(expected)

Expected: >= 1
Received: 0
[Chrome 85.0.4183.102 win32 #0-0] Error: expect(received).toBeGreaterThanOrEqual(expected)
[Chrome 85.0.4183.102 win32 #0-0]
[Chrome 85.0.4183.102 win32 #0-0] Expected: >= 1
[Chrome 85.0.4183.102 win32 #0-0] Received: 0
[Chrome 85.0.4183.102 win32 #0-0] at _default (C:#####\cucumber-boilerplate-master\cucumber-boilerplate-master\src\support\lib/checkIfElementExists.js:27:37)
[Chrome 85.0.4183.102 win32 #0-0] at World._default (C:#####\cucumber-boilerplate-master\cucumber-boilerplate-master\src\support\action/clickElement.js:22:5)
[Chrome 85.0.4183.102 win32 #0-0] at World.executeSync (C:#####\cucumber-boilerplate-master\cucumber-boilerplate-master\node_modules@wdio\sync\build\index.js:5
6:18)
[Chrome 85.0.4183.102 win32 #0-0] at C:########\cucumber-boilerplate-master\cucumber-boilerplate-master\node_modules@wdio\sync\build\index.js:80:70

Spec Files: 0 passed, 1 failed, 1 total (100% completed) in 00:00:24

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
##############################################################################

Re-run service always generates rerun.sh file although the test result was passed or failed

hi, Re-run service is so amazing, it helps me a lot.
Now I am using this one with webdriverIo + typescript and facing some issues:

  1. Re-run service always generates rerun.sh file although the test result was passed or failed. The rerun.sh contains all of the feature files that were run
  2. When I tried sh rerun.sh , it cannot start the WebdriverIO
    I see the command for running webdriverIo on rerun.sh is node_modules/.bin/wdio run --spec=/Users/hungnguyen/Documents/Mantra-web/features/login.feature:3
    Missing the path to wdio config file. It should be like this one: node_modules/.bin/wdio run wdio.conf.ts --spec=/Users/hungnguyen/Documents/Mantra-web/features/login.feature:3

This is my package.json

{
    "name": "webdriverio-tests",
    "version": "0.1.0",
    "private": true,
    "devDependencies": {
        "@wdio/allure-reporter": "^7.19.1",
        "@wdio/appium-service": "^7.19.1",
        "@wdio/cli": "^7.19.4",
        "@wdio/crossbrowsertesting-service": "^7.19.4",
        "@wdio/cucumber-framework": "^7.19.4",
        "@wdio/devtools-service": "^7.19.4",
        "@wdio/local-runner": "^7.19.4",
        "@wdio/selenium-standalone-service": "^7.19.1",
        "@wdio/spec-reporter": "^7.19.1",
        "chromedriver": "^100.0.0",
        "ts-node": "^10.7.0",
        "typescript": "^4.6.3",
        "wdio-cucumberjs-json-reporter": "^4.4.1",
        "wdio-docker-service": "^3.2.0",
        "wdio-eslinter-service": "0.0.4",
        "wdio-html-nice-reporter": "^8.0.0",
        "wdio-json-reporter": "^3.0.0",
        "wdio-rerun-service": "^1.7.3",
        "wdio-timeline-reporter": "^5.1.4",
        "wdio-video-reporter": "^3.2.1",
        "wdio-vscode-service": "^2.0.1",
        "wdio-wait-for": "^2.2.5",
        "webdriverio": "^7.19.5"
    },
    "scripts": {
        "wdio": "wdio run test/wdio.conf.ts",
        "re-run": "sh rerun.sh "
    }
}
```

This is my config file
```
import type { Options } from "@wdio/types";
import RerunService from 'wdio-rerun-service' ;

export const config: Options.Testrunner = {
  //
  // ====================
  // Runner Configuration
  // ====================
  //
  //
  // =====================
  // ts-node Configurations
  // =====================
  //
  // You can write tests using TypeScript to get autocompletion and type safety.
  // You will need typescript and ts-node installed as devDependencies.
  // WebdriverIO will automatically detect if these dependencies are installed
  // and will compile your config and tests for you.
  // If you need to configure how ts-node runs please use the
  // environment variables for ts-node or use wdio config's autoCompileOpts section.
  //

  autoCompileOpts: {
    autoCompile: true,
    // see https://github.com/TypeStrong/ts-node#cli-and-programmatic-options
    // for all available options
    tsNodeOpts: {
      transpileOnly: true,
      project: "test/tsconfig.json",
    },
    // tsconfig-paths is only used if "tsConfigPathsOpts" are provided, if you
    // do please make sure "tsconfig-paths" is installed as dependency
    // tsConfigPathsOpts: {
    //     baseUrl: './'
    // }
  },
  port: 4723,
  //
  // ==================
  // Specify Test Files
  // ==================
  // Define which test specs should run. The pattern is relative to the directory
  // from which `wdio` was called.
  //
  // The specs are defined as an array of spec files (optionally using wildcards
  // that will be expanded). The test for each spec file will be run in a separate
  // worker process. In order to have a group of spec files run in the same worker
  // process simply enclose them in an array within the specs array.
  //
  // If you are calling `wdio` from an NPM script (see https://docs.npmjs.com/cli/run-script),
  // then the current working directory is where your `package.json` resides, so `wdio`
  // will be called from there.
  //
  specs: ["./features/**/*.feature"],
  // Patterns to exclude.
  exclude: [
    // 'path/to/excluded/files'
  ],
  //
  // ============
  // Capabilities
  // ============
  // Define your capabilities here. WebdriverIO can run multiple capabilities at the same
  // time. Depending on the number of capabilities, WebdriverIO launches several test
  // sessions. Within your capabilities you can overwrite the spec and exclude options in
  // order to group specific specs to a specific capability.
  //
  // First, you can define how many instances should be started at the same time. Let's
  // say you have 3 different capabilities (Chrome, Firefox, and Safari) and you have
  // set maxInstances to 1; wdio will spawn 3 processes. Therefore, if you have 10 spec
  // files and you set maxInstances to 10, all spec files will get tested at the same time
  // and 30 processes will get spawned. The property handles how many capabilities
  // from the same test should run tests.
  //
  maxInstances: 10,
  //
  // If you have trouble getting all important capabilities together, check out the
  // Sauce Labs platform configurator - a great tool to configure your capabilities:
  // https://saucelabs.com/platform/platform-configurator
  //
  capabilities: [
    {
      // maxInstances can get overwritten per capability. So if you have an in-house Selenium
      // grid with only 5 firefox instances available you can make sure that not more than
      // 5 instances get started at a time.
      maxInstances: 5,
      //
      browserName: "chrome",
      acceptInsecureCerts: true,
      // If outputDir is provided WebdriverIO can capture driver session logs
      // it is possible to configure which logTypes to include/exclude.
      // excludeDriverLogs: ['*'], // pass '*' to exclude all driver session logs
      // excludeDriverLogs: ['bugreport', 'server'],
    },
  ],
  //
  // ===================
  // Test Configurations
  // ===================
  // Define all options that are relevant for the WebdriverIO instance here
  //
  // Level of logging verbosity: trace | debug | info | warn | error | silent
  logLevel: "info",
  //
  // Set specific log levels per logger
  // loggers:
  // - webdriver, webdriverio
  // - @wdio/browserstack-service, @wdio/devtools-service, @wdio/sauce-service
  // - @wdio/mocha-framework, @wdio/jasmine-framework
  // - @wdio/local-runner
  // - @wdio/sumologic-reporter
  // - @wdio/cli, @wdio/config, @wdio/utils
  // Level of logging verbosity: trace | debug | info | warn | error | silent
  // logLevels: {
  //     webdriver: 'info',
  //     '@wdio/appium-service': 'info'
  // },
  //
  // If you only want to run your tests until a specific amount of tests have failed use
  // bail (default is 0 - don't bail, run all tests).
  bail: 0,
  //
  // Set a base URL in order to shorten url command calls. If your `url` parameter starts
  // with `/`, the base url gets prepended, not including the path portion of your baseUrl.
  // If your `url` parameter starts without a scheme or `/` (like `some/path`), the base url
  // gets prepended directly.
  baseUrl: "http://localhost",
  //
  // Default timeout for all waitFor* commands.
  waitforTimeout: 10000,
  //
  // Default timeout in milliseconds for request
  // if browser driver or grid doesn't send response
  connectionRetryTimeout: 120000,
  //
  // Default request retries count
  connectionRetryCount: 3,
  //
  // Test runner services
  // Services take over a specific job you don't want to take care of. They enhance
  // your test setup with almost no effort. Unlike plugins, they don't add new
  // commands. Instead, they hook themselves up into the test process.
  services: [
    [RerunService, {
      // ...
  }],
  [
    'chromedriver',
    {
      logFileName: 'wdio-chromedriver.log', // default
      outputDir: 'driver-logs', // overwrites the config.outputDir
      args: ['--silent'],
    },
  ],
    // [
        // 'selenium-standalone',
        // {
        //   logPath: 'logs',
        //   installArgs: {
        //     chrome: { version: '98.0.4758.102' },
        //     drivers: {
        //       chrome: {
        //         version: '98.0.4758.102'
        //       }
        //     }
        //   }, // drivers to install
        //   args: {
        //     chrome: { version: '98.0.4758.102' },
        //     drivers: {
        //       chrome: {
        //         version: '98.0.4758.102'
        //       }
        //     }
        //   } // drivers to use
        // }
      // ],
    // "vscode",
    // "devtools",
    // "appium",
    // "crossbrowsertesting",
    // "eslinter",
    // "docker",
    // "docker",
    "rerun",
  ],

  // Framework you want to run your specs with.
  // The following are supported: Mocha, Jasmine, and Cucumber
  // see also: https://webdriver.io/docs/frameworks
  //
  // Make sure you have the wdio adapter package for the specific framework installed
  // before running any tests.
  framework: "cucumber",
  //
  // The number of times to retry the entire specfile when it fails as a whole
  specFileRetries: 3,
  //
  // Delay in seconds between the spec file retry attempts
  specFileRetriesDelay: 5,
  //
  // Whether or not retried specfiles should be retried immediately or deferred to the end of the queue
  // specFileRetriesDeferred: false,
  //
  // Test reporter for stdout.
  // The only one supported by default is 'dot'
  // see also: https://webdriver.io/docs/dot-reporter
  reporters: [
    "spec",
    // ["allure", { outputDir: "allure-results" }],
    // "video",
    // "json",
    // "cucumberjs-json",
    // "timeline",
    // "html-nice",
  ],

  //
  // If you are using Cucumber you need to specify the location of your step definitions.
  cucumberOpts: {
    // <string[]> (file/dir) require files before executing features
    require: ["./features/step-definitions/steps.ts"],
    // <boolean> show full backtrace for errors
    backtrace: false,
    // <string[]> ("extension:module") require files with the given EXTENSION after requiring MODULE (repeatable)
    requireModule: [],
    // <boolean> invoke formatters without executing steps
    dryRun: false,
    // <boolean> abort the run on first failure
    failFast: false,
    // <boolean> hide step definition snippets for pending steps
    snippets: true,
    // <boolean> hide source uris
    source: true,
    // <boolean> fail if there are any undefined or pending steps
    strict: false,
    // <string> (expression) only execute the features or scenarios with tags matching the expression
    tagExpression: "",
    // <number> timeout for step definitions
    timeout: 60000,
    // <boolean> Enable this config to treat undefined definitions as warnings.
    ignoreUndefinedDefinitions: false,
  },

  //
  // =====
  // Hooks
  // =====
  // WebdriverIO provides several hooks you can use to interfere with the test process in order to enhance
  // it and to build services around it. You can either apply a single function or an array of
  // methods to it. If one of them returns with a promise, WebdriverIO will wait until that promise got
  // resolved to continue.
  /**
   * Gets executed once before all workers get launched.
   * @param {Object} config wdio configuration object
   * @param {Array.<Object>} capabilities list of capabilities details
   */
  onPrepare: function (config, capabilities) {
  },
  /**
   * Gets executed before a worker process is spawned and can be used to initialise specific service
   * for that worker as well as modify runtime environments in an async fashion.
   * @param  {String} cid      capability id (e.g 0-0)
   * @param  {[type]} caps     object containing capabilities for session that will be spawn in the worker
   * @param  {[type]} specs    specs to be run in the worker process
   * @param  {[type]} args     object that will be merged with the main configuration once worker is initialized
   * @param  {[type]} execArgv list of string arguments passed to the worker process
   */
  // onWorkerStart: function (cid, caps, specs, args, execArgv) {
  // },
  /**
   * Gets executed just after a worker process has exited.
   * @param  {String} cid      capability id (e.g 0-0)
   * @param  {Number} exitCode 0 - success, 1 - fail
   * @param  {[type]} specs    specs to be run in the worker process
   * @param  {Number} retries  number of retries used
   */
  // onWorkerEnd: function (cid, exitCode, specs, retries) {
  // },
  /**
   * Gets executed just before initialising the webdriver session and test framework. It allows you
   * to manipulate configurations depending on the capability or spec.
   * @param {Object} config wdio configuration object
   * @param {Array.<Object>} capabilities list of capabilities details
   * @param {Array.<String>} specs List of spec file paths that are to be run
   * @param {String} cid worker id (e.g. 0-0)
   */
  // beforeSession: function (config, capabilities, specs, cid) {
  // },
  /**
   * Gets executed before test execution begins. At this point you can access to all global
   * variables like `browser`. It is the perfect place to define custom commands.
   * @param {Array.<Object>} capabilities list of capabilities details
   * @param {Array.<String>} specs        List of spec file paths that are to be run
   * @param {Object}         browser      instance of created browser/device session
   */
  // before: function (capabilities, specs) {
  // },
  /**
   * Runs before a WebdriverIO command gets executed.
   * @param {String} commandName hook command name
   * @param {Array} args arguments that command would receive
   */
  // beforeCommand: function (commandName, args) {
  // },
  /**
   * Cucumber Hooks
   *
   * Runs before a Cucumber Feature.
   * @param {String}                   uri      path to feature file
   * @param {GherkinDocument.IFeature} feature  Cucumber feature object
   */
  // beforeFeature: function (uri, feature) {
  // },
  /**
   *
   * Runs before a Cucumber Scenario.
   * @param {ITestCaseHookParameter} world    world object containing information on pickle and test step
   * @param {Object}                 context  Cucumber World object
   */
  // beforeScenario: function (world, context) {
  // },
  /**
   *
   * Runs before a Cucumber Step.
   * @param {Pickle.IPickleStep} step     step data
   * @param {IPickle}            scenario scenario pickle
   * @param {Object}             context  Cucumber World object
   */
  // beforeStep: function (step, scenario, context) {
  // },
  /**
   *
   * Runs after a Cucumber Step.
   * @param {Pickle.IPickleStep} step             step data
   * @param {IPickle}            scenario         scenario pickle
   * @param {Object}             result           results object containing scenario results
   * @param {boolean}            result.passed    true if scenario has passed
   * @param {string}             result.error     error stack if scenario failed
   * @param {number}             result.duration  duration of scenario in milliseconds
   * @param {Object}             context          Cucumber World object
   */
  // afterStep: function (step, scenario, result, context) {
  // },
  /**
   *
   * Runs after a Cucumber Scenario.
   * @param {ITestCaseHookParameter} world            world object containing information on pickle and test step
   * @param {Object}                 result           results object containing scenario results
   * @param {boolean}                result.passed    true if scenario has passed
   * @param {string}                 result.error     error stack if scenario failed
   * @param {number}                 result.duration  duration of scenario in milliseconds
   * @param {Object}                 context          Cucumber World object
   */
  // afterScenario: function (world, result, context) {
  // },
  /**
   *
   * Runs after a Cucumber Feature.
   * @param {String}                   uri      path to feature file
   * @param {GherkinDocument.IFeature} feature  Cucumber feature object
   */
  // afterFeature: function (uri, feature) {
  // },

  /**
   * Runs after a WebdriverIO command gets executed
   * @param {String} commandName hook command name
   * @param {Array} args arguments that command would receive
   * @param {Number} result 0 - command success, 1 - command error
   * @param {Object} error error object if any
   */
  // afterCommand: function (commandName, args, result, error) {
  // },
  /**
   * Gets executed after all tests are done. You still have access to all global variables from
   * the test.
   * @param {Number} result 0 - test pass, 1 - test fail
   * @param {Array.<Object>} capabilities list of capabilities details
   * @param {Array.<String>} specs List of spec file paths that ran
   */
  // after: function (result, capabilities, specs) {
  // },
  /**
   * Gets executed right after terminating the webdriver session.
   * @param {Object} config wdio configuration object
   * @param {Array.<Object>} capabilities list of capabilities details
   * @param {Array.<String>} specs List of spec file paths that ran
   */
  // afterSession: function (config, capabilities, specs) {
  // },
  /**
   * Gets executed after all workers got shut down and the process is about to exit. An error
   * thrown in the onComplete hook will result in the test run failing.
   * @param {Object} exitCode 0 - success, 1 - fail
   * @param {Object} config wdio configuration object
   * @param {Array.<Object>} capabilities list of capabilities details
   * @param {<Object>} results object containing test results
   */
  // onComplete: function(exitCode, config, capabilities, results) {
  // },
  /**
   * Gets executed when a refresh happens.
   * @param {String} oldSessionId session ID of the old session
   * @param {String} newSessionId session ID of the new session
   */
  // onReload: function(oldSessionId, newSessionId) {
  // }
};

rerun.sh isn't being generated in WDIO v8

wdio-rerun-service has been working great for me on WDIO v7, but after updating to WDIO v8, it no longer generates the rerun.sh file, so it is no longer working. I have tried with multiple versions of wdio-rerun-service, all with the same result.

I created a simple project to demonstrate the issue. You can see the code HERE.

If it matters, the project is also using Cucumber.

Rerun service is not working with selenium-standalone & cucumber gherkin

Hello,

I followed the instruction as mentioned in https://webdriver.io/docs/wdio-rerun-service

But still rerun is not working for me. Below are the changes I made to package.json and wdio.config.ts file.

----------------------------------------- package.json ---------------------------------------------------
"wdio-rerun-service": "^1.7.2",

image

---------------------------------------- wdio.config**.ts** ----------------------------------------------
const RerunService = require('wdio-rerun-service');
..
..
..
services: [
[
'selenium-standalone',
{
drivers: { chrome: true, chromiumedge: 'latest' },
logs: 'logs',
},
],
[
RerunService,
{
rerunDataDir: './e2e/custom-rerun-directory',
rerunScriptPath: './e2e/custom-rerun-directory/custom-path-for-rerun.sh',
},
],
],

image

==============================================================

I dont know what I am missing. Do I need to add any code in hook.

`rerun.sh` not being generated in 2.x

rerun.sh is not being generated and service is failing with the following error:

[0-0] 2023-01-19T01:44:37.369Z ERROR @wdio/utils:shim: TypeError: Cannot read properties of undefined (reading 'framework')
[0-0]     at RerunService.afterScenario (/Users/esaari/code/jwplayer-commercial/test/jwp-wdio/node_modules/wdio-rerun-service/src/index.ts:106:19)

How to configure wdio-rerun-service for Cucumber

Hi,
I am trying to follow the instructions on how to configure this service, but it seems like i am missing something. I don't have a file named "wdio.conf.js" in my project and we don't configure "services" array. How \ where i am supposed to configure this service and define the options?

Re-run service does not work if scenario fails in a Background step

If we have a feature like this:

Feature: My Feature
  As an advanced cucumber user
  I like to use all the cucumber features in my features

  Background:
    Given this Step Fails

  Scenario: Test all the things
    Then this step will never be reached

When the background step fails, I would expect the scenario to be added to the rerun script, instead we get an error:

ERROR @wdio/utils:shim: TypeError: Cannot read property 'id' of null at ./node_modules/wdio-rerun-service/build/index.js:46:68

Looks like child.scenario.id does not exist as child only has child.background and not scenario.

Versions:

"@wdio/cli": "^7.6.0",
"@wdio/cucumber-framework": "^7.6.0",
"wdio-rerun-service": "^1.7.0"

Re-run service cannot get the scenarioLineNumber on afterScenario Hook for Cucumber

Environment (please complete the following information):

  • Node.js version: [14.16]
  • NPM version: [6.14.11]
  • webdriver.io version: [7.5.3]
  • @wdio/cucumber-framework version: [7.5.3]
  • wdio-rerun-service version: [1.7.0"]

Config of webdriver.io

cucumberOpts: {
    backtrace: false,
    requireModule: ['@babel/register'],
    failAmbiguousDefinitions: true,
    failFast: false,
    ignoreUndefinedDefinitions: false,
    names: [],
    snippets: true,
    source: true,
    profile: [],
    require: [
        './tests/steps/*.js',
    ],
    scenarioLevelReporter: false,
    order: 'defined',
    snippetSyntax: undefined,
    strict: true,
    // <string> (expression) only execute the features or scenarios with
    // tags matching the expression, see
    // https://docs.cucumber.io/tag-expressions/
    tagExpression: 'not @Pending',
    // <boolean> add cucumber tags to feature or scenario name
    tagsInTitle: false,
    timeout: 720000,
},

// ====================
// Appium Configuration
// ====================
services: [
    [
        'appium',
        {
            args: {
                relaxedSecurity: true,
            },
            command: 'appium',
            logPath: './'
        },
    ],
    [
        RerunService, 
        {
            rerunDataDir: './custom-rerun-directory',
            rerunScriptPath: './custom-path-for-rerun.sh',
        }
    ],

],

Describe the bug
When Re-run service is configured, the scenarioLineNumber cannot be obtained in the afterScenario hook, so that, the JSON data cannot be created and Re-run does not work properly

To Reproduce

  1. Configure Re-run service as above
  2. Run any .feature file getting one or more scenarios failed
  3. See how Re-run is not performed for failed scenarios

Expected behavior
Failed scenarios should be re-run

Snapshots

Additional context
I have uncomment all "console.log" on the index.js of "wdio-rerun-service" only for debug purpose, and this is the output :

[0-0] RUNNING in Android - /tests/features/login.feature
[0-0] Re-run service is activated. Data directory: ./custom-rerun-directory
[0-0] Error in "1: I expect that element "Next_Bun" from "OnboardingScreen" is displayed"
Error: selector needs to be typeof `string` or `function`
    at $ (/Users/-------/Documents/Repos/------/node_modules/@wdio/runner/build/index.js:189:42)
    at OnboardingScreen.getElement (/Users/---/Documents/Repos/---/tests/screenobjects/onboarding.screen.js:26:16)
    at World.<anonymous> (/Users/---/Documents/Repos/----/tests/steps/then.js:14:38)
[0-0] Re-run service did not detect any non-passing scenarios or tests.
[0-0] FAILED in Android - /tests/features/login.feature```

wdio-rerun-service

Hi Team ,

We are working on the "wdio-rerun-service" , I am executing multiple feature (for e.g. A.feature, B.feature, C.feature)files for my e2e cucumber test cases and once execution is completed, I should have multiple "json" files generated(for e.g. rerunA.json, rerunB.json, rerunC.json ) with each ".json" file tells me about the number of failures on each ".feature" file.

And since, these ".json" files are getting generated on the basis of current date and time, we have this behaviour when workers for test is being created at the same time then rerun.json files get generated with the same name and next failures gets overwritten with the old failures.

Because of this we are unable to get the list of all failures and eventually unable to rerun the all failed scenarios again.

https://github.com/webdriverio-community/wdio-rerun-service/blob/eca59de23dc1304d57c5041f44e6320f65011055/src/index.ts#L68

and we also tried to change the code to some random UUID generation, and it worked so you can also take a look this way to fix or advise us the workaround.

Feature: Add possibility to add flags/parameters to wdio rerun command.

Some users launching wdio with custom flags or parameters.
ReRunService should be able to handle this. It could be done with additional property.
Example:

        [
            RerunService,
            {
                customParameters: ['--firefox'],
            },
        ],

Generated .sh:
DISABLE_RERUN=true node_modules/.bin/wdio ./config/wdio.conf.js --firefox --spec=test/features/login/login-errors.feature:14

How to execute the re-run shell script after its been generated.

Hi,
I have followed the documentation available here: https://v7.webdriver.io/docs/wdio-rerun-service/#re-run-vs-retry
I am getting the expected output of a json file and a Shell script. So far so good.
[RerunService, { rerunDataDir: 'output/wdio-rerun-service', rerunScriptPath: 'output/wdio-rerun-script.sh', ignoredTags: ['@wip', '@web'], }],

The piece I am having trouble with is how to incorporate this into my run command/ package.json?

I tried this kind of thing within my scripts:
"wdio-rerun-ios": "MOBILE_OS=IOS wdio run src/config/wdio.local.conf.ts && output/wdio-rerun-script.sh"
then running yarn run wdio-rerun-ios

It, however, basically ignores the re-run script and the other way around it says it can't find the wdio config so that's a non-starter.
If anyone can point me in the right direction or to some useful documentation around this in general it would be greatly appreciated.

I realise this is not really an issue with the service but maybe an oversight in the docs/ or at least worthy of an example.

Thanks

Rerunning individual Cucumber scenario reruns entire feature file

This is not a problem with the rerun service itself, but due to an upstream issue somewhere in the main WDIO codebase:

webdriverio/webdriverio#7735

The rerun.sh file specifies a line number like so:

rerun.sh
-----------
DISABLE_RERUN=true node_modules/.bin/wdio ./wdio-configs/local-chrome.conf.js   --spec=/Users/esaari/code/jwplayer-commercial/test/cucumber-features/basic/basic_mp4.feature:5

Due to the bug listed above, the entire feature file is being run which is causing unintended scenarios to be rerun.

Leaving this issue open for visibility in case anybody running Cucumber is experiencing the same behavior. A fix was attempted by the original issue reporter, but abandoned: webdriverio/webdriverio#7752

Move project to WebdriverIO Community Org

Hey @mikesalvia ,

first off, thank you for creating this plugin and extending the WebdriverIO ecosystem!

I was wondering if you would be interested moving this repository into the WebdriverIO Community organisation. It would allow us to help maintain this package, automate a few things like releases and allow others to contribute to it and make the package even better. You would of course continue to have full access to it.

What do you think? Please let me know if you have any questions.

The Passed Scenarios was added into rerun.sh

This is my package.json:

"wdio-rerun-service": "^1.7.2"

This is the rerun JSON data
File 1:
[{"location":"/Users/hungnguyen/xxxxx-web/e2e/e2e/src/features/xxxxx/Login.feature:3","failure":"Error: \u001b[2mexpect(\u001b[22m\u001b[31mreceived\u001b[39m\u001b[2m).\u001b[22mtoEqual\u001b[2m(\u001b[22m\u001b[32mexpected\u001b[39m\u001b[2m) // deep equality\u001b[22m\n\nExpected: \u001b[32m\"xxxxx - LinkedSite\"\u001b[39m\nReceived: \u001b[31m\"Signin\"\u001b[39m\n at World._default (/Users/hungnguyen/xxxxx-web/e2e/e2e/src/support/check/checkTitle.js:17:19)"}]
File 2:
[{"location":"/Users/hungnguyen/xxxxx-web/e2e/e2e/src/features/xxxxx/SortResrouce.feature:6"}]

This is rerun.sh file:
DISABLE_RERUN=true node_modules/.bin/wdio e2e/config/wdio.conf.js --spec=/Users/hungnguyen/xxxxx-web/e2e/e2e/src/features/xxxxx/Login.feature:3 --spec=/Users/hungnguyen/xxxxx-web/e2e/e2e/src/features/xxxxx/SortResrouce.feature:6

I see it add all files regardless of pass or fail into rerun.sh

Rerun-service status map does not match with result

status map does not match with scenario result status, it produces an 'unknown' which writes rerun json files for all scenarios and appends them into the rerun script even if the scenarios have passed

image

Cucumber message enum contains UPPER CASE properties
image

Generate a new rerun.sh file after running the original

Hello, I just started using this package today and it is very useful. I have one question though, is there a way to generate a new rerun.sh file after running the original rerun.sh file?

For example say you run a set of feature files and a few scenarios fail and are added to the rerun.sh file. You now run the generated rerun.sh file and one of the scenarios fails again. Is there a way to generate a new rerun.sh file that only contains the one failing scenario?

I tried to do this by manually editing the originally generated rerun.sh file and changing the env variable DISABLE_RERUN to false but that did not work.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.