This is related to #15

Display the current version, the newer version, and the command to install it.

No additional test cases must be produced when all the UI Elements receive (fixed) values

Since a complete constraint solver is not available, the test case FORMAT_INVALID should be avoided when value or length constraints are defined, because it is not able to consider them.

It is not generating the Test Case tag @fail when two or more UI Elements receive invalid values in a Test Case, but one of them does not have Otherwise steps defined for the affected property.

This problem could be circumvented by using --comb-invalid=1. After the fix, this parameter is no longer needed.

Allow Test Events in the CodeceptJS plugin

First version could allow database scripts.
Second version could allow console commands.
Third version could allow file operations.

Columns and lines of Abstracts Tests Scripts are being produced incorrectly. Thus, test scripts keep a wrong reference to them.

If you can create any documentation, this project needs your help!

1- Creating tutorials.
2- Creating videos to explain some part of the language or some interesting feature or trick.
3- Adding examples.
4- Improving the current language documentation.

Stuff in English or Portuguese is welcome!

Implement Test Events

First version could allow database scripts.
Second version could allow console commands.
Third version could allow file operations.

Currently, Concordia relies on queries to get the current date, time or datetime. Examples:

- value comes from "SELECT CURRENT_DATE()"

- value comes from "SELECT CURRENT_TIME()"

- value comes from "SELECT NOW()"

Formatted values are also accepted:

# dd/mm/yyyy
- value comes from 'SELECT DATE_FORMAT( CURRENT_DATE(), "%d/%m/%Y")"

# hh:mm
- value comes from 'SELECT TIME_FORMAT( CURRENT_TIME(), "%h:%i")'

# dd/mm/yyyy hh:mm:ss
- value comes from 'SELECT DATE_FORMAT( NOW(), "%d/%m/%Y %h:%i:%s")'

Instead, it should support a declaration that produces the value without having to deal with queries.

Proposal

1. Language constructions for date and time

The proposed constructions should consider the property data type.
For example, now for a date must return the current date, while now for a time must return the current time.

Date constructions

• now
• current date
• today
• yesterday
• tomorrow
• next + ( week | month | year )
  Example: next week
• last + ( week | month | year )
  Example: last month
• number + ( day | days | week | weeks | year | years ) + ( ago | ahead | later )
  Example: 2 days ago

Time constructions

• now
• current time
• current + ( second | minute | hour )
• number + ( second | seconds | minute | minutes | hour | hours ) + ( ago | ahead | later )
  Example: 1 second ago

Datetime constructions

• current datetime
• all the date and time constructions

2. Allow constructions for integer values

Example:

UI Element: Sales Last Year
  - data type is integer
  - max value is the current year # current year value, like 2019

UI Element: Meeting Day
  - data type is integer
  - max value is today  # current day value, like 25

Some Examples

UI Element: Meeting Date
  - data type is date
  - min value is today

UI Element: Sales Last Year
  - data type is integer
  - max value is this year # current year value

UI Element: Birth Date
  - data type is date
  - max value is 18 years ago # the current date, but for 18 years ago
    Otherwise I see "You must be at least 18 years old."

Future work

👉 New Issues may address them.

1. Built-in functions date(), time(), and datetime()

• Both can receive an optional parameter format. Its value should default to the UI Element locale's date/time/datetime format, if available in the language (see #30). Otherwise, the locale defined by the document's language.

• The parameter format should use the PHP format for date and time values.

• By default, date() should return day, month and year. Example for the English locale: 12/31/2018.

• By default, time() should return hour and minute (without second). Example for the Portuguese locale: 13:00.

• By default, datetime() should return day, month, year, hour, minute and second. Example for the English locale: 12/31/2018 10:20:30.

2. Allow to use the proposed functions inside declared tables.

3. Allow time zones (as an additional optional parameter for the functions?).

When Variants declare input steps with UI Elements and fixed values, they have being used to combine test cases - producing more test cases than needed.

Some valid imports are being reported in the console as invalid imports. Although, everything is working as expected.

To add a parameter --init that guides the user to configure some parameters of the Concordia Compiler, such as:

• directory with features (default to ./features)
• language
• plugin (default option for CodeceptJS + WebDriverIO)
• test scripts folder (default to test)
• test output folder (default to output)

The configuration must be stored in the current configuration file, .concordiarc.

Plugin for CodeceptJS is currently ignoring codecept.json. Whether the file exist, it must be considered. Otherwise, it must be generated.

A sentence like

  When I wait for "/foo" during 3 seconds

is currently not mapped.

When a test executes, it produces an output file, e.g., output.json, junit.xml. Every plug-in should now return the name of the file produced by default, in order to allow the parameter --just-result reading it.

Auto-detect the language of the current document, when #language is not declared.

Visual testing helps to verify if:

• the user interface is appearing correctly for your users
• elements of the user interface appear in the correct position and with the correct size
• elements change appropriately

Currently a lof of tools do visual regression testing. The same applies to CodeceptJS, which now integrates with Resemble.js. So, it's an interesting improvement for the Concordia language to support visual testing comparison.

Here it is a proposal for such support:

Language

Currently, the following sentence is already supported:

I save a screenshot "MyFeature.png"

This sentence makes a testing framework to take a picture of the screen.

So the new feature would be to introduce a sentence like

I see a visual difference considering "MyFeature.png"

Let's see some examples.

Full screen comparison

   I save a screenshot "MyFeature.png"
   and I see a visual difference considering "MyFeature.png"

or

   I save a screenshot "MyFeature.png"
   and I see a visual difference of 20% considering "MyFeature.png"

When no percentage is given, it is assumed 10%.

Element comparison

   I save a screenshot "MyFeature.png"
   and I see a visual difference for the element <#someElement> considering "MyFeature.png"

or

   I save a screenshot "MyFeature.png"
   and I see a visual difference of 20% for the element <#someElement> on "MyFeature.png"

When no percentage is given, it is assumed 10%.

Behavior

A plug-in must compare the visual difference between a previously-save image with the current screen or part of the screen. This difference must consider some tolerance value (e.g., misMatchPercentage in Resemble.js).

Plug-ins

The plugin for CodeceptJS with WebDriverIO can support this feature using the Resemble Helper. The is no support yet for CodeceptJS with Appium.

Sometimes a report generated with CodeceptJS and Mocha Multi Report lacks some properties, such as stats. The plugin for CodeceptJS must be able to handle this problem appropriately.

Parameter --language in the CLI is being ignored.

Currently, when we add the tag @ignore to a Variant (for instance), its Test Cases are not generated and the corresponding Test Cases are pruned from the .testcase file. However, the corresponding test script files (ex. .js files) are kept untouched.

Thus, since it's annoying to have to delete these test scripts before running the tests, I'm proposing to change the behavior of @ignore to propagate the change to test script files.

Thanks to Rebeca Costa for suggesting it 👍.

When a Test Case has the tag @fail, it indicates that the test script will fail, but the failure is expected as a normal behavior. In this case, any plugin must count the FAILURE as a PASS. The current plugin for CodeceptJS must implement that behavior.

Ao utilizar o campo informo para preenchimento de caixas de texto, este não está sendo executado quando o primeiro caractere a ser preenchido é um numeral. O comando é ignorado.

Configuration file .concordiarc, which is produced when the parameter --save-config is used, is not being saved in the execution path, but in the path of the Concordia "executable".

MacacaJS supports a wide range of platforms including native apps, web apps, and desktop apps. It has implementations for NodeJS, Java and Python.

Example:

const wd = require('macaca-wd');

describe('my site test', function() {

  var driver = wd.promiseChainRemote( { host: 'localhost', port: 3456 } );
  driver.configureHttp( { timeout: 600 * 1000 } );

  it('has title', function() {
      return driver
        .get('https://www.mysite.com')
        .sleep( 2 * 1000 )
        .title()
        .then( title => {
          title.eql( 'My Site' )
        });
  } );

  it('is searchable', function() {
      return driver
        .get('https://www.mysite.com')
        .waitForElementById('search')
        .sendKeys('Soda')
        .elementById('seachButton')
        .click()
        .waitForElementById('searchResults')
        .source()
        .then(html => {
          html.should.containEql('Soda');
        });
  } );

} );

Currently, when a constraint of a UI Element does not specify Otherwise steps, a Test Case that explores that constraint (with an invalid input value) receives a tag @fail, since the Test Case should not pass the tests. After executing the corresponding test scripts, Concordia evaluates expected results and considers expected failures as successful results - labelled as adjusted instead of passed in test reports:

An alternative would be using NLP to negate postconditions (Then steps) and, therefore, to invert their expectations. In this case, the tag @fail would be no longer needed and test reports would not needed to analyze results before presenting them.

Example (today):

   ...
   Variant: To add an item updates the total
   ...
     When I fill {Product Code}
        and fill {Quantity}
        and I click on {Add Item}
     Then I do not see {Total} with "0,00"
   ...
   UI Element: Quantity
    - data type is integer
    - minimum value is 1

Concordia will produce a test case in which Quantity will receive the value 0, in order to explore the constraint minimum value is 1. Thus, the postcondition Then I do not see {Total} with "0,00" should fail, since the input value is considered incorrect. However, Quantity does not declare an Otherwise sentence for the corresponding constraint, aiming to describe the expected behavior. Concordia will then simply add a flag @fail to the Test Case to indicate the expectation of failure.

Proposed

Negating the original post-conditions may give the same effect, without having to analyze test cases' expectations:

   Then I do not see {Total} with "0,00"

would become

   Then I see {Total} with "0,00"

That is, when using an invalid/icorrect input value for Quantity - such as 0 -, the feature should not be able to produce the original post-conditions. So negate them is a way to assert that.

Olá, gostaríamos de saber se tem como colocar um tempo de espera entre passos ou esperar até que um elemento apareça na tela.

They stopped working after updating meow

👉 Please read Issue #44

Example

UI Element: Name
  - minimum length is 2
    Otherwise I see "Please inform at least {Name|minimum length} characters."

The value in the Otherwise sentence should be transformed into "Please inform at least 2 characters.".

Properties

• id
• type
• datatype
• value 🠖 Implemented in v1.3.0
• minlength
• maxlength
• minvalue
• maxvalue
• format
• required

Executing commands in the console/terminal can be useful during Test Events.

Example:

Before Feature:
  When I run the command 'mkdir foo'
    and I run the command './script.sh'

Generated lines of code are referencing a same line of a test case. It is probable that more than one Test Case are sharing a same line in memory.

e.g., some file.feature is generating some%20file.testcase and some%20file.js

A masked input field usually does not accept invalid input data. Usually such components are well tested and we can assume that they work as intended. Therefore, Concordia does not need to generate invalid input data for them. To disable that generation, a new tag could be defined to be used in UI Element's properties. For example:

UI Element: Year
  - data type is integer
  @generate-only-valid-values
  - format is "/^[0-9]{1-3}$/"
    Otherwise I must see "Year must be a number."

In the above example, a tag (@generate-only-valid-values) is used to denote that the property that follows it (format) should never be used to generate invalid input values.

Candidate names for the tag:

1. @generate-only-valid-values
2. @generate-no-invalid-values
3. @valid-values-only
4. @no-invalid-values

Any suggestions?

👉 Please read Issue #35 before continuing.

Example

Then I see "Welcome, {Username|value}"

Whether {Username|value} is bob, the corresponding Test Case must be transformed into

Then I see "Welcome, bob"

Properties

• id
• type
• datatype
• value 🠖 Implemented in v1.3.0
• minlength
• maxlength
• minvalue
• maxvalue
• format
• required

See also

• Recognize UI Element Properties inside values of Otherwise sentences

Example 1:

Then I see "P" in the element <foo>

Example 2:

Then I see "P" in <foo>

Example 3:

Then I see "P" inside <foo>

Example 4:

Then I see <foo> with "P"

Currently we can define values for ui literals and ui elements like this:

When I fill <foo> with "a"
  and I fill {bar} with "b"

The following syntax can be also useful:

When I fill <foo> and {bar} with "x"

to make both <foo> and {bar} to receive the value x

Bom dia Thiago, segui os passos para instalar a dependência mochawesome, porém a opção timestamp não está salvando novos relatórios para cada geração de teste e a opção autoOpen não abre uma o relatório no navegador.Estou usando o Google Chrome como navegador padrão.

Example:

      Given that I am in the [Lista de tarefas]
      When I fill <#inputtask2> with "test"
        And I click on <#btnAdicionar>
      Then I see "test"
      Then I click on <#btnEdit>
      Then I see <#btnCancelarEdicao>
        And I see <#inputtask2> with "test"

Console output:

 expected element #inputtask2 to include "test"
      + expected - actual

      +test


  Scenario Steps:

  - I.see("test", "#inputtask2") at Test.Scenario (src\__tests__\e2e\tests\task.js:32:7)
  - I.seeElement("#btnCancelarEdicao") at Test.Scenario (src\__tests__\e2e\tests\task.js:31:7)
  - I.click("#btnEdit") at Test.Scenario (src\__tests__\e2e\tests\task.js:30:7)
  - I.see("test") at Test.Scenario (src\__tests__\e2e\tests\task.js:29:7)
  - I.click("#btnAdicionar") at Test.Scenario (src\__tests__\e2e\tests\task.js:28:7)
  - I.fillField("#inputtask2", "test") at Test.Scenario (src\__tests__\e2e\tests\task.js:27:7)
  - I.amOnPage("http://localhost:4200/") at Test.Scenario (src\__tests__\e2e\tests\task.js:26:7)

Concordia Compiler notifies when a new version is available (since May 9th, 2018, see #1).
It would be useful for users to display an additional information when an update has a breaking change. A breaking change occurs when the MAJOR version changes. For example, an update from version 0.99.2 to version 1.0.0 is a breaking change, since the major version upgraded (from 0 to 1).

Proposal
Display a message asking the user to read the release notes before updating. The message should include the URL of the Release Notes page. For example:

PLEASE READ THE RELEASE NOTES BEFORE UPDATING
https://github.com/thiagodp/concordialang/releases

Introduction

This Issue is centralizing the desire to have new plug-ins for different testing frameworks or programming languages.

Issues #28 , #32, and #33 described plugins for MacacaJS, Cypress, and TestCafe. These are desirable platforms for JavaScript.

However, if you would like to contribute with a plug-in for any other programming language (say PHP, Python, Java, Ruby, Delphi, C#, Go, whatever), that would be awesome!

Information

A plug-in must be coded in JavaScript or any language that "transpiles" to JavaScript, such as TypeScript (recommended), CoffeeScript, and Dart.

A plugin can generate source code for any programming language or testing framework able to perform functional tests.

Suggestions

JavaScript and friends

• TestCafe - web, mobile, desktop
• Cypress - web
• MacacaJS - web, mobile, desktop

PHP

• Codeception - web

Ruby

• Capybara - web
• Calabash - mobile applications developed with Xamarim

Java

• Selenium - web applications
• AssertJ Swing - swing desktop applications
• Robotium - android applications

Python

• Selenium - web applications

C++

• QtTest - QT
• DUnit - C++ RAD Studio

Delphi

• DUnitX - FireMonkey GUI
• DUnit

Other suggestions? Please comment.

Example:

Then I see that {Age} is {Age|minvalue}

The | operator separates a UI Element from one of its properties.

Whether the property id is #age and the property minvalue is 21, the above sentence will be converted to:

Then I see that <#age> is 21

Language-independent property names

Allow the following properties:

• id
• type
• datatype
• value 🠖 Implemented in v1.2.0
• minlength
• maxlength
• minvalue
• maxvalue
• format
• required

Example:

Then I see {Age|value}

Language-specific property names

Allow properties to be written in the document's language. Examples:

Then I see {Age|minimum value} # english

Then I see {Age|valor mínimo} # portuguese

Support properties from other Features

Allow to reference UI Elements from other Features involved in a Variant. Example:

# login.feature
Feature: Login
  Scenario: Successful login
    Variant: Login by entering credentials
      Given that I am on the [Login Screen]
      When I type {Username}
         and I type {Password}
         and I click on {OK}
      Then I have ~user logged in~
...

# logout.feature
import "login.feature"
Feature: Logout
  Scenario: Successful logout
    Variant: Logout by selecting the menu bar option
      Given that I have ~user logged in~
        and I see {Login:Username|value}
      When I click on {Logout}
      Then I have ~user logged out~
...

When the sentence and I see {Login:Username|value} is converted to a Test Case sentence, it will receive the corresponding value of Login's Username.

Future plans

• Recognize UI Element Properties in Otherwise sentences

Currently, plug-ins are part of the tool.

PROS :

• easier for newcomers, i.e., users don't have to worry about how to deal with npm installs

CONS :

• plug-ins can't evolve freely - e.g., new versions of used tools and frameworks
• users need to update Concordia to get newer plug-in versions
• version messy - tool vs. plug-ins vs. frameworks and tools

Thus, to separate them is a clear need.

SOLUTION DETAILS

• Plug-ins could be installed with npm
• Both local and global installation could be accepted. Local has priority over global.
• Plug-in data could be read from a custom property of their package.json files.

COMPATIBILITY BREAK :

• Users who already use Concordia will have to:
  • install their plug-ins
  • change the plugin name in the configuration file, .concordiarc, if used (e.g., codeceptjs → codeceptjs-webdriverio)
• Command --plugin-list will list installed plug-ins, instead of available ones.

SOME OPTIONS BEING CONSIDERED :

• Command --plugin-install <plugin-name> could execute npm install --save-dev <plugin-name> (where <plugin-name> can be prefixed with concordialang- if needed) and then run plugin's install commands.
• Command --plugin-uninstall <plugin-name> could execute npm uninstall --save-dev <plugin-name> (where <plugin-name> can be prefixed with concordialang- if needed) and then run plugin's uninstall commands.
• Internal plugin properties install and uninstall may be removed and needed applications installed via package.json scripts like postinstall and postuninstall. Example: "postinstall": "((npm list -g appium) || (npm i -g appium)) && ((npm list -g codeceptjs) || (npm i -g codeceptjs))".
• Internal plugin properties install and uninstall may be removed and needed applications installed as plugin's dependencies and executed with NPX. Concordia is currently compatible with Node 8 or above. NodeJS 8.2.0 comes with NPX. The postinstall script of package.json could install NPX whether not installed, i.e., (npm list -g npx) || (npm i -g npx). By using NPX, global dependencies will not be needed anymore. However, users shall need to use it to execute any third-party command-line applications adopted by a plug-in, e.g., npx selenium-standalone start, npx codeceptjs run, npx appium. Note: A plug-in is installed by the compiler as a development dependency so that the target application's dependencies are not affected. Every plug-in may declare external applications as its dependencies. All of them will be removed automatically when a plug-in is uninstalled.
• Internal plugin properties install and uninstall may be removed and the plugin perform the expected behavior programmatically, instead of using external applications.
• Command --init may try to install the chosen plugin.

TestCafe supports desktop, mobile, remote and cloud browsers (UI or headless).

Example:

import { Selector } from 'testcafe'; // first import testcafe selectors

fixture `Getting Started`// declare the fixture
    .page `https://devexpress.github.io/testcafe/example`;  // specify the start page


//then create a test and place your code there
test('My first test', async t => {
    await t
        .typeText('#developer-name', 'John Smith')
        .click('#submit-button')

        // Use the assertion to check if the actual header text is equal to the expected one
        .expect(Selector('#article-header').innerText).eql('Thank you, John Smith!');
});

Proposal:

1. New property locale

The property locale would assume the document's language by default.

Usage example:

UI Element: Birth Date
  - data type is date
  - locale is "pt"

Every supported language should define the default formats for:

• date
• time
• datetime

2. New function dtf

NOTE: I'm still not satisfied about using a function. Maybe special characters to denote a different kind of string could fit better.

The function means "date time format".

Currently there is the property format, which defines a UI Element's format using a regular expression. The proposed function aims to facilitate that definition by adopting a simpler format, like the PHP date and time format.

Examples:

UI Element: Birth Date
  - data type is date
  - format is dtf( "d/m/Y" )

UI Element: Meeting Hour
  - data type is time
  - format is dtf( "h:i" )

UI Element: Last Update
  - data type is datetime
  - format is dtf( "d/m/Y h:i:s" )

With the proposal, format should assume the locale's format by default. Example:

UI Element: Birth Date
  - data type is date
# format is dtf( `m/d/Y` ) since the language is English and so is the locale.

Create a plug-in for CodeceptJS with Appium.

Cypress is a promising opensource testing framework for Chrome and cross browser support is in the roadmap.

Example:

describe('My First Test', function() {
  it('Gets, types and asserts', function() {
    cy.visit('https://example.cypress.io')

    cy.contains('type').click()

    // Should be on a new URL which includes '/commands/actions'
    cy.url().should('include', '/commands/actions')

    // Get an input, type into it and verify that the value has been updated
    cy.get('.action-email')
      .type('[email protected]')
      .should('have.value', '[email protected]')
  })
})

Currently, the term field (or similar) should be added to sentences, in order to make Concordia aware that it is an input field - see #37.

For example,

Given that I see the field <#name> with "Bob"

will generate

I.seeInField('#name', 'Bob');

For a sentence that refers to a UI Element that has an input-like type, it would be nice if Concordia could complete the sentence with field automatically. For example:

Variant: ...
   ...
   Given that I see {Name} with "Bob"  # sentence without `field`
...
UI Element: Name
  - id is '#name'
# - type is textbox

It could produce:

Test Case: ...
   ...
   Given that I see the field <#name> with "Bob"  # sentence with `field`

Currently, UI literals do not support escaped CSS selectors. For example:

Then I see <div \> a>

to locate div > a.

These cases are currently supported as ids of UI Elements. For example:

Then I see {Foo}

UI Element: Foo
  - id is "div > a"

Natural Language Processing is removing diacritics because of the Bravey library. Changes on this library may be necessary as a workaround.

When more than one precondition state is declared, the first one is combined correctly but the other ones are not. They are being inserted in the old step index (the index before combination).