integrity-tf / integrity Goto Github PK

Something like this does not work currently:

constant transactionConst {
        subject: "Lottery Win"
        value: 1000.00
    }, { 
        subject : "Cash Withdrawal" 
        value : -220.00 
    }

This is due to the fact that constants and variables can only be pre-allocated with single values, not multiple values, as supported in pretty much all other places where fixed values are used.

It would be beneficial to extend the constant/variable feature to support multiple values as well.

It would be nice when the Integrity Plugin would fully support the outline feature of eclipse. This is especially very useful for big test-suites.

Currently the plugin supports only outlines for packagedef, testdef and suitedef. Outlines of tests are not supported (like test and tabletest), see this screenshot:

Even though the exact same test script is run, different forks may have different colors in the Test Runner view. It would be nicer if they had the same color.

The documentation from Tutorial3 mentions that:

These dependencies are executed before the actual suite that depends on those is run, but only if the dependent suites haven't already been run in the current "execution stack trace". This sounds complicated, but is actually a very simple system.

Dependency management in general is actual far from beeing very simple.
The current implementation doesn't explain what the current execution stack trace is, but from what I've learned it seems to be the current test execution, making the statement an the dependency is run if it is not run in this test execution before.
It would be analogous to an @BeforeClass in JUnit on the top test suite.

However the concludedby keyword is tied to a different scope than the duplication mechanism, which prevents the dependency from beeing triggered more than once in it's scope.

Imagine a suite login which is concluded by it's equivalent logout. Given the following scenario:

Suite Root
  +- Suite A  (requires login)
  +- Suite B  (requires login)

will result in the following execution:

1. Suite Root
2. Suite login as a dependency of Suite A
3. Suite A
4. Suite logout because login is a dependency of A and is concluded by logout - this is the place where the concluded by triggers
5. Suite B (note that Suite login is not triggered as it is already run in the current execution stack trace)

The duplication detection of the requires keyword is test execution scope where as the concludedby keyword scope is Suite scope.
I would assume those two to be a pair (like @Before and @After) and would at least expect them to have an equivalent scope.
However this seems to be not the case, the duplication detection of requires is somewhat equivalent of an @BeforeClass and the concludedby keyword is equivalent to @After, which is an odd pair.

A fixture that is defined to have two parameters "one" and two" is usually called this way:

call testFixture one: 1 two: 2

but this is also allowed:

call testFixture one: 1 two: 2 one: 2

the fixture will be called with param "one" = 2.

If one uses a suite which is not part of any file passed to the console runner, the following exception is shown.

Exception in thread "main" org.jdom.IllegalDataException: The data "null" is not legal for a JDOM attribute: A null is not a legal XML value.

Because it can't get the name of the suite which is referenced (in this case the suite which was specified as required could not be resolved).
This is not a very helpful error message, please provide a better one, which shows the error cause and the information which is available (which should be at least the name of the referenced suite as given in the suite which is referencing it).

Title is pretty self-explanatory; feature is commonly known with Java files.

• Local variables (in scope) shall always be displayed in front of global variables
• Global variables in the current package shall always be displayed in front of global variables from other packages

rejected

The XSLT transform after running all tests can take some time when dealing with big test suites. During that time, the test control view shows "running tests" as info text. There should be a special text for this phase, to let the user know that the client has finished test execution and is now post-processing result data.

Integrity cannot map between a variable that is defined using an operation and the target type of String (and maybe others).

The parameter param1 will get result the value: (de.gebit.integrity.dsl.impl.StandardOperationImpl@1b5c3de (operators: [+]))
instead of 13 as expected

variable bla initially 12
variable blubb initially (bla + 1)

call simpleParamFixture obj: {
param1: blubb
}

where simpleParameterFixture is a Fixture that gets an Object composed of a String parameter named "param1".

Loaded test resource 'C:\ew\integrity_test\SuiteOrganization\Integrity\suite.integrity': 0 errors.
Loaded test resource 'C:\ew\integrity_test\SuiteOrganization\Integrity\fixture.integrity': 0 errors.
Resolving the test model...done!
Test execution has begun...
Now entering suite 2: suiteorganization.rootSuite
Now entering suite 3: suiteorganization.otherSuite
Defined variable result with initial value: 8
Now executing call 1: Multiply 12.5 with 8...
SUCCESS!
Now running test 1: Divide 100.0 by 8...
SUCCESS!
Now leaving suite 3: suiteorganization.otherSuite
Now leaving suite 3: suiteorganization.rootSuite
Finished executing 3 suites with 1 tests and 1 calls in 79 msecs!
1 tests finished sucessfully, accompanied by 0 failures and 0 exceptions.

Even in the tutorial example, suite numbering seems to be offset, and the "leaving" messages are wrong as well. Total suite count is wrong, too.

call createBO class: "Country"
    +twolettercode: "US"
    +threelettercode: "USA"
    +digitcode: "840"
    +name: "United States of America"
    +phonecode: "049" -> usa

call createBO class: "Province"
    +code: "AL"
    +name: "Alabama"
    +country: usa

Assuming "createBO" is a fixture which creates an instance of a simple bean class, the code above results in an undesired autoconversion of the object inside the variable "usa" to a map. In this situation, this conversion is counterproductive and should not be done.

Is is quite exhausting to compare expected maps with result maps in order to find out which values does not match. This is especially true if many attributes are involved.

DefaultResultComparator.performEqualityCheckForMaps() knows which value differs. It should pass this information to the reporting layer and the reporting layer should visualize this information either by color, detail info or both.

I learned from #50 that there is some kind of operation possible on variables of the kind. That leaves me more puzzled than it was enlightening to me as I could not find documentation for this here: http://www.integrity-tf.org/docs/tutorials/tutorial2.

That leaves me with the following questions:

• What kind of types with which restrictions does Integrity know?
• What kind of operators (read: methods) are possible for each of them?
• How can one define custom operator for a type? How are they reflected on the JVM or how are they passed to Fixtures on the JVM (Java).
• Is there any kind of relationship between those types (e.g. inheritance)?
• Are custom operations scoped or are they globally applied to the types?
• What are the exact syntactical conditions to use them. What is an literal with an operator applied to it in contrast to say a plain literal. Is one.toLowercase actually "one".toLowercase() or "one.toLowercase"?

Possibly more things one should know about operators, but I never dared to ask?
My very first impression: Feels like pandora's boxed which just opened...

Eclipse plugin does not support:
operations (12 + 1)
or custom operations ["Helo" tLowercase]

Something like this creates an exception during conversion:

call arbitraryCall +testparam1: 1, 2, 3, 4

When primitive types are used as types in fixture methods, IllegalArgumentExceptions can be thrown if there is no value provided in the test script for that parameter.

Sensible defaults (0 in case of numbers) should be delivered instead.

If Unix-style line endings are used on Windows, or Windows-style endings on Unix, the formatter cannot correctly format line indentations currently. Ideally this should work even with the wrong line endings, since the Eclipse editor generally supports any line ending style on any system by auto-detecting the used line endings and adapting file display.

When using primitive types for a parameter in a fixture signature the call to it may fail, if the user does not give any parameter for it.
The runner tries to call the fixture with an Object[1] = { null } array, which leads to a stacktrace like:

java.lang.IllegalArgumentException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at de.gebit.integrity.fixtures.FixtureWrapper.execute(FixtureWrapper.java:262)
...[snip]...

Please:

• Improve the error log to give the user a more useful help message stating that a mandatory argument was missing
• Forbid the combination of java primitive type and @FixtureParameter(optional = true), which is not necessary to reproduce this, but is an indication that something is wrong here.
• Even better: Let the editor check if mandatory arguments are missing and raise a validation error.

This name override attribute doesn't really seem to work correctly, and it has never really been used anywhere and is generally a bit too much abstraction. It shall be removed.

packagedef testpack with
suitedef suite1 with
suiteend
suitedef suite2 requires suite1 with
suiteend
suitedef suite3 requires suite2 with
suiteend
packageend

should result in execution order
suite1
suite2
suite3

but suite1 is completly ignored

Dots are used to split fully or partly qualified identifiers in package name (parts) and identifier name. Thus it is illegal to declare a variable which has a dot in its identifier. However, the validator currently doesn't seem to see such a construct as an error (even though it's not usable correctly later).

That handling should be improved.

If a construct like this is created...

variable aVariable
call echoMap map: {
    key: (aVariable + 2)
} 

...a NullPointerException is thrown during dry run phase. This does not interfere with test execution, but is of course irritating.

There are conversions converting various kinds of dates to SQLDate, but none for SQLTime.

I defined a call simpleParamFixture that receives 2 objects. One object is defined as java.lang.Number. Afterwards I call the that fixture with a numerical String value that exceeds the maximum integer e.g. "2147483648". Integrity will try to convert the String into a Number using the StringToInteger conversion which obviously fails. Integrty than returns with an error.
However Integrity should use the StringToLong conversion in this case which would not fail.

suitedef number_test with

    variable bla
    call hello.echo value: "2147483648" -> bla
    call hello.echo value: bla

    call simpleParamFixture obj: {
            param1: bla
            param2: bla
        }

suiteend

In case of a fixture like this...

@FixtureMethod
public Object echo(@FixtureParameter(name="value") Object aValue) {
    System.out.println("echo got value: " + aValue);
    return aValue;
}

...the Integrity-internal ...Impl classes wrapping script-provided values are delivered with no conversion. Thus a test fixture based on the method above fails since for example IntegerImpl is not meant to be compared to another IntegerImpl. Generally it should probably be best to apply default conversions to values going in a fixture method like the above one (specifying just Object as target type), because the fixture author shouldn't have to deal with Integritys' internal types.

At least the ISO time format (possibly others as well) should be able to deal with millisecond precision. Currently the grammar doesn't even allow to enter them.

Possibly, for conversion in a java.sql.Timestamp, even nanoseconds might be useful.

Currently the resulting html page can expand and collapse individual entries. 📜
However what get's annoying is that if one wants to copy data from an entry the entry collapses on the first click.

What would be better is when there would be an explicit collapse button which collapses the entry instead of "anywhere on the panel" as it is now.

...even though it should, since it conforms to the ISO time format

This does seem to depend on a certain test structure, as it does not happen in any case, but is reproducible if you've got the problem once.

Please provide some kind of hook for failed tests.

Background is that we are using Integrity with Selenium Fixtures and we would love to have the chance to include additional information for failed tests to the output.

What we want to be able to do is to include a plain page dump 📃 and a screenshot 📷 to each failed test, so we can either see or reproduce a failed test more easily.
(The plain page dump allows us to rerun a given selector on the plain html output)

I could think of more platforms (e.g. Swing) where screenshots would be helpful or other trigger (e.g. screenshots after each command, for debugging purposes) - but for a first, a trigger for failed tests would be the most helpful one.

What could be a kind of problem is that one most likely needs some kind of context from the fixture (the Selenium instance in our case) which collects the actual data and pipes this to the test output, where it should be included somehow.

Example, two suite calls:

suite aSuite optionalParam: 123
suite aSuite

The latter call will receive the parameter "optionalParam" set to "123", even though that's clearly not intended.

One should be able to request a java.sql.Date instead of a java.util.Date and get dates converted into that object type. However, at the same time the conversion to java.util.Date should still be used if just a java.util.Date is requested.

If a fixture which has a description with conditional parts in it is called inside a suite, which itself offers some parameters and forwards an optional one of those to the fixture call, the conditional text for the "parameter is present" situation is used, even though the parameter is actually not present in the call to the suite.