fabric8io / fabric8-test Goto Github PK

View Code? Open in Web Editor NEW

4.0 23.0 28.0 2.22 MB

License: Apache License 2.0

Ruby 1.25% Shell 23.38% Java 1.05% Python 43.06% TypeScript 26.24% Gherkin 0.60% Dockerfile 0.23% Go 4.19%

fabric8-test's Introduction

Fabric8-Test

Functional tests for OSiO (OpenShift.io) platform.

End to end test on OSiO

See: https://github.com/fabric8io/fabric8-test/blob/master/ee_tests/README.md

API tests on OSiO

Our API tests use pyresttest a Python test framework that makes all your API tests declarative. No need to learn Python, your tests are plain old yaml file!

Pre-requisite

Install Python pip

sudo python get-pip.py

Install pyresttest (>=1.7.1)

git clone https://github.com/svanoort/pyresttest.git
cd pyresttest
sudo python setup.py install

Install pyresttest dependencies

cd EE_API_automation/pyresttest/setup
pip install -U -r requirements.txt

Run API test against PROD

WIT API

To run create space script:

cd EE_API_automation/pyresttest/
pyresttest https://api.openshift.io get_a_space.yaml --vars="{'token': 'YOUR_OSIO_TOKEN', 'userid': 'YOUR_OSIO_ACCOUNT', 'space_name_var': 'spacename'}"

where YOUR_OSIO_ACCOUNT is your OSiO account and YOUR_OSIO_TOKEN can be either taken form your browser devtools searching localStorage for auth_token key or going to your profile page -> update profile -> Advanced -> Personal Access Token copy button.

Forge API

The forge API target Forge REST backend using Forge addon logic. As a pre-requisites, Forge API still need to create a space in WIT API. The is 2 main flow tested:

To run import wizard flow:

cd EE_API_automation/pyresttest/
./run_forge_import_api_test.sh YOUR_OSIO_ACOUNT YOUR_OSIO_TOKEN

NOTE: Depending whether your GitHub account belongs to an organisation or not the API response is different. The bash script wraps the difference and runs the relevant tests.

To run import wizard flow:

cd EE_API_automation/pyresttest/
./run_forge_quickstart_api_test.sh YOUR_OSIO_ACOUNT YOUR_OSIO_TOKEN [GITHUB_ACCOUNT] [GITHUB_TOKEN_WITH_DELETE_SCOPE]

NOTE: [GITHUB_ACCOUNT] and [GITHUB_TOKEN_WITH_DELETE_SCOPE] are optionals. If not provided your GitHub account will keep the newly created repository. This repository is prefixed with a UUID.

fabric8-test's People

Contributors

Stargazers

Watchers

fabric8-test's Issues

End-to-end tests are failing in Jenkins

We had to do some changes related to the new root directory for the end-to-end tests. However, the tests seem to fail now due to other reasons. It seems like there are some bugs in the code, https://ci.centos.org/job/devtools-test-end-to-end/1636/console.

ping @ldimaggi

Pipelines page = testaug41501850501980 created a few seconds ago
Source Repository: https://github.com/osiotestmachine/testaug41501850501980.git
No pipeline builds have run for testaug41501850501980.
Verify that only one pipeline is created - https://github.com/fabric8-ui/fabric8-ui/issues/1707
�[31mF�[0mA Jasmine spec timed out. Resetting the WebDriver Control Flow.


Failures:
1) openshift.io End-to-End POC test - Scenario - Existing user:  should perform Burr's demo
  Message:
�[31m    Expected false to be true.�[0m
  Stack:
    Error: Failed expectation
        at UserContext.<anonymous> (/home/fabric8/fabric8-ui/src/tests/work-item/work-item-list/EE/openshift-io-burr.spec.js:198:90)
        at /home/fabric8/fabric8-ui/node_modules/jasminewd2/index.js:112:25
        at new ManagedPromise (/home/fabric8/fabric8-ui/node_modules/selenium-webdriver/lib/promise.js:1067:7)
        at ControlFlow.promise (/home/fabric8/fabric8-ui/node_modules/selenium-webdriver/lib/promise.js:2396:12)
        at schedulerExecute (/home/fabric8/fabric8-ui/node_modules/jasminewd2/index.js:95:18)
        at TaskQueue.execute_ (/home/fabric8/fabric8-ui/node_modules/selenium-webdriver/lib/promise.js:2970:14)
        at TaskQueue.executeNext_ (/home/fabric8/fabric8-ui/node_modules/selenium-webdriver/lib/promise.js:2953:27)
        at asyncRun (/home/fabric8/fabric8-ui/node_modules/selenium-webdriver/lib/promise.js:2860:25)
        at /home/fabric8/fabric8-ui/node_modules/selenium-webdriver/lib/promise.js:676:7
  Message:
�[31m    Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.�[0m
  Stack:
    Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
        at tryOnTimeout (timers.js:232:11)
        at Timer.listOnTimeout (timers.js:202:5)
  Message:
�[31m    Failed: Session ID is null. Using WebDriver after calling quit()?
    Build info: version: '3.4.0', revision: 'unknown', time: 'unknown'
    System info: host: '04e692bc0cb8', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-514.26.2.el7.x86_64', java.version: '1.8.0_141'
    Driver info: driver.version: EventFiringWebDriver�[0m
  Stack:
    WebDriverError: Session ID is null. Using WebDriver after calling quit()?
    Build info: version: '3.4.0', revision: 'unknown', time: 'unknown'
    System info: host: '04e692bc0cb8', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-514.26.2.el7.x86_64', java.version: '1.8.0_141'
    Driver info: driver.version: EventFiringWebDriver
        at WebDriverError (/home/fabric8/fabric8-ui/node_modules/selenium-webdriver/lib/error.js:27:5)
        at Object.checkLegacyResponse (/home/fabric8/fabric8-ui/node_modules/selenium-webdriver/lib/error.js:505:15)
        at parseHttpResponse (/home/fabric8/fabric8-ui/node_modules/selenium-webdriver/lib/http.js:509:13)
        at doSend.then.response (/home/fabric8/fabric8-ui/node_modules/selenium-webdriver/lib/http.js:440:13)
        at process._tickCallback (internal/process/next_tick.js:103:7)
    From: Task: WebDriver.findElements(By(xpath, .//a[contains(@class,'card-title') and contains(text(),'testaug41501850501980')]/../../..//a[contains(text(),'Input Required')]))

cheTest fails to list projects (may be related to @ char in username)

@ibuziuk could not run cheTest because it fail to list projects in osio. The only difference we could spot between my accound and his is that @ibuziuk osio's login is his email: [email protected]. He can't login using username ibuziuk, he needs to provide his full email.

Add readme

Add a readme that describes the different types of tests. Including links and description to other tests like: https://github.com/fabric8-analytics/fabric8-analytics-common/wiki/QA-and-testing

Writes tests for License information

See Experience 1704E469 [P0] – License information Testcases

Configure EE tests to push test results to zabbix - for production and prod-preview

https://ci.centos.org/view/Devtools/job/devtools-test-end-to-end-openshift.io-runTest/
https://ci.centos.org/view/Devtools/job/devtools-test-end-to-end-openshift.io-quickstartTest/
https://ci.centos.org/view/Devtools/job/devtools-test-end-to-end-openshift.io-chequickstartTest/

use typescript for test code?

I'm used to writing TypeScript for UI stuff rather than pure JavaScript and I'm surprised how many compile errors get missed when working on test code ;) Plus we lose lots of nice IDE completion due to lack of known types etc

I wonder if we could switch to TypeScript for the test code but still keep the rapid develop / debug cycle of working on the E2E Tests?

Implement mechanism to verify thru the API actions taken in the UI by the EE tests

on test failure lets run a number of oc commands and capture their output so we can more easily diagnose why things go wrong

e.g.

oc export bc -n ${ID}
oc get dc -n ${ID}-stage
oc get pod -n ${ID}-stage
oc get dc -n ${ID}-run
oc get pod -n ${ID}-run

we can tweak them over time. It'd be nice to know what the current space name is when the test failed too.

Note the above are useful for quickstart/import failures; they're less useful for diagnosing che failures

remove all uses of "./oc" from the E2E tests

as they break when running inside docker images in kubernetes.

There should be no need to prefix a binary with ./ - you could always add . to the $PATH.

@ldimaggi could you please make this change so that the ci.centos E2E tests still work ok?

Question: How to see only the summary of test results?

When I run the API tests I see the responses from the individual tests. Why this might be useful for debugging it is a lot of noise if you just want to see the test results.

How can I run the tests to only see the results?

Example:

$ jasmine-node api/1704E468-cve-scoring.spec.js --config targetURL https://recommender.api.openshift.io/api/v1/

......{ result: 
   { data: 
      [ { package: 
           { ecosystem: [ �[32m'maven'�[39m ],
             gh_forks: [ �[33m-1�[39m ],
             gh_issues_last_month_closed: [ �[33m-1�[39m ],
             gh_issues_last_month_opened: [ �[33m-1�[39m ],
             gh_issues_last_year_closed: [ �[33m-1�[39m ],
             gh_issues_last_year_opened: [ �[33m-1�[39m ],
             gh_prs_last_month_closed: [ �[33m-1�[39m ],
             gh_prs_last_month_opened: [ �[33m-1�[39m ],
             gh_prs_last_year_closed: [ �[33m-1�[39m ],
             gh_prs_last_year_opened: [ �[33m-1�[39m ],
             gh_stargazers: [ �[33m-1�[39m ],
             last_updated: [ �[33m1496417783.87�[39m ],
             latest_version: [ �[32m'20040616'�[39m ],
             name: [ �[32m'commons-collections:commons-collections'�[39m ],
             package_dependents_count: [ �[33m-1�[39m ],
             package_relative_used: [ �[32m'not used'�[39m ],
             tokens: [ �[32m'collections'�[39m, �[32m'commons'�[39m ],
             vertex_label: [ �[32m'Package'�[39m ] },
          version: 
           { cm_avg_cyclomatic_complexity: [ �[33m-1�[39m ],
             cm_loc: [ �[33m64414�[39m ],
             cm_num_files: [ �[33m286�[39m ],
             cve_ids: [ �[32m'CVE-2015-6420:7.5'�[39m ],
             dependents_count: [ �[33m-1�[39m ],
             description: [ �[32m'types that extend and augment the java collections framework'�[39m ],
             last_updated: [ �[33m1493052149.69�[39m ],
             licenses: [ �[32m'ASL 2.0'�[39m ],
             pecosystem: [ �[32m'maven'�[39m ],
             pname: [ �[32m'commons-collections:commons-collections'�[39m ],
             relative_used: [ �[32m''�[39m ],
             shipped_as_downstream: [ �[33mtrue�[39m ],
             version: [ �[32m'3.2.1'�[39m ],
             vertex_label: [ �[32m'Version'�[39m ] } } ],
     recommendation: { �[32m'component-analyses'�[39m: { cve: [ { cvss: �[33m7.5�[39m, id: �[32m'CVE-2015-6420'�[39m } ] } } },
  schema: 
   { name: �[32m'analyses_graphdb'�[39m,
     url: �[32m'http://recommender.api.openshift.io/api/v1/schemas/api/analyses_graphdb/1-2-0/'�[39m,
     version: �[32m'1-2-0'�[39m } }
.{ result: 
   { data: 
      [ { package: 
           { ecosystem: [ �[32m'maven'�[39m ],
             gh_forks: [ �[33m-1�[39m ],
             gh_issues_last_month_closed: [ �[33m-1�[39m ],
             gh_issues_last_month_opened: [ �[33m-1�[39m ],
             gh_issues_last_year_closed: [ �[33m-1�[39m ],
             gh_issues_last_year_opened: [ �[33m-1�[39m ],
             gh_prs_last_month_closed: [ �[33m-1�[39m ],
             gh_prs_last_month_opened: [ �[33m-1�[39m ],
             gh_prs_last_year_closed: [ �[33m-1�[39m ],
             gh_prs_last_year_opened: [ �[33m-1�[39m ],
             gh_stargazers: [ �[33m-1�[39m ],
             last_updated: [ �[33m1493054564.43�[39m ],
             latest_version: [ �[32m'1.3.2'�[39m ],
             name: [ �[32m'commons-fileupload:commons-fileupload'�[39m ],
             package_dependents_count: [ �[33m-1�[39m ],
             package_relative_used: [ �[32m'not used'�[39m ],
             tokens: [ �[32m'commons'�[39m, �[32m'fileupload'�[39m ],
             vertex_label: [ �[32m'Package'�[39m ] },
          version: 
           { cm_avg_cyclomatic_complexity: [ �[33m-1�[39m ],
             cm_loc: [ �[33m7149�[39m ],
             cm_num_files: [ �[33m39�[39m ],
             cve_ids: [ �[32m'CVE-2016-1000031:7.5'�[39m, �[32m'CVE-2016-3092:7.8'�[39m ],
             dependents_count: [ �[33m-1�[39m ],
             description: [ �[32m'the apache commons fileupload component provides a simple yet flexible means of adding support for multipart    file upload functionality to servlets and web applications'�[39m ],
             last_updated: [ �[33m1492821853.72�[39m ],
             licenses: [ �[32m'ASL 2.0'�[39m ],
             pecosystem: [ �[32m'maven'�[39m ],
             pname: [ �[32m'commons-fileupload:commons-fileupload'�[39m ],
             relative_used: [ �[32m''�[39m ],
             shipped_as_downstream: [ �[33mfalse�[39m ],
             version: [ �[32m'1.3.1'�[39m ],
             vertex_label: [ �[32m'Version'�[39m ] } } ],
     recommendation: 
      { change_to: �[32m'1.3.2'�[39m,
        �[32m'component-analyses'�[39m: 
         { cve: 
            [ { cvss: �[33m7.5�[39m, id: �[32m'CVE-2016-1000031'�[39m },
              { cvss: �[33m7.8�[39m, id: �[32m'CVE-2016-3092'�[39m } ] },
        message: �[32m'CVE/s found for Package - commons-fileupload:commons-fileupload, Version - 1.3.1\nCVE-2016-1000031, CVE-2016-3092 with a max cvss score of - 7.8\n It is recommended to use Version - 1.3.2\n It is recommended to use Version - 1.3.2\n It is recommended to use Version - 1.3.2\n It is recommended to use Version - 1.3.2\n It is recommended to use Version - 1.3.2\n It is recommended to use Version - 1.3.2'�[39m } },
  schema: 
   { name: �[32m'analyses_graphdb'�[39m,
     url: �[32m'http://recommender.api.openshift.io/api/v1/schemas/api/analyses_graphdb/1-2-0/'�[39m,
     version: �[32m'1-2-0'�[39m } }
.{ result: 
   { data: 
      [ { package: 
           { ecosystem: [ �[32m'maven'�[39m ],
             gh_forks: [ �[33m-1�[39m ],
             gh_issues_last_month_closed: [ �[33m-1�[39m ],
             gh_issues_last_month_opened: [ �[33m-1�[39m ],
             gh_issues_last_year_closed: [ �[33m-1�[39m ],
             gh_issues_last_year_opened: [ �[33m-1�[39m ],
             gh_prs_last_month_closed: [ �[33m-1�[39m ],
             gh_prs_last_month_opened: [ �[33m-1�[39m ],
             gh_prs_last_year_closed: [ �[33m-1�[39m ],
             gh_prs_last_year_opened: [ �[33m-1�[39m ],
             gh_stargazers: [ �[33m-1�[39m ],
             last_updated: [ �[33m1497540953.71�[39m ],
             latest_version: [ �[32m'2.0.5'�[39m ],
             name: [ �[32m'org.apache.pdfbox:pdfbox'�[39m ],
             package_dependents_count: [ �[33m-1�[39m ],
             package_relative_used: [ �[32m'not used'�[39m ],
             tokens: [ �[32m'apache'�[39m, �[32m'org'�[39m, �[32m'pdfbox'�[39m ],
             vertex_label: [ �[32m'Package'�[39m ] },
          version: 
           { cm_avg_cyclomatic_complexity: [ �[33m-1�[39m ],
             cm_loc: [ �[33m127509�[39m ],
             cm_num_files: [ �[33m593�[39m ],
             cve_ids: [ �[32m'CVE-2016-2175:7.5'�[39m ],
             dependents_count: [ �[33m-1�[39m ],
             description: [ �[32m'the apache pdfbox library is an open source java tool for working with pdf documents'�[39m ],
             last_updated: [ �[33m1492825323.68�[39m ],
             licenses: [ �[32m'APAFML'�[39m, �[32m'ASL 2.0'�[39m, �[32m'BSD'�[39m ],
             pecosystem: [ �[32m'maven'�[39m ],
             pname: [ �[32m'org.apache.pdfbox:pdfbox'�[39m ],
             relative_used: [ �[32m''�[39m ],
             shipped_as_downstream: [ �[33mfalse�[39m ],
             version: [ �[32m'1.8.11'�[39m ],
             vertex_label: [ �[32m'Version'�[39m ] } } ],
     recommendation: { �[32m'component-analyses'�[39m: { cve: [ { cvss: �[33m7.5�[39m, id: �[32m'CVE-2016-2175'�[39m } ] } } },
  schema: 
   { name: �[32m'analyses_graphdb'�[39m,
     url: �[32m'http://recommender.api.openshift.io/api/v1/schemas/api/analyses_graphdb/1-2-0/'�[39m,
     version: �[32m'1-2-0'�[39m } }
.{ result: 
   { data: 
      [ { package: 
           { ecosystem: [ �[32m'maven'�[39m ],
             gh_forks: [ �[33m1112�[39m ],
             gh_issues_last_month_closed: [ �[33m22�[39m ],
             gh_issues_last_month_opened: [ �[33m27�[39m ],
             gh_issues_last_year_closed: [ �[33m231�[39m ],
             gh_issues_last_year_opened: [ �[33m290�[39m ],
             gh_prs_last_month_closed: [ �[33m32�[39m ],
             gh_prs_last_month_opened: [ �[33m38�[39m ],
             gh_prs_last_year_closed: [ �[33m260�[39m ],
             gh_prs_last_year_opened: [ �[33m268�[39m ],
             gh_stargazers: [ �[33m-1�[39m ],
             last_updated: [ �[33m1498245532.97�[39m ],
             latest_version: [ �[32m'3.4.1'�[39m ],
             name: [ �[32m'io.vertx:vertx-core'�[39m ],
             package_dependents_count: [ �[33m-1�[39m ],
             package_relative_used: [ �[32m'not used'�[39m ],
             tokens: [ �[32m'core'�[39m, �[32m'io'�[39m, �[32m'vertx'�[39m ],
             vertex_label: [ �[32m'Package'�[39m ] },
          version: 
           { cm_avg_cyclomatic_complexity: [ �[33m1.34�[39m ],
             cm_loc: [ �[33m80586�[39m ],
             cm_num_files: [ �[33m441�[39m ],
             dependents_count: [ �[33m-1�[39m ],
             description: [ �[32m'sonatype helps open source projects to set up maven repositories on httpsosssonatypeorg'�[39m ],
             last_updated: [ �[33m1498245532.97�[39m ],
             licenses: [ �[32m'Apache 2.0'�[39m, �[32m'EPL 1.0'�[39m, �[32m'MIT License'�[39m ],
             pecosystem: [ �[32m'maven'�[39m ],
             pname: [ �[32m'io.vertx:vertx-core'�[39m ],
             relative_used: [ �[32m''�[39m ],
             shipped_as_downstream: [ �[33mfalse�[39m ],
             version: [ �[32m'3.4.1'�[39m ],
             vertex_label: [ �[32m'Version'�[39m ] } } ],
     recommendation: {} },
  schema: 
   { name: �[32m'analyses_graphdb'�[39m,
     url: �[32m'http://recommender.api.openshift.io/api/v1/schemas/api/analyses_graphdb/1-2-0/'�[39m,
     version: �[32m'1-2-0'�[39m } }
.{ result: 
   { data: 
      [ { package: 
           { ecosystem: [ �[32m'maven'�[39m ],
             gh_forks: [ �[33m-1�[39m ],
             gh_issues_last_month_closed: [ �[33m-1�[39m ],
             gh_issues_last_month_opened: [ �[33m-1�[39m ],
             gh_issues_last_year_closed: [ �[33m-1�[39m ],
             gh_issues_last_year_opened: [ �[33m-1�[39m ],
             gh_prs_last_month_closed: [ �[33m-1�[39m ],
             gh_prs_last_month_opened: [ �[33m-1�[39m ],
             gh_prs_last_year_closed: [ �[33m-1�[39m ],
             gh_prs_last_year_opened: [ �[33m-1�[39m ],
             gh_stargazers: [ �[33m-1�[39m ],
             last_updated: [ �[33m1498245621.73�[39m ],
             latest_version: [ �[32m'3.4.1'�[39m ],
             name: [ �[32m'io.vertx:vertx-web'�[39m ],
             package_dependents_count: [ �[33m-1�[39m ],
             package_relative_used: [ �[32m'not used'�[39m ],
             tokens: [ �[32m'io'�[39m, �[32m'vertx'�[39m, �[32m'web'�[39m ],
             vertex_label: [ �[32m'Package'�[39m ] },
          version: 
           { cm_avg_cyclomatic_complexity: [ �[33m1.22�[39m ],
             cm_loc: [ �[33m37699�[39m ],
             cm_num_files: [ �[33m239�[39m ],
             dependents_count: [ �[33m-1�[39m ],
             description: [ �[32m'sonatype helps open source projects to set up maven repositories on httpsosssonatypeorg'�[39m ],
             last_updated: [ �[33m1498245621.73�[39m ],
             licenses: [ �[32m'Apache 2.0'�[39m, �[32m'EPL 1.0'�[39m ],
             pecosystem: [ �[32m'maven'�[39m ],
             pname: [ �[32m'io.vertx:vertx-web'�[39m ],
             shipped_as_downstream: [ �[33mfalse�[39m ],
             version: [ �[32m'3.4.1'�[39m ],
             vertex_label: [ �[32m'Version'�[39m ] } } ],
     recommendation: {} },
  schema: 
   { name: �[32m'analyses_graphdb'�[39m,
     url: �[32m'http://recommender.api.openshift.io/api/v1/schemas/api/analyses_graphdb/1-2-0/'�[39m,
     version: �[32m'1-2-0'�[39m } }
.{ result: 
   { data: 
      [ { package: 
           { ecosystem: [ �[32m'maven'�[39m ],
             gh_forks: [ �[33m9704�[39m ],
             gh_issues_last_month_closed: [ �[33m0�[39m ],
             gh_issues_last_month_opened: [ �[33m0�[39m ],
             gh_issues_last_year_closed: [ �[33m0�[39m ],
             gh_issues_last_year_opened: [ �[33m0�[39m ],
             gh_prs_last_month_closed: [ �[33m33�[39m ],
             gh_prs_last_month_opened: [ �[33m34�[39m ],
             gh_prs_last_year_closed: [ �[33m327�[39m ],
             gh_prs_last_year_opened: [ �[33m358�[39m ],
             gh_stargazers: [ �[33m-1�[39m ],
             last_updated: [ �[33m1498245706.49�[39m ],
             latest_version: [ �[32m'4.3.7.RELEASE'�[39m ],
             name: [ �[32m'org.springframework:spring-core'�[39m ],
             package_dependents_count: [ �[33m-1�[39m ],
             package_relative_used: [ �[32m'not used'�[39m ],
             tokens: [ �[32m'core'�[39m, �[32m'org'�[39m, �[32m'springframework'�[39m, �[32m'spring'�[39m ],
             vertex_label: [ �[32m'Package'�[39m ] },
          version: 
           { cm_avg_cyclomatic_complexity: [ �[33m-1�[39m ],
             cm_loc: [ �[33m69221�[39m ],
             cm_num_files: [ �[33m376�[39m ],
             dependents_count: [ �[33m-1�[39m ],
             description: [ �[32m'spring core'�[39m ],
             last_updated: [ �[33m1492815354.51�[39m ],
             licenses: [ �[32m'ASL 2.0'�[39m, �[32m'LBNL BSD'�[39m ],
             pecosystem: [ �[32m'maven'�[39m ],
             pname: [ �[32m'org.springframework:spring-core'�[39m ],
             relative_used: [ �[32m''�[39m ],
             shipped_as_downstream: [ �[33mfalse�[39m ],
             version: [ �[32m'4.3.3.RELEASE'�[39m ],
             vertex_label: [ �[32m'Version'�[39m ] } } ],
     recommendation: {} },
  schema: 
   { name: �[32m'analyses_graphdb'�[39m,
     url: �[32m'http://recommender.api.openshift.io/api/v1/schemas/api/analyses_graphdb/1-2-0/'�[39m,
     version: �[32m'1-2-0'�[39m } }
F

Failures:

  1) Frisby Test: Connecting to REST API - endpoint :: https://recommender.api.openshift.io/api/v1/maven/org.springframework:spring-core/4.3.3.RELEASE 
	[ GET https://recommender.api.openshift.io/api/v1/component-analyses/maven/org.springframework:spring-core/4.3.3.RELEASE ]
   Message:
     Expected [ 'ASL 2.0', 'LBNL BSD' ] to contain 'Apache 2.0'.
   Stacktrace:
     Error: Expected [ 'ASL 2.0', 'LBNL BSD' ] to contain 'Apache 2.0'.
    at Frisby.<anonymous> (/Users/mdl/workspace/git-hub-lib/redhat/fabric8-test/analytics_EE_tests/api/1704E468-cve-scoring.spec.js:89:58)
    at Frisby.<anonymous> (/Users/mdl/workspace/git-hub-lib/redhat/fabric8-test/analytics_EE_tests/node_modules/frisby/lib/frisby.js:951:8)
    at jasmine.Spec.<anonymous> (/Users/mdl/workspace/git-hub-lib/redhat/fabric8-test/analytics_EE_tests/node_modules/frisby/lib/frisby.js:1112:18)
    at ontimeout (timers.js:386:14)
    at tryOnTimeout (timers.js:250:5)

Finished in 3.543 seconds
12 tests, 17 assertions, 1 failure, 0 skipped

Create EE tests through REST API to mirror UI-based EE tests

allow 'just jenkins' or 'just che' flavours of the E2E tests

it would be nice to be able to just run Jenkins tests and ignore Che and vice versa so that we can run the E2E tests in the che & jenkins pipelines and avoid unnecessary time and resources spent testing if Che works or Jenkins

I figure we could just introduce some environment variable that gets populated into the browser params:
https://github.com/fabric8io/fabric8-test/blob/master/ee_tests/protractorEE-env.config.js#L88-L120

that makes the tests ignore the Jenkins parts or the Che parts

Failue while pulling docker image

Step 17 : COPY ../analytics_EE_tests/ $WORKSPACE
Forbidden path outside the build context: ../analytics_EE_tests/ ()
Unable to find image 'fabric8-ui-builder:latest' locally
Pulling repository docker.io/library/fabric8-ui-builder
docker: Error: image library/fabric8-ui-builder:latest not found.
See 'docker run --help'.
Error response from daemon: No such container: fabric8-ui-builder
Error response from daemon: No such container: fabric8-ui-builder
Error response from daemon: No such container: fabric8-ui-builder
Error response from daemon: No such container: fabric8-ui-builder

Refactor tests to replace object references that rely on text to use UI element ID's

The tests today are too brittle - references to objects' text should be replaced with object ID's.

stack traces not including the messages?

i've noticed when timeouts fail we lose the useful message in the code. See the sample output of the test below. The code that actually failed is this line:

    browser.wait(until.presenceOf(OpenShiftIoDashboardPage.jenkinsStatusPoweredOn), constants.RESET_TENANT_WAIT), "Timeout waiting for Jenkins to start after tenant update - see: https://github.com/openshiftio/openshift.io/issues/595";

yet that nice message is lost so the stack trace is not that meaningful (unless you then go through the stack lines to find the code that caused the error).

Its really nice getting tests to give as meaningful errors as possible when things go wrong to minimise time wasted trying to grok the failure. I wonder if we can try get the message included in the failure message?

Failed: Wait timed out after 300005ms
TimeoutError: Wait timed out after 300005ms
    at /workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/promise.js:2107:17
    at ManagedPromise.invokeCallback_ (/workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/promise.js:1366:14)
    at TaskQueue.execute_ (/workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/promise.js:2970:14)
    at TaskQueue.executeNext_ (/workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/promise.js:2953:27)
    at asyncRun (/workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/promise.js:2813:27)
    at /workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/promise.js:676:7
    at process._tickCallback (internal/process/next_tick.js:103:7)
From: Task: <anonymous wait>
    at scheduleWait (/workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/promise.js:2094:20)
    at ControlFlow.wait (/workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/promise.js:2408:12)
    at thenableWebDriverProxy.wait (/workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/webdriver.js:943:29)
    at run (/workspace/java/fabric8-test/ee_tests/node_modules/protractor/lib/browser.ts:66:27)
    at ProtractorBrowser.to.(anonymous function) [as wait] (/workspace/java/fabric8-test/ee_tests/node_modules/protractor/lib/browser.ts:74:12)
    at Object.loginCleanUpdate (/workspace/java/fabric8-test/ee_tests/src/tests/work-item/work-item-list/testSupport.js:178:13)
    at runTheTest (/workspace/java/fabric8-test/ee_tests/src/tests/work-item/work-item-list/EE/openshift-io-burr-quickstart.spec.js:132:44)
    at UserContext.<anonymous> (/workspace/java/fabric8-test/ee_tests/src/tests/work-item/work-item-list/EE/openshift-io-burr-quickstart.spec.js:90:7)
    at /workspace/java/fabric8-test/ee_tests/node_modules/jasminewd2/index.js:112:25
    at new ManagedPromise (/workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/promise.js:1067:7)
    at ControlFlow.promise (/workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/promise.js:2396:12)
    at schedulerExecute (/workspace/java/fabric8-test/ee_tests/node_modules/jasminewd2/index.js:95:18)
    at TaskQueue.execute_ (/workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/promise.js:2970:14)
    at TaskQueue.executeNext_ (/workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/promise.js:2953:27)
    at asyncRun (/workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/promise.js:2860:25)
    at /workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/promise.js:676:7
    at process._tickCallback (internal/process/next_tick.js:103:7)
From: Task: Run it("should perform Quickstart test - CREATE project - Run Pipeline - Vert.x - Basic") in control flow
    at UserContext.<anonymous> (/workspace/java/fabric8-test/ee_tests/node_modules/jasminewd2/index.js:94:19)
    at attempt (/workspace/java/fabric8-test/ee_tests/node_modules/jasmine/node_modules/jasmine-core/lib/jasmine-core/jasmine.js:4297:26)
    at QueueRunner.run (/workspace/java/fabric8-test/ee_tests/node_modules/jasmine/node_modules/jasmine-core/lib/jasmine-core/jasmine.js:4217:20)
    at runNext (/workspace/java/fabric8-test/ee_tests/node_modules/jasmine/node_modules/jasmine-core/lib/jasmine-core/jasmine.js:4257:20)
    at /workspace/java/fabric8-test/ee_tests/node_modules/jasmine/node_modules/jasmine-core/lib/jasmine-core/jasmine.js:4264:13
    at /workspace/java/fabric8-test/ee_tests/node_modules/jasmine/node_modules/jasmine-core/lib/jasmine-core/jasmine.js:4172:9
    at /workspace/java/fabric8-test/ee_tests/node_modules/jasminewd2/index.js:64:48
    at ControlFlow.emit (/workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/events.js:62:21)
    at ControlFlow.shutdown_ (/workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/promise.js:2565:10)
    at shutdownTask_.MicroTask (/workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/promise.js:2490:53)
    at MicroTask.asyncRun (/workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/promise.js:2619:9)
    at /workspace/java/fabric8-test/ee_tests/node_modules/selenium-webdriver/lib/promise.js:676:7
    at process._tickCallback (internal/process/next_tick.js:103:7)

Minor: Add node_modules to .gitignore

Generate screenshots for failing UI-based tests

This utility should generate screenshots:

protractor-jasmine2-screenshot-reporter

lets remove the use of local_cleanup.sh / local_cleanup_che.sh and instead use the new `Reset Environment`

Lets use the Update Profile page of the Profile page (either navigating via the top right drop down or using the canonical URL of $URL/_profile/_update when the new canonical https://openshift.io/_profile page goes live in prod today hopefully) - and then click the Reset Environment button before the test to ensure things are all cleaned down etc

Clean occurrences of openshift.io in tests

Test's should be environment agnostic - i.e. they should be able to run on production and preview with the right configuration. It seems there are some URLs hardcoded across the repository

All of these should be replaced by a variable setup through arguments from CLI

There might be others which I missed, I just did a quick search on the repo

Test ID should be in the file

How would I find the associated test?

When I run the license test I get a failure. But unfortunately I can't see the test-case. I need to jump to line 55 to see that it's Testcase: Test-1704E469-04. Can we report the test-case when a failure happens?

jasmine-node api/1704E469-license.spec.js --config targetURL https://recommender.api.openshift.io/api/v1/
.....F

Failures:

  1) Frisby Test: Connecting to REST API - endpoint :: https://recommender.api.openshift.io/api/v1/maven/org.springframework:spring-core/4.3.3.RELEASE
	[ GET https://recommender.api.openshift.io/api/v1/component-analyses/maven/org.springframework:spring-core/4.3.3.RELEASE ]
   Message:
     Expected [ 'ASL 2.0', 'LBNL BSD' ] to contain 'Apache 2.0'.
   Stacktrace:
     Error: Expected [ 'ASL 2.0', 'LBNL BSD' ] to contain 'Apache 2.0'.
    at Frisby.<anonymous> (/Users/mdl/workspace/git-hub-lib/redhat/fabric8-test/analytics_EE_tests/api/1704E469-license.spec.js:55:58)
    at Frisby.<anonymous> (/Users/mdl/workspace/git-hub-lib/redhat/fabric8-test/analytics_EE_tests/node_modules/frisby/lib/frisby.js:951:8)
    at jasmine.Spec.<anonymous> (/Users/mdl/workspace/git-hub-lib/redhat/fabric8-test/analytics_EE_tests/node_modules/frisby/lib/frisby.js:1112:18)
    at ontimeout (timers.js:386:14)
    at tryOnTimeout (timers.js:250:5)

Finished in 2.161 seconds
6 tests, 8 assertions, 1 failure, 0 skipped

Extend EE tests to exercise deployed app on stage and run

Refactor EE import tests to accept project repo as parameter - for booster tests

So that the tests can can run against any repo - including all boosters.

parameterise the E2E tests so we can specify tenant parameters

if you navigate to the new tenant page there are a number of properties you can customise on a per tenant basis now: https://openshift.io/_profile/_tenant

so these can be used to test a specific PR of the jenkins / che / team YAML using a custom maven repository etc. Or they can be used to test a specific PR / fork of the Forge booster catalogs.

So it'd be nice to be able to pass in all those values (optionally) to the E2E test and if any of the values are present, to navigate to the page: https://openshift.io/_profile/_tenant then update the form and click the Save and Update Tenant button before running the tests - then clearing the values at the end of the test.

This will mean we can reuse the E2E tests and pass in environment variables for these values to test PRs of booster catalogs or che/jenkins/team YAMLs

Extend EE tests to cover all supported deployment strategies

Followup EE test changes in response to new app dialog changes

The following changes should be made to the EE tests

Implement check for successful creation of new app
Implement scrolling in new app dialog to support selection of all quickstarts
Extend existing Che and Analytics tests beyond POC test coverage

Configure tests to run on prod-preview, in addition to current production config

Run tests unattended - Resolve tests that randomly fail with: Chrome failed to start: exited abnormally

Extend EE tests to cover user accounts that are/are not in github organizations

Resolve EE test failures caused by - Failed: unknown error: Chrome failed to start: exited abnormally

openshift.io End-to-End POC test - Scenario - CREATE project - Run Pipeline:  should perform Burr's demo - CREATE project - Run Pipeline
  Message:
�[31m    Failed: unknown error: Chrome failed to start: exited abnormally
      (Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 3.10.0-693.2.2.el7.x86_64 x86_64) (WARNING: The server did not provide any stacktrace information)
    Command duration or timeout: 60.07 seconds
    Build info: version: '3.5.3', revision: 'a88d25fe6b', time: '2017-08-29T12:54:15.039Z'
    System info: host: 'b46ae730141d', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-693.2.2.el7.x86_64', java.version: '1.8.0_144'
    Driver info: driver.version: unknown�[0m
  Stack:
    WebDriverError: unknown error: Chrome failed to start: exited abnormally
      (Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 3.10.0-693.2.2.el7.x86_64 x86_64) (WARNING: The server did not provide any stacktrace information)
    Command duration or timeout: 60.07 seconds
    Build info: version: '3.5.3', revision: 'a88d25fe6b', time: '2017-08-29T12:54:15.039Z'
    System info: host: 'b46ae730141d', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-693.2.2.el7.x86_64', java.version: '1.8.0_144'
    Driver info: driver.version: unknown
        at WebDriverError (/opt/fabric8-test/node_modules/selenium-webdriver/lib/error.js:27:5)
        at Object.checkLegacyResponse (/opt/fabric8-test/node_modules/selenium-webdriver/lib/error.js:505:15)
        at parseHttpResponse (/opt/fabric8-test/node_modules/selenium-webdriver/lib/http.js:509:13)
        at doSend.then.response (/opt/fabric8-test/node_modules/selenium-webdriver/lib/http.js:440:13)
        at process._tickCallback (internal/process/next_tick.js:103:7)
    From: Task: WebDriver.createSession()
        at Function.createSession (/opt/fabric8-test/node_modules/selenium-webdriver/lib/webdriver.js:777:24)
        at Function.createSession (/opt/fabric8-test/node_modules/selenium-webdriver/chrome.js:709:29)
        at createDriver (/opt/fabric8-test/node_modules/selenium-webdriver/index.js:167:33)
        at Builder.build (/opt/fabric8-test/node_modules/selenium-webdriver/index.js:623:16)
        at Hosted.getNewDriver (/opt/fabric8-test/node_modules/protractor/lib/driverProviders/driverProvider.ts:60:29)
        at Runner.createBrowser (/opt/fabric8-test/node_modules/protractor/lib/runner.ts:225:39)
        at ProtractorBrowser.browser_.forkNewDriverInstance (/opt/fabric8-test/node_modules/protractor/lib/runner.ts:293:33)
        at replaceBrowser (/opt/fabric8-test/node_modules/protractor/lib/runner.ts:313:33)
        at ProtractorBrowser.browser_.restartSync (/opt/fabric8-test/node_modules/protractor/lib/runner.ts:340:14)
        at ProtractorBrowser.browser_.restart (/opt/fabric8-test/node_modules/protractor/lib/runner.ts:326:25)
    From: Task: Run afterEach in control flow
        at UserContext.<anonymous> (/opt/fabric8-test/node_modules/jasminewd2/index.js:94:19)
        at /opt/fabric8-test/node_modules/jasminewd2/index.js:64:48
        at ControlFlow.emit (/opt/fabric8-test/node_modules/selenium-webdriver/lib/events.js:62:21)
        at ControlFlow.shutdown_ (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:2565:10)
        at shutdownTask_.MicroTask (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:2490:53)
    From asynchronous test: 
    Error
        at Suite.<anonymous> (/opt/fabric8-test/src/tests/work-item/work-item-list/EE/openshift-io-burr-che.spec.js:82:3)
        at Object.<anonymous> (/opt/fabric8-test/src/tests/work-item/work-item-list/EE/openshift-io-burr-che.spec.js:68:1)
        at Module._compile (module.js:556:32)
        at Object.Module._extensions..js (module.js:565:10)
        at Module.load (module.js:473:32)
        at tryModuleLoad (module.js:432:12)
    From: Task: WebDriver.manage().window().setSize(1920, 1080)
        at thenableWebDriverProxy.schedule (/opt/fabric8-test/node_modules/selenium-webdriver/lib/webdriver.js:816:17)
        at Window.setSize (/opt/fabric8-test/node_modules/selenium-webdriver/lib/webdriver.js:1583:25)
        at Object.setBrowserMode (/opt/fabric8-test/src/tests/work-item/work-item-list/testSupport.js:30:42)
        at UserContext.<anonymous> (/opt/fabric8-test/src/tests/work-item/work-item-list/EE/openshift-io-burr-pipeline.spec.js:71:17)
        at /opt/fabric8-test/node_modules/jasminewd2/index.js:112:25
        at new ManagedPromise (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:1067:7)
        at ControlFlow.promise (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:2396:12)
        at schedulerExecute (/opt/fabric8-test/node_modules/jasminewd2/index.js:95:18)
        at TaskQueue.execute_ (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:2970:14)
        at TaskQueue.executeNext_ (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:2953:27)
    From: Task: Run beforeEach in control flow
        at UserContext.<anonymous> (/opt/fabric8-test/node_modules/jasminewd2/index.js:94:19)
    From asynchronous test: 
    Error
        at Suite.<anonymous> (/opt/fabric8-test/src/tests/work-item/work-item-list/EE/openshift-io-burr-pipeline.spec.js:70:3)
        at Object.<anonymous> (/opt/fabric8-test/src/tests/work-item/work-item-list/EE/openshift-io-burr-pipeline.spec.js:66:1)
        at Module._compile (module.js:556:32)
        at Object.Module._extensions..js (module.js:565:10)
        at Module.load (module.js:473:32)
        at tryModuleLoad (module.js:432:12)
  Message:
�[31m    Failed: unknown error: Chrome failed to start: exited abnormally
      (Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 3.10.0-693.2.2.el7.x86_64 x86_64) (WARNING: The server did not provide any stacktrace information)
    Command duration or timeout: 60.07 seconds
    Build info: version: '3.5.3', revision: 'a88d25fe6b', time: '2017-08-29T12:54:15.039Z'
    System info: host: 'b46ae730141d', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-693.2.2.el7.x86_64', java.version: '1.8.0_144'
    Driver info: driver.version: unknown�[0m
  Stack:
    WebDriverError: unknown error: Chrome failed to start: exited abnormally
      (Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 3.10.0-693.2.2.el7.x86_64 x86_64) (WARNING: The server did not provide any stacktrace information)
    Command duration or timeout: 60.07 seconds
    Build info: version: '3.5.3', revision: 'a88d25fe6b', time: '2017-08-29T12:54:15.039Z'
    System info: host: 'b46ae730141d', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-693.2.2.el7.x86_64', java.version: '1.8.0_144'
    Driver info: driver.version: unknown
        at WebDriverError (/opt/fabric8-test/node_modules/selenium-webdriver/lib/error.js:27:5)
        at Object.checkLegacyResponse (/opt/fabric8-test/node_modules/selenium-webdriver/lib/error.js:505:15)
        at parseHttpResponse (/opt/fabric8-test/node_modules/selenium-webdriver/lib/http.js:509:13)
        at doSend.then.response (/opt/fabric8-test/node_modules/selenium-webdriver/lib/http.js:440:13)
        at process._tickCallback (internal/process/next_tick.js:103:7)
    From: Task: WebDriver.createSession()
        at Function.createSession (/opt/fabric8-test/node_modules/selenium-webdriver/lib/webdriver.js:777:24)
        at Function.createSession (/opt/fabric8-test/node_modules/selenium-webdriver/chrome.js:709:29)
        at createDriver (/opt/fabric8-test/node_modules/selenium-webdriver/index.js:167:33)
        at Builder.build (/opt/fabric8-test/node_modules/selenium-webdriver/index.js:623:16)
        at Hosted.getNewDriver (/opt/fabric8-test/node_modules/protractor/lib/driverProviders/driverProvider.ts:60:29)
        at Runner.createBrowser (/opt/fabric8-test/node_modules/protractor/lib/runner.ts:225:39)
        at ProtractorBrowser.browser_.forkNewDriverInstance (/opt/fabric8-test/node_modules/protractor/lib/runner.ts:293:33)
        at replaceBrowser (/opt/fabric8-test/node_modules/protractor/lib/runner.ts:313:33)
        at ProtractorBrowser.browser_.restartSync (/opt/fabric8-test/node_modules/protractor/lib/runner.ts:340:14)
        at ProtractorBrowser.browser_.restart (/opt/fabric8-test/node_modules/protractor/lib/runner.ts:326:25)
    From: Task: Run afterEach in control flow
        at UserContext.<anonymous> (/opt/fabric8-test/node_modules/jasminewd2/index.js:94:19)
        at /opt/fabric8-test/node_modules/jasminewd2/index.js:64:48
        at ControlFlow.emit (/opt/fabric8-test/node_modules/selenium-webdriver/lib/events.js:62:21)
        at ControlFlow.shutdown_ (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:2565:10)
        at shutdownTask_.MicroTask (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:2490:53)
    From asynchronous test: 
    Error
        at Suite.<anonymous> (/opt/fabric8-test/src/tests/work-item/work-item-list/EE/openshift-io-burr-che.spec.js:82:3)
        at Object.<anonymous> (/opt/fabric8-test/src/tests/work-item/work-item-list/EE/openshift-io-burr-che.spec.js:68:1)
        at Module._compile (module.js:556:32)
        at Object.Module._extensions..js (module.js:565:10)
        at Module.load (module.js:473:32)
        at tryModuleLoad (module.js:432:12)
    From: Task: WebDriver.manage().window().setSize(1920, 1080)
        at thenableWebDriverProxy.schedule (/opt/fabric8-test/node_modules/selenium-webdriver/lib/webdriver.js:816:17)
        at Window.setSize (/opt/fabric8-test/node_modules/selenium-webdriver/lib/webdriver.js:1583:25)
        at Object.setBrowserMode (/opt/fabric8-test/src/tests/work-item/work-item-list/testSupport.js:30:42)
        at UserContext.<anonymous> (/opt/fabric8-test/src/tests/work-item/work-item-list/EE/openshift-io-burr-pipeline.spec.js:71:17)
        at /opt/fabric8-test/node_modules/jasminewd2/index.js:112:25
        at new ManagedPromise (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:1067:7)
        at ControlFlow.promise (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:2396:12)
        at schedulerExecute (/opt/fabric8-test/node_modules/jasminewd2/index.js:95:18)
        at TaskQueue.execute_ (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:2970:14)
        at TaskQueue.executeNext_ (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:2953:27)
    From: Task: Run beforeEach in control flow
        at UserContext.<anonymous> (/opt/fabric8-test/node_modules/jasminewd2/index.js:94:19)
    From asynchronous test: 
    Error
        at Suite.<anonymous> (/opt/fabric8-test/src/tests/work-item/work-item-list/EE/openshift-io-burr-pipeline.spec.js:70:3)
        at Object.<anonymous> (/opt/fabric8-test/src/tests/work-item/work-item-list/EE/openshift-io-burr-pipeline.spec.js:66:1)
        at Module._compile (module.js:556:32)
        at Object.Module._extensions..js (module.js:565:10)
        at Module.load (module.js:473:32)
        at tryModuleLoad (module.js:432:12)
    From: Task: WebDriver.findElements(By(xpath, .//*[@id='login']))
        at thenableWebDriverProxy.schedule (/opt/fabric8-test/node_modules/selenium-webdriver/lib/webdriver.js:816:17)
        at thenableWebDriverProxy.findElements (/opt/fabric8-test/node_modules/selenium-webdriver/lib/webdriver.js:1057:22)
        at ptor.waitForAngular.then (/opt/fabric8-test/node_modules/protractor/lib/element.ts:166:36)
        at ManagedPromise.invokeCallback_ (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:1366:14)
        at TaskQueue.execute_ (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:2970:14)
        at TaskQueue.executeNext_ (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:2953:27)
        at asyncRun (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:2813:27)
        at /opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:676:7Error
        at ElementArrayFinder.applyAction_ (/opt/fabric8-test/node_modules/protractor/lib/element.ts:482:23)
        at ElementArrayFinder.(anonymous function) [as click] (/opt/fabric8-test/node_modules/protractor/lib/element.ts:96:21)
        at ElementFinder.(anonymous function) [as click] (/opt/fabric8-test/node_modules/protractor/lib/element.ts:873:14)
        at OpenShiftIoStartPage.clickLoginButton (/opt/fabric8-test/src/tests/work-item/work-item-list/page-objects/openshift-io-start.page.js:315:23)
        at Object.loginCleanUpdate (/opt/fabric8-test/src/tests/work-item/work-item-list/testSupport.js:194:26)
        at UserContext.<anonymous> (/opt/fabric8-test/src/tests/work-item/work-item-list/EE/openshift-io-burr-pipeline.spec.js:94:44)
        at /opt/fabric8-test/node_modules/jasminewd2/index.js:112:25
        at new ManagedPromise (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:1067:7)
        at ControlFlow.promise (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:2396:12)
        at schedulerExecute (/opt/fabric8-test/node_modules/jasminewd2/index.js:95:18)
    From: Task: Run it("should perform Burr's demo - CREATE project - Run Pipeline") in control flow
        at UserContext.<anonymous> (/opt/fabric8-test/node_modules/jasminewd2/index.js:94:19)
        at /opt/fabric8-test/node_modules/jasminewd2/index.js:64:48
        at ControlFlow.emit (/opt/fabric8-test/node_modules/selenium-webdriver/lib/events.js:62:21)
        at ControlFlow.shutdown_ (/opt/fabric8-test/node_modules/selenium-webdriver/lib/promise.js:2565:10)
    From asynchronous test: 
    Error
        at Suite.<anonymous> (/opt/fabric8-test/src/tests/work-item/work-item-list/EE/openshift-io-burr-pipeline.spec.js:85:3)
        at Object.<anonymous> (/opt/fabric8-test/src/tests/work-item/work-item-list/EE/openshift-io-burr-pipeline.spec.js:66:1)
        at Module._compile (module.js:556:32)
        at Object.Module._extensions..js (module.js:565:10)
        at Module.load (module.js:473:32)
        at tryModuleLoad (module.js:432:12)

How to identify & run individual tests

The review document lists individual test-cases. e.g. Test-1704E468-03 & Test-1704E468-04. To see the overall review status we need a way to connect these test-cases with tests described here. Note that we don't necessarily need to keep the naming scheme Test-ExperienceID-TestNr. but we need to identify them.

Testcase Test-1704E469-02 should match for exactly the same set

The expected result for Licenses on io.vertx:vertx-core/3.4.1 is two licenses: Apache and MIT. See also https://mvnrepository.com/artifact/io.vertx/vertx-core/3.4.1

The API returns three licenses:

curl -sH "Authorization: Bearer $OSIO_TOKEN" https://recommender.api.openshift.io/api/v1/component-analyses/maven/io.vertx:vertx-core/3.4.1 | jq .result.data[0].version.licenses
[
  "Apache 2.0",
  "EPL 1.0",
  "MIT License"
]

But the testcase only checks if the results of the API is in the set of Apache and EPL. The additional MIT License should make the test fail.

Configure each test to reset user environment before running to ensure a clean starting point

Blocked by: openshiftio/openshift.io#1034

Also note - The tests are reaching a limit on vertical page size within Xvfb - if the defined DISPLAY size is > 1800, Chrome fails to open in Xvfb.

Refactor EE tests following creation of new 'add to space' dialog

Restore missing tests - remove all obsolete test code

Make forge import wizard API testing configurable

Depending wether we want to run the script with a OSiO account:

linked to a github account with no github organisation (step1.2, step1.3 is removed)
linked to a gh account with organisation (steo1.2, step 1.3 required)
We should wrap the shell script with an if statement.

Refactor EE tests - remove all sleep statements

It should be possible to rely on Protractor for all delays and waiting

Refactor tests to handle desktop, tablet, mobile configurations

Tests fail to login when openshift.io login name is different than github one

The test fails here:

at Object.<anonymous> (/Users/corinne/workspace/devtools/fabric8-test/ee_tests/src/tests/work-item/work-item-list/EE/openshift-io-burr.spec.js:121:30)

The problem is because it uses the same login for github and Openshift.io.
I launched the script with :

.sh ./local_run_EE_tests.sh USERNAME PASSWORD https://openshift.io

Code commented out in PR - https://github.com/fabric8io/fabric8-test/pull/172 - caused multiple tests to fail

The code that is commented out here: #172

Caused all tests other than the first test in a suite to fail with this error:
This driver instance does not have a valid session ID (did you call WebDriver.quit()?) and may no longer be used.

The problem is the manner in which new browser windows are opened.

Update Tests to Run on MacOS

Issue:
When trying to run npm run test on MacOS, they console hangs and never gets passed the 'Starting Webdriver and Selenium' step. In order to fix this, the user needs to manually change the run_functional_tests.sh file to use "nc" instead of "ncat".

Request:
Update run_functional_tests.sh to use "nc" rather than "ncat". This will allow the tests to run on both Linux and MacOS.

if a test fails can we (optionally) keep the browser window open?

when tests fail due to some missing UI button/feature not being found; it'd be nice to keep the browser window open for a while so we can try grok why the test failed; sometimes the message isn't super clear so it'd be nice to fix up the test to make things more obvious what went wrong & why. Sometimes its just the test needs to be a bit more clever/resilient - sometimes its a real bug/issue but just not reporting the real error nicely.

So it'd be nice to have a flag we can use to keep the browser around, giving the user a chance to look at where the browser got to & maybe grabbing the HTML and whatnot to try figure out what went wrong if the failure message is not good/obvious

I've tried googling and adding optional shutdown hooks to the protractor config.js to no avail so far - anyone else figured out a nice way?

Forge Wizard API tests for import repositories flow

We don't have any API testing for this flow.
Using pyresttest, add the 4 steps.

Add tests for /stack-analysis API

After component tests we need to add support for tests for the /stack-analysis API.

Note that getting results requires two steps: triggering an analysis to get a request id and than polling the api. A POC of writing these tests in bash can be found here: https://github.com/luebken/osio-tests

Refactor tests to replace object references that rely on xpath text to use UI element ID's

The tests today are too brittle - references to objects' xpaths should be replaced with object ID's.

add a CI pipeline to this git repo so that we verify all changes to the E2E tests work on OSIO

so that we can be sure any changes for fabric8 don't break the E2E tests

Explore use of BDD tools to generate feature coverage

The proposal is to explore tools (like cucumber) where we can define Features and test scenarios which would provide us some insight into what Features and scenarios are being tested.

Implement KC refresh token for EE tests

when working on tests, it'd be nice to reuse the browser session & reuse the login?

it can take a while to login from scratch each time; then clean down tenants etc. It might be nice to add some optional short circuit flags via env vars to let tests run faster by reusing a browser session & minimising re-logging in each time we create a quickstart or whatever

e.g. if you just want to re-run the create quickstart test which maybe has some issue multiple times locally; it'd be nice to not have to totally login from scratch, delete + recreate your tenant etc

provide an option to reuse the UIs 'Reset Environment'

we really should be using a fully clean and up to date tenant when running the full E2E tests - so using the UI's Profile -> Reset Environment UI to ensure that the tenant is totally cleaned before the test and that the most up to date tenant software is used (plus taking into account any PR for jenkins / che / boosters being tested too)

So it'd be nice to do something like:

reset the enivronment using the UI (so it really does a full reset & update of tenant before tests)
run just the Jenkins / pipeline tests first (which seem to work fine after a full reset)
only when all that has happened, try testing Che (which by then should have started ;)

Use container image to simplify creation of development and test env

Currently we have development and test env that is setup differently. Development env itself can be different based on how it is create - using nvm or system-packages and running the test can break due to these differences.

As a solution to this lets use a Docker image that provides the necessary tools node, npm, yarn and use that to run tests locally and in CI. The goal is to make it easy to reproduce failures.