comtihon / catcher Goto Github PK

View Code? Open in Web Editor NEW

94.0 94.0 13.0 1.13 MB

microservices and data pipelines end-to-end test tool

License: Apache License 2.0

Makefile 0.09% Python 94.73% Dockerfile 0.14% Java 0.13% Kotlin 0.04% JavaScript 0.09% CSS 1.35% HTML 3.44%

catcher e2e e2e-testing e2e-tests migration-tool test-automation test-framework testing-tools

catcher's People

Contributors

Stargazers

Watchers

Forkers

olegbukatchuk anandsunderraman latestalexey koloseum25 fluke667 brunojensen kbelova hazer-hazer kcbsbo daniel-han-yang sermojohn

catcher's Issues

ansible step

there should be a step to run an ansible playbook.
Probably with cloning it from git before (or use git, as another step)

Format json does't work with wait for module

I encountered a problem when using the --format json and wait (for) module:

For example, if I run following test with --format json:

steps:
  - wait:                                                                                                                                                                                                                
        name: 'Waiting for postgres'                                                                                                                                                                                               
        seconds: 30                                                                                                                                                                                                                
        for:                                                                                                                                                                                                                 
          postgres:                                                                                                                                                                                                            
            request:                                                                                                                                                                                                             
              conf: '{{ postgres_conf }}'                                                                                                                                                                                        
              query: 'select 1 as test;'

I get this error message:

Traceback (most recent call last):
  File "/usr/local/bin/catcher", line 33, in <module>
    sys.exit(load_entry_point('catcher==1.32.3', 'console_scripts', 'catcher')())
  File "/usr/local/lib/python3.7/site-packages/catcher-1.32.3-py3.7.egg/catcher/__main__.py", line 50, in main
    result = run_tests(path, arguments)
  File "/usr/local/lib/python3.7/site-packages/catcher-1.32.3-py3.7.egg/catcher/__main__.py", line 81, in run_tests
    return runner.run_tests(output=output)
  File "/usr/local/lib/python3.7/site-packages/catcher-1.32.3-py3.7.egg/catcher/core/runner.py", line 70, in run_tests
    logger.log_storage.write_report(join(self.path, 'reports'))
  File "/usr/local/lib/python3.7/site-packages/catcher-1.32.3-py3.7.egg/catcher/modules/log_storage.py", line 79, in write_report
    formatter.format(path, self._data)
  File "/usr/local/lib/python3.7/site-packages/catcher-1.32.3-py3.7.egg/catcher/modules/formatter.py", line 23, in format
    json.dump(data, fp)
  File "/usr/local/lib/python3.7/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/usr/local/lib/python3.7/json/encoder.py", line 429, in _iterencode
    yield from _iterencode_list(o, _current_indent_level)
  File "/usr/local/lib/python3.7/json/encoder.py", line 325, in _iterencode_list
    yield from chunks
  File "/usr/local/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/local/lib/python3.7/json/encoder.py", line 325, in _iterencode_list
    yield from chunks
  File "/usr/local/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/local/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/local/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  [Previous line repeated 1 more time]
  File "/usr/local/lib/python3.7/json/encoder.py", line 325, in _iterencode_list
    yield from chunks
  File "/usr/local/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/local/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/local/lib/python3.7/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/usr/local/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type function is not JSON serializable

If I run test without --format json all is correct.

But if I run simple wait module with --format json:

steps:
  - wait:
      seconds: 5

Everything works well.

I did a little research and here's what I found out.

The error data object

File "/usr/local/lib/python3.7/site-packages/catcher-1.32.3-py3.7.egg/catcher/modules/formatter.py", line 23, in format json.dump(data, fp)

looks like this:

[{
	'end_time': '2020-11-10 13:32:50',
	'start_time': '2020-11-10 13:32:49',
	'file': 'tests/001/main.yml',
	'type': 'test',
	'output': [{
		'time': '2020-11-10 13:32:49',
		'step': {
			'wait': {
				'name': 'Waiting for postgres',
				'seconds': 30,
				'for': {
					'postgres': {
						'request': {
							'conf': '{{ postgres_conf }}',
							'query': 'select 1 as test;'
						},
						'_get_actions': < function StepFactory.ensure_body_dict. < locals > .get_actions_fun at 0x7faf85cb30e0 > ,
						'_get_action': < function StepFactory.ensure_body_dict. < locals > .get_action_fun at 0x7faf85cb3170 > ,
						'_path': '/opt/catcher'
					}
				}
			}
		},
		'variables': {
			'PATH': '/root/.sdkman/candidates/kotlin/current/bin:/usr/lib/jvm/java-1.8-openjdk/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
			'HOSTNAME': 'aksenk',
			'LANG': 'C.UTF-8',
			'GPG_KEY': '0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D',
			'PYTHON_VERSION': '3.7.9',
			'PYTHON_PIP_VERSION': '20.2.3',
			'PYTHON_GET_PIP_URL': 'https://github.com/pypa/get-pip/raw/fa7dc83944936bf09a0e4cb5d5ec852c0d256599/get-pip.py',
			'PYTHON_GET_PIP_SHA256': '6e0bb0a2c2533361d7f297ed547237caf1b7507f197835974c0dd7eba998c53c',
			'JAVA_HOME': '/usr/lib/jvm/java-1.8-openjdk',
			'HOME': '/root',
			'postgres_conf': {
				'dbname': 'db',
				'user': 'postgres',
				'host': 'localhost',
				'password': 'password'
			},
			'INVENTORY': 'ci',
			'INVENTORY_FILE': 'inventory/ci.yml',
			'CURRENT_DIR': '/opt/catcher',
			'RESOURCES_DIR': '/opt/catcher/resources',
			'TEST_NAME': 'tests/001/main.yml'
		},
		'nested': 0
	}, {
		'time': '2020-11-10 13:32:50',
		'data': 'Step Waiting for postgres [0s]\x1b[32m OK\x1b[0m',
		'level': 'info'
	}, {
		'time': '2020-11-10 13:32:50',
		'step': {
			'wait': {
				'name': 'Waiting for postgres',
				'seconds': 30,
				'for': {
					'postgres': {
						'request': {
							'conf': '{{ postgres_conf }}',
							'query': 'select 1 as test;'
						},
						'_get_actions': < function StepFactory.ensure_body_dict. < locals > .get_actions_fun at 0x7faf85cb30e0 > ,
						'_get_action': < function StepFactory.ensure_body_dict. < locals > .get_action_fun at 0x7faf85cb3170 > ,
						'_path': '/opt/catcher'
					}
				}
			}
		},
		'variables': {
			'PATH': '/root/.sdkman/candidates/kotlin/current/bin:/usr/lib/jvm/java-1.8-openjdk/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
			'HOSTNAME': 'aksenk',
			'LANG': 'C.UTF-8',
			'GPG_KEY': '0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D',
			'PYTHON_VERSION': '3.7.9',
			'PYTHON_PIP_VERSION': '20.2.3',
			'PYTHON_GET_PIP_URL': 'https://github.com/pypa/get-pip/raw/fa7dc83944936bf09a0e4cb5d5ec852c0d256599/get-pip.py',
			'PYTHON_GET_PIP_SHA256': '6e0bb0a2c2533361d7f297ed547237caf1b7507f197835974c0dd7eba998c53c',
			'JAVA_HOME': '/usr/lib/jvm/java-1.8-openjdk',
			'HOME': '/root',
			'postgres_conf': {
				'dbname': 'db',
				'user': 'postgres',
				'host': 'localhost',
				'password': 'password'
			},
			'INVENTORY': 'ci',
			'INVENTORY_FILE': 'inventory/ci.yml',
			'CURRENT_DIR': '/opt/catcher',
			'RESOURCES_DIR': '/opt/catcher/resources',
			'TEST_NAME': 'tests/001/main.yml'
		},
		'nested': 0,
		'success': True,
		'output': None
	}, {
		'time': '2020-11-10 13:32:50',
		'data': 'Test main\x1b[32m passed.\x1b[0m',
		'level': 'info'
	}],
	'status': 'OK',
	'comment': None
}]

I think error in this section:

'_get_actions': < function StepFactory.ensure_body_dict. < locals > .get_actions_fun at 0x7faf85cb30e0 > ,
'_get_action': < function StepFactory.ensure_body_dict. < locals > .get_action_fun at 0x7faf85cb3170 > ,

http module basic auth feature

Does http module support basic auth feature.

I was looking for a way to do basic authentication with REST api.
I am not able to see a way to do it in docs.
The other option would be to write an external step that creates the Base64 encoding and then use that in the http step for the headers

Concurrent test execution

In test conf object add concurrency group:

variables:
    foo: bar
config:
   concurrency:
         enabled: true
         group: teamA
steps:
....

If config.concurrency.enabled is true - test can be run concurrent.
If config.concurrency.group is true - no tests of the same group can be run at once.
Should use multiprocessing and create pool with CPU + 1
Test results in json/html should be concurrent safe.

timing for each step

there should be timing for every step. F.e.

postgres:
  action:
     query: <some_query>

the output will be step postgres OK/fail in 4 seconds

Ability to run FE test steps

For full E2E test frontend also needs to be triggered.
Need to make selenium step?

Need to provide a working example on FE + BE E2E testing.

int and string variables

Need to revisit str vs int variables.
Is it possible to check them?
F.e.
if list [1,2,3,4] contains 2 or list ['1', '2', '3','4'] contains '2'

date 'yyyy-mm-dd' is handled as math operation on <3.7.5 python versions

Need to find a workaround for older python versions.

different sessions for http

For now only one http session is allowed.
It prevents test to run multiple users tests.
http step needs to have a named session param to specify which session should be used for the request.

Support templates in start variables

variables:
   phone_num: "{{ random(phone_number) }}"
steps:
  ...

instead of:

steps:
   - echo: {from: '{{ random("phone_number") }}', register: {phone_num: '{{ OUTPUT }}'}}

run output

run steps output should be differentiated from the main steps output.
Also there should be a possibility to hide them.

fix documentation

after catcher_modules 4.x.x documentation and examples needs to be updated.

requirements support

In case there is a requirements.txt in the resource folder - catcher should install these requirements before running tests

asdate filter should deal with different timestamp format.

f.e. 1586995200000 should be translated to 2020-04-16.
Current approach will throw ValueError: year 52259 is out of range error.
We need to delete this TS by 1000 to receive real value.

Dependabot couldn't authenticate with https://pypi.python.org/simple/

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

skip/when condition

Add ability to skip step based on a logical condition like in Ansible
F.e.:

steps:
  - docker: 
          start: 
            image: 'jamesdbloom/mockserver'
            ports:
              '1080/tcp': 8000
          when: 
            equals: {the: '{{ my_http_service_url }}', is: 'localhost'}

Make colorful output optional

In case of CI without tty it is unreadable.

Skip_if not working in loop

Using catcher-1.36.2

Let's try simple test:

variables:
  test_data: [ "NOT_PRESENT", "NOT_PRESENT", "PRESENT" ]
steps:
  - echo:
      name: 'echo test items'
      from: 'Items are {{ test_data }}'
  - loop:
      foreach:
        in: test_data
        do:
          - echo:
              name: 'echo test item'
              from: 'Item is {{ ITEM }}'
              register: {found: '{{ ITEM == "PRESENT" }}'}
          - echo:
              name: 'echo flag'
              from: 'Found flag is {{ found }}'
          - check:
              name: 'truefalse test'
              skip_if: '{{ not found }}'
              equals: {the: '{{ found }}', is: True}

"check" step must NOT fail test in any case
Also, first step in loop prints there some garbage: INFO:catcher:Item is t

testrail integration

would be good to have a testrail integration.
to think (need a server as well)

Run from specific step

Web api needs catcher cli to run step-by-step. With ability to run from some (perhaps failed) step and see variables input/output for every step.

Getting too many values to unpack warning

This is my yaml

steps:
# Status endpoint
- http: {get: {url: 'http://localhost:5000/api/v2/status'}}

# Cluster health endpoint
- http: {get: {url: 'http://localhost:5000/api/v2/clusterHealth'}}

# indexAndType endpoint
- http:
  post:
      url: 'http://localhost:5000/api/v2/indexAndType'
      body: {email: '[email protected]'}

I checked the yaml and it is valid and I was following the format from the docs but I still get this error WARNING:catcher:Test catcher_test.yml failed: too many values to unpack (expected 1) on the last step and it doesn't run because of it

date filters

from/to timestamp
and others

ingore the whole test

ignored: true
variables: ...
steps: ...

We should be able to ignore the test. Then test will be skipped.
Ignored should also support check-equal (to ignore tests for env var/inventory)

Is there support for Elasticsearch?

Just like there is support for kafka and postgres, I was wondering if there is support for ES because we use those heavily in our microservices and were wondering if that would be possible, because writing ES request bodies with queries and aggreagations if writing them as a simple http request in catcher can be very cumbersome so it would be nice to have something like

elasticsearch:
    search:
        index: indexName
        query:
            bool: {{filter_query_body}}

I could try sending a PR for this if you don't have the time to implement this. Also I took a look at the codebase but I can't understand how the actions for kafka and postgres are implemented.
cc @comtihon

get current inventory as a built-in variable

workaround: include env var in each inventory and name it as inventory name.

regex_replace filter

Need to add the same regex_replace filter, as Ansible has, as Jinja's replace doesn't support regexp.

Random function instead of random vars

It is better to have global functions with configurable params.
Like random_int(1,10) instead of RANDOM_INT

idea plugin

It would be nice to have an intellij idea plugin.

variables navigation
ability to run (in docker or locally)
steps autocomplete

test time limit

add ability to set max time test can run for every test.
fail test if it is running longer.

Tutorial for custom jvm-based step

Need a tutorial for creating a jvm based step.
How to pack and launch it (ideally with gradle/maven set up).
Maybe add catcher generate -l java cmd for generating skeleton for a module

Summary in case of multiple tests

In case of multiple tests to run I need a nice summary at the output end:

every test and it's status (which step failed)
how many tests run (success/fail)

variables:
   <variables block>
steps:
  <steps block>
finally:
  <steps block> with additional param (run: always/fail/pass)

sh step

need to be able to execute simple steps:

sh:
  run: "git clone my {{repo}}"

Abiliti to skip test's step based on the variable

In case some steps need to be run only locally. On some conditional steps, based on the other steps result