redhatqe / dva Goto Github PK

We'll have to update this test because it seems that Amazon started to pre-format ephemeral devices (now called Instance Store devices) to ext3, while the test in question expects the block device to be empty.

Before the test is adjusted, we'll have to explore whether this change in Instance Store devices is limited to certain regions, or possibly RHEL versions, or whether it is system-wide.

The terminate function doesn't inform about progress until an instance is terminated.
On a large input file, the user might spend some time till any progress info is shown.

https://github.com/RedHatQE/dva/blob/master/dva/connection/cache.py is not able to recognize connection to same instance therefor creating new connections for every test. We need to fix it to decrease spam and bandwidth.

It would by great to have list of instances that WERE NOT terminated during termination run - as a summary output at the end of run. User can then scroll up to check what happened. Something like:

dva -c ~/.ssh/dva.yaml terminate -i /tmp/to_terminate.yaml
 i-03f770ae in a region eu-west-1 will be terminated.
The instance i-03f770ae in eu-west-1 region may not be terminated. Modify its 'disableApiTermination' instance attribute and try again.
i-b04db61d in a region eu-west-1 will be terminated.
i-b04db61d eu-west-1 was terminated
i-7b4cb7d6 in a region eu-west-1 will be terminated.
i-7b4cb7d6 eu-west-1 was terminated

Summary of instances that WERE NOT terminated:
i-03f770ae

Or in case everything was fine:

dva -c ~/.ssh/dva.yaml terminate -i /tmp/to_terminate.yaml

i-b04db61d in a region eu-west-1 will be terminated.
i-b04db61d eu-west-1 was terminated
i-7b4cb7d6 in a region eu-west-1 will be terminated.
i-7b4cb7d6 eu-west-1 was terminated

All instances WERE terminated - congratulations, Scrooge.

Please, consider showing ssh key names in the output as that might help preventing accidentally terminating instances of our team mates sharing the same IAM role/account in AWS.
See also Issue #53

Simply as it is. Just add another dependency to dva project for gevent.

We're missing a description of a dva test case. This might be an obstacle for the new-comers...

Currently, dva configured to launch isntance types/configurations which are not supported for SAP RHEL 8.* images.
Those should be updated.

Investigate why the error below happens time to time

ERROR:dva.work.serial_process:unhandled exception: <class 'dva.cloud.base.UnknownCloudException'>(Error during instance creation: timeout in pending state)
ERROR:paramiko.transport:Socket exception: Connection reset by peer (104)

Besides text and bugzilla report it would be nice to support TCMS reports, too

the terminate function doesn't terminate any instance because of an invalid ostmeam variable value on line 42 of the file dva/work/terminate.py.
Actually, the except statement prevents the error from being shown.
When changed to except None: the error manifests itself and the function finishes without terminating:

[mkovacik@zrnko globe]$ dva -c ~/.pem/validation-staging.yaml terminate -i curl-result.yaml 
There will be 3 termination calls
Traceback (most recent call last):
  File "/usr/bin/dva", line 5, in <module>
    pkg_resources.run_script('dva==0.6', 'dva')
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 483, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1348, in run_script
    exec(script_code, namespace, namespace)
  File "/usr/lib/python2.7/site-packages/dva-0.6-py2.7.egg/EGG-INFO/scripts/dva", line 146, in <module>

  File "build/bdist.linux-x86_64/egg/aaargh/app.py", line 176, in run
  File "/usr/lib/python2.7/site-packages/dva-0.6-py2.7.egg/EGG-INFO/scripts/dva", line 112, in terminate

  File "build/bdist.linux-x86_64/egg/dva/work/terminate.py", line 42, in main
NameError: global name 'ostmeam' is not defined
[mkovacik@zrnko globe]$

RH release engineering adopts a process similar to Fedmsg. Thus we need to implement an event listener for image-built announcement messages. The listener should trigger the validation command for incoming events and post result messages back on the bus. The format of the message is still under discussion. Also Fedora Cloud Community might want to adopt our test harness in case the listener supports the Fedimg messaging as well.

When trying to complete filename using it won't show files/complete filename:

Reproducer:
dva summary -i ~/res

boto version should be specified in dependencies.

When you run 'dva summary' for hvm amis it shows instance type failure creation instead of 'skipped instance type creation as it is not supported in this region'. this could be a bit misleading:
dva summary -i res65.txt
AMI: ami-xxxxxxxxx
AMI: ami-zzzzzzzzz
!! Failed stage create_instance (cc1.4xlarge)
!! Failed stage create_instance (cg1.4xlarge)
AMI: ami-yyyyyyyyy
!! Failed stage create_instance (cc2.8xlarge)
!! Failed stage create_instance (cr1.8xlarge)
!! Failed stage create_instance (hi1.4xlarge)
!! Failed stage create_instance (cc1.4xlarge)
!! Failed stage create_instance (cg1.4xlarge)

All dva options are with dash, however --pool_size option for validate and bugzilla subcommands has underscore.

Start with pages describing how the validation input looks like and how to validate single OS release.

the docs need enhancing:

• what the tool is/does
• sample data-/hwp-file
• sample execution steps

Execution of many tests is limited only to a certain product and version, this is taken care of by an attribute called "applicable" which is defined in every test case object.

The whole test suite should remain the same between two minor versions and if there is a change of number of tests executed, it's something that needs to be reviewed. Test cases are often added for a newer minor version, but the number definitely shouldn't get lower, unless there is a very good reason behind it.

It happened to us recently that because of a wrong "applicable" attribute definition, a test case ceased to be executed and we only noticed that later.

Deliverables:

1. gather number of test cases that are executed for each product and version, and save this information somewhere in the data directory.
2. create a mechanism that would check the expected number of test cases for the given product/version that is being validated, and provide a warning if the expected number didn't match the number of tests actually executed.

For every new version, a review will have to be made to make sure all the applicable test cases are really executed and the data file will need to be updated.

successful instance termination needn't follow by retrying to terminate the same instance in other regions

... to support custom or non-public test cases...

Otherwise there are python and gevent libraries errors, which require some workaround to work on newer versions of Fedora.

For debugging and special use-cases fix propagating ec2 region endpoint setting to boto.
There already is some code in place but it doesn't seem to work

Ask dparalen for more specification...

https://bugzilla.redhat.com/show_bug.cgi?id=1245027 debuginfo-6-server channel in rh-amazon-rhui-client

For purpose of a test case error it would be nice to have the option to waive it, esp. in the bugzilla cmd, while still being able to report other tests results.

Yaml file that is result of validation does not contain information how long did it take for each test to finish. This information might be important.

dva having multiple commands and options, it would be nice to support Bash Tab-completion...

see: http://www.gevent.org/changelog.html#a1-oct-27-2016

This will need to be renamed for RHEL 7, but we'll have to think of a way not to break it for older environments. Needs investigation on RHEL 6 and other distros.

Some AMIs fail the testcase_33_userdata, but when the same steps are followed manually, the AMI and userdata actually do work. We'll need to find out why that happens, unfortunately, this test fails in a rather unpredictable manner. Maybe the test takes too long under some circumstances and the operation doesn't finish before the reboot (just a wild guess).

One AMI that failed userdata test regularly was ami-5346943d in ap-northeast-2, so that would be a good place to start the investigation with.

In both the userdata testcase and content testcases an already running yum command may result in test case failure. Need to implement yum retries in this case.

There is no sample of data file.

... can be reproduced as:

    echo "{ami:  ami-1234, ...}" | dva -l debug validate -t testcase_01_bash_history

The test-stages are separated by reboots. But if only one test stage is required (no tests in other stages) the reboot is not necessary and makes the test run longer

At the moment, performance of dva w/r data input size is limited by it's design. dva utilizes gevent pools of a single process to handle IO events. This approach is limited by both the number of file-descriptors a process may use and by not utilizing multiple processors/cores of a system to scale. Given Python's threading limitations, utilizing multiprocessing seems the way to take. Possibly, one could distribute the data load by mapping it to a worker-process-pool.

Let's assume config file is required and if user doesn't specify it with -c option it should give user-friendly output than awful traceback mess.

For details see: PnT JIRA (RHELDST-1428) plus Jerome's source of truth doc.

RHEL releases share the same packages. ATM, there are lots of duplication in data/packages files. Maybe there is a way to avoid it and, for example, have a nested/included structure. Storing it here, like an idea for the future investigation.

Dva should have an option to run tests over existing amis

For debugging purposes --- such as debugging a test case against a manually fine-tuned node --- implement instantiation stage skipping.
Actually, there is already one skip in place: --keepalive which avoids terminating instances.

For safety reasons, please consider implementing a -n/--no-action switch that would prevent dva terminate from actually terminating the instances, instead, just make it display what would have happened.

if one presses CTRL+C to terminate the terminate function, the CTRL+C exception gets caught instead and nothing happens really.

add terminate function into tab completion script

dva -c /etc/validation.yaml validate -i ~/git_projects/dva/data.yaml -o ~/validace.yaml -G content --parallel-tests=10

Traceback (most recent call last):
File "/usr/bin/dva", line 5, in dva -c /etc/validation.yaml validate -i ~/git_projects/dva/data.yaml -o ~/validace.yaml -G content --parallel-tests=10
Traceback (most recent call last):
File "/usr/bin/dva", line 5, in
pkg_resources.run_script('dva==0.5', 'dva')
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 492, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1357, in run_script
exec_(script_code, namespace, namespace)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 47, in exec_
exec("""exec code in globs, locs""")
File "", line 1, in
File "/usr/lib/python2.7/site-packages/dva-0.5-py2.7.egg/EGG-INFO/scripts/dva", line 133, in

File "build/bdist.linux-x86_64/egg/aaargh/app.py", line 176, in run
File "/usr/lib/python2.7/site-packages/dva-0.5-py2.7.egg/EGG-INFO/scripts/dva", line 79, in validate

File "build/bdist.linux-x86_64/egg/dva/work/data_parallel_main.py", line 38, in main
File "build/bdist.linux-x86_64/egg/dva/work/data.py", line 177, in load
File "build/bdist.linux-x86_64/egg/dva/work/data.py", line 102, in record_cloud_config
dva.work.data.ConfigError: config /etc/validation.yaml: [Errno 2] No such file or directory: '/etc/validation.yaml'
pkg_resources.run_script('dva==0.5', 'dva')
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 492, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1357, in run_script
exec_(script_code, namespace, namespace)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 47, in exec_
exec("""exec code in globs, locs""")
File "", line 1, in
File "/usr/lib/python2.7/site-packages/dva-0.5-py2.7.egg/EGG-INFO/scripts/dva", line 133, in

File "build/bdist.linux-x86_64/egg/aaargh/app.py", line 176, in run
File "/usr/lib/python2.7/site-packages/dva-0.5-py2.7.egg/EGG-INFO/scripts/dva", line 79, in validate

File "build/bdist.linux-x86_64/egg/dva/work/data_parallel_main.py", line 38, in main
File "build/bdist.linux-x86_64/egg/dva/work/data.py", line 177, in load
File "build/bdist.linux-x86_64/egg/dva/work/data.py", line 102, in record_cloud_config
dva.work.data.ConfigError: config /etc/validation.yaml: [Errno 2] No such file or directory: '/etc/validation.yaml'

I believe it's ok to show only line: No such file or directory: '/etc/validation.yaml'

We'd like to move away from Bugzilla results rendering. Ideally, we'd keep the results in Jenkins.
This would require some jobs designing for Jenkins wr. multiple products and versions being validated.
However, it should be possible to implement a generic xunit translator, maybe based on the summary command.

For example if you try to terminate instance that has api termination forbidden, you will get operation not permitted xml:

Exception EC2ResponseError: 400 Bad Request

OperationNotPermittedThe instance 'i-fed7dabc' may not be terminated. Modify its 'disableApiTermination' instance attribute and try again.0cb5104d

the sriov net support should be a default flag when registering an HVM ami (when RHEL6.7 is out)
Please, verify&close the bug 1094996 and bug 1103921 once it is out; see also testcase_63_sriov.

When I run the bugzilla command, I get this warning:

/usr/local/lib/python2.7/dist-packages/urllib3/util/ssl_.py:133: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecurePlatformWarning