gluster / glusto-tests Goto Github PK

Functional test framework for glusterfs.

License: GNU General Public License v3.0

Python 99.79% Shell 0.12% Dockerfile 0.02% Jinja 0.07%

glusto-tests's Introduction

Glusto Tests

glusto-tests repo contains automated testcases for testing gluster software. It provides the Libraries/Modules necessary for automating the gluster tests. Latest Code for this repo is managed on review.gluster.org

The Libraries/Modules/Tests in glusto-tests are written using the glusto framework. TestCases in glusto-tests can we written/run using standard PyUnit, PyTest or Nose methodologies as supported by glusto framework.

Refer the glusto-doc for info on glusto framework. Issues need to be filled against the Github repo.

To automate/run glusto-tests we need to install the following packages:

glusto
glustolibs-gluster
glustolibs-io
glustolibs-misc
gdeploy

How to install glusto:

One can use either of the two methods.

using pip

# pip install --upgrade git+git://github.com/loadtheaccumulator/glusto.git

using git

# git clone https://github.com/loadtheaccumulator/glusto.git
# cd glusto
# python setup.py install

For more info refer the docs.

How to install the glustolibs-gluster, glustolibs-io and glustolibs-misc libraries:

# git clone http://review.gluster.org/glusto-tests
# cd glusto-tests/glustolibs-gluster
# python setup.py install
# cd ../../glusto-tests/glustolibs-io
# python setup.py install
# cd ../../glusto-tests/glustolibs-misc
# python setup.py install

How to install gdeploy:

Install latest version of gdeploy from the following link.

To install glusto-tests dependencies:

python-docx, sh and numpy has to be installed to run IO and validate it on client node.

To install run :

      # curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
      # python get-pip.py
      # pip install --pre python-docx
      # pip install numpy
      # pip install sh

arequal needs to be installed on all servers and clients.

To install download the below repo into /etc/yum.repos.d/

  # wget https://copr.fedorainfracloud.org/coprs/nigelbabu/arequal/repo/epel-7/nigelbabu-arequal-epel-7.repo
  # yum install arequal

Pre-requisites to run glusto-tests:

Make sure glusto, glusto-tests are installed on the node from where you would want to run the gluster tests.
Running Gluster Cluster( 6 Servers and 2 Clients )
Gluster client packages should be installed on Clients
Setup passwordless ssh from the glusto-tests management node to all.
Install glusto-tests dependencies on servers and clients.
Crefi should be installed on all the clients.
```
$ pip install crefi
$ pip install pyxattr
```
Setup bricks on all servers:
- To create bricks refer to doc OR Run gdeploy as shown below.
  1. Edit the gdeploy_sample_config.conf present in examples as shown below and also configure passwordless ssh to all servers:
```
[hosts]
server-vm1
server-vm2
server-vm3
server-vm4
server-vm5
server-vm6

[backend-setup]
devices
vgs
pools
lvs
mountpoints
```
  Note: For more details you can view a sample config file avaliable at /usr/share/doc/gdeploy/examples/gluster.conf.sample which will be installed with gdeploy.
  1. Run gdeploy using the below command:
```
gdeploy -c gdeploy_sample_config.conf
```

Note:

- To run cifs protocol:
	1.CIFS packages need to be installed on the server
	2.Samba services need to be ACTIVE
	3.cifs-utils need to be installed on the client
- To run nfs protocol, nfs packages must be installed on server and client

For more info how to run glusto-tests from the scratch including creating OS, server, etc.. refer link

How to run the test case:

Update the information about the servers, clients, servers_info, client_info on the config_file, this information is enough to run all test cases. But if you need to override the default values of volumes, mount.. etc which is defined in gluster_base_class then use config and update the information accordingly. Refer the following for more info link.
glusto-tests are run using the glusto command available after installing the glusto framework. The various options to run tests as provided by glusto framework: PyUnit Tests, PyTest Tests, Nose Tests. The most common used is Pytest.
- Running PyTest Tests
  - To run all tests that are marked with tag 'bvt':
```
  # glusto -c config.yml --pytest='-v -x tests -m bvt'
```
  - To run all tests that are under bvt folder:
```
  # glusto -c config.yml --pytest='-v -s bvt/'
```
  - To run a single test case:
```
  # glusto -c config.yml --pytest='-v -s -k test_demo1'
```

For more info about running tests on PyUnit, Pytest and Nose Tests, refer the docs.

glusto-tests can also be executed using tox:

   # tox -e functional -- glusto -c 'config.yml' --pytest='-v -s -k test_demo1'

glusto-tests can also be executed with python3 using tox:

   # tox -e functional3 -- glusto -c 'config.yml' --pytest='-v -s -k test_demo1'

NOTE:

Please note that glusto-tests is not completely compatible with python3.
You would not need to install the glusto or glusto-tests libraries while running it using tox. For more info about tox refer the docs.

Writing tests in glusto-tests:

tests directory in glusto-tests contain testcases. Testcases are written as component wise. Testcases name and file name should should start with test_.
TestCases in glusto-tests can be written using standard PyUnit, PyTest or Nose methodologies as supported by glusto framework.
- One can follow the PyUnit docs to write PyUnit tests, or PyTest docs to write PyTest tests, or Nose docs to write Nose tests.

While writing testcases or libraries follow:

Please follow the PEP008 style-guide.

Makes sure all the pylint and pyflakes error are fixed. For example

  - C0326: Exactly one space required around assignment
  - C0111: Missing module doc-string (missing-doc string)
  - W: 50: Too long line

For more information on pylint and on pyflakes. We can check for pyflakes and pylint errors:

 # flake8 <test_script.py>
 or
 # flake8 <path_to_directory>
 # pylint -j 4 --rcfile=/glusto-tests/.pylintrc <test_script.py>

Optimize the code as much as possible. Eliminate the repetitive steps, write it has separate function.
Use proper python standards on returning values. This style guide is a list of do's and don’ts for Python programs.

Add docstring to every function you write For example: This is an example of a module level function

     def module(param1, param2):
     """
     Explain what the module function does in breif

     Args:
         param1: The first parameter.
         param2: The second parameter.

     Returns:
         The return value of the function.
     """

Make sure the log messages are grammatically correct and have no spelling mistakes.

Comment every step of the test case/libraries, log the test step, test result, failure and success. For example:

def test_peer_status(self):
    # peer status from mnode
    g.log.info("Get peer status from node %s", self.mnode)
    ret, out, err = peer_status(self.mnode)
    self.assertEqual(ret, 0, "Failed to get peer status from node %s: %s" % (self.mnode, err))
    g.log.info("Successfully got peer status from node %s:\n%s", self.mnode, out)

Don't not use print statements in test-cases/libraries because prints statements are not captured in log files. Use logger functions to dump messages into log file.

For more info how to write testcases developing-guide

Logging:

Log file name and Log level can be passed as argument to glusto command while running the glusto-tests. For example:

# glusto -c 'config.yml' -l /tmp/glustotests_bvt.log --log-level DEBUG --pytest='-v -x tests -m bvt'

One can configure log files, log levels in the testcases as well. For details on how to use glusto framework for configuring logs in tests Refer the following docs.

Default log location is: /tmp/glustomain.log

Note: When using glusto via the Python Interactive Interpreter, the default log location is /tmp/glusto.log.

License

GPLv3

glusto-tests's People

Contributors

Stargazers

Watchers

glusto-tests's Issues

dht/test_induce_holes_in_layout_by_removebrick_force_then_lookup.py::test_induce_holes_then_lookup

In funcation test_induce_holes_then_lookup, according to the test case the code is not correct ret = validate_files_in_dir(self.clients[0], self.mounts[0].mountpoint, test_type=LAYOUT_IS_COMPLETE, file_type=FILETYPE_DIRS) ,i think can change to ret = validate_files_in_dir(self.mnode, mount_point + dirpath, test_type=LAYOUT_IS_COMPLETE, file_type=FILETYPE_DIRS). The unit-test always self.assertTrue(ret, "LAYOUT_IS_COMPLETE: FAILED") if we don't change it.

Can the 'framework' be hosted in this repo itself...

Problem statement

I am a new user, and while installing the test framework, going through the readme, I see that the pip install and git clone talks about a repo hosted by another user (not org). While it is fine to have the dependency on any repo / project, as a common user, it doesn't give me confidence to install from a user's repo.

What is ideal?

I would like the repo to be backed by an org (like gluster.org), hosted in their github org, which can then have an infra team for managing, would follow more open guidelines about the future of the project. Ideally the user can become maintainer, and transfer the ownership to gluster org, or gluster.org can fork the repo, and keep the repo up-to-date with enhancements, and security fixes.

@loadtheaccumulator @nigelbabu @amye @mykaul @vavuthu @Akarsha-rai

Move from gerrit to github workflow

Process of moving from gerrit to github:

Update CONTRIBUTING.md to github workflow.
Make changes in CentOS-CI and Gluster Build system to point to github.
Post Feb 12, 2021 we wil have to stop sending patches to review.gluster.org instead everyone would be required to send, their patches to https://github.com/gluster/glusto-tests.
Merge or abandon all the existing patches present in the gerrit server.

glustolibs-io/shared_files/scripts/generate_io.py function always return true

glustolibs-io/shared_files/scripts/generate_io.py::check_if_percent_to_fill_or_timeout_is_met return code always return true.

trouble removing custom attribute

Need a function to remove custom file attributes. In glusto-libs.gluster.glusterfile, the function set_fattr doesn't allow user to pass options for the setfattr command. To remove the custom attribute, the option '-x' is required with the setfattr command.

have an option to change number of volumes based on input or config file

We need to be able to change the number of volumes created/undergo test for a specific testcase.
With gluster support for containers, we must enable glusto to start creating multiple volumes for each testcases.
Eg: in brickmux, we need to create multiple volumes for each testcase. Also we must not be hardcoding these number of volumes. Eg: I can tell glusto to create 10 or even 200 vols, and it should be able to(if the environment permits based on h/w limitations)
So we need an option or variable which can go into the config file which asks for number of vols to be created.
We must look forward to enabling glusto for scale and not just for functional validation.
this helps in non-functional testing like checking consistency on scale.

The 'warn' method is deprecated, use 'warning' instead

Running with Python 3 part 4 branch:

============================================================================================================== warnings summary ==============================================================================================================
functional/dht/test_access_file.py::TestFileAccessSubvolDown_cplex_distributed-replicated_glusterfs::test_file_access
  /usr/local/lib/python3.6/site-packages/glustolibs_gluster-0.22-py3.6.egg/glustolibs/gluster/gluster_base_class.py:1057: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
    g.log.warn(ret)

-- Docs: https://docs.pytest.org/en/stable/warnings.html

[RFE] Create A Centralized Logs Tar

As logs are one of the primary tools used for analysis and debugging, adding an option which will fetch all gluster logs from all clients and servers and create a TAR of them at a given location (should be configurable, with default being the local machine) would be welcomed.

Glusto removes all mountpoint

When running tests, on cleanup glusto performs 'rm -rf /mnt/', which deletes ALL mountpoints.
E.g.:
2020-08-09 12:16:21,591 ERROR (scratch_cleanup) failed to unmount/already unmounted barak-pc
2020-08-09 12:16:21,591 INFO (run) root@barak-pc (cp): rm -rf /mnt/

Glusto should only remove mountpoints used by it for running tests, and not delete data which is unrelated to it.

[RFE] Enable stale bot work flow for glusto-tests

Problem:
We have received a lot of issues in the past which have been abandoned for quite sometime and were not closed every after a couple of years, one such example is [1] which was open for almost 2 years.

Hence to take care of such issues we have decided to add stale bot which will close stale issues automatically.

What we are planning to do?
The process which is to be followed will be as below:

Once we get a new issue by default we will be adding ToBeTriaged label.
Once the issue is triaged and is to be fixed it will be assigned to a contributor and the label ToBeFixed will be added.
If he issue is not be be fixed then we will be removing the ToBeTriaged label after this the bot will wait for 90 days for any comment.
Once 90 days are complete stale bot will drop a last comment and will wait for 14 days post which the bot will automatically close the issue with wontfix label.

More about stale bot is available at [2].

Links:
[1] #15
[2] https://github.com/marketplace/stale

correct logging must be reflected for testcase failure

while analysing one of the test runs, one test case failed saying it had hit the timeout of 20min, but that was not the case.
Later it was seen that this case failed, because of heal info xml showing one brick down.
We need to wait for more time and recheck for heal info to reflect the changes, instead of failing them at the first failure itself.

I was looking into logs of the failed case "tests.functional.bvt.test_cvt.TestGlusterHealSanity_cplex_distributed-replicated_glusterfs.test_self_heal_when_io_in_progress"
in http://jenkins-rhs.lab.eng.blr.redhat.com:8080/view/Auto%20RHEL%206.9/job/auto-RHGS_Downstream_BVT_RHEL_6_9_RHGS_3_4_brew/ws/glusto_3.log
It failed with "AssertionError: Self heal didn't complete even after waiting for 20 minutes. 20 minutes is too much a time for current test workload"

However, when I looked into the details, I see that the whole testcase was run only for 2 minutes.

Remove shared storage volume in delete_nfs_ganesha_cluster method

Description of problem:
Failing to remove shared storage volume in cleanup leads to improper cleanup of entire cluster. when ran with nfs enabled.

Exact glusto-tests log where the issue was observed:

Testcase or library:
delete_nfs_ganesha_cluster method
teardown_nfs_ganesha_cluster

How reproducible:
Everytime

Additional info:

Sequence of test execution in framework does not make sense

In the current state of things, many gluster-related operations (e.g. peer probe, volume creation, mount) are being performed by complex tests, before these operations are actually being tested by the basic (unit) tests.

This means that if a bug exist in these basic operation, instead of having a basic (unit) test fail, a complex test will fail, which makes the basic (unit) tests pretty useless.

The proper sequence of events is for the framework to start by running tests for the most basic operations and then progress to running complex tests which involves multiple, more basic operations.

improve functionality by accepting variable as well

In glusto test cases, we are supposed to pass list of arguments. When we want to pass a single argument if we pass it as a variable instead of list, it is handling the variable as a list.
My suggestion is we can improve the function in such a way that,
it also accepts a variable so that when we want to pass a single argument we can pass
variable instead of list.

We can improve the functionality with the help of "instanceof" in python. We can check whether
the argument is list or not, if not we can convert it into a list.

Simplify Glusto-test setup and usage

The setup and usage of Glusto-test are very not trivial - It's very hard to setup the framework on local machines and as a result a shared setup was created, however there are clear limitation for working with a shared setup.

As Gluster itself is extremely easy to deploy locally, there's no reason why the test framework deployment should be so complicated.
Ideally, The test framework should not perform validation of the configuration and instead pass the info to Gluster as-is and let Gluster try to operate in the provided environment. If it's successful, it means the config is valid, if it's unsuccessful, it means the configuration is wrong.

Efforts should be to modify the framework so setup and usage is no more complex than Gluster itself.

[RFE] Improve data verification mechanism

A recent discussion brought into light a concern regarding the glusto-test framework's ability to perform data correctness verification on large data sets.
Currently data verification is performed using arequal checksum, which was observed to take an unreasonable amount of time while verifying large data sets.
An alternative mechanism may be used, which could speed up the data verification of large data sets and reduce the CI overall.

Glusto tests do not support running on nightly Gluster upstream bits

    def get_gluster_version(host):
        """Checks the gluster version on the nodes
   
        Args:
            host(str): IP of the host whose gluster version has to be checked.
   
        Returns:
            (float): The gluster version value.
        """
        command = 'gluster --version'
        _, out, _ = g.run(host, command)
        g.log.info("The Gluster verion of the cluster under test is %s",
                   out)
>       return float(out.split(' ')[1])
E       ValueError: could not convert string to float: '20200719.9334a8d\nRepository

afr/test_gfid_assignment_on_lookup.py when access dir can't heal dir

afr/test_gfid_assignment_on_lookup.py in gluster‘s version 6 should set afr heal options . "metadata-self-heal": "on", "entry-self-heal": "on", "data-self-heal": "on".

Change NFS-Ganesha function names to unique

Description of problem:
NFS-Ganesha multiple cases are mapped to the same function name test_root_squash_enable.

Testcase or library:
https://github.com/srivickynesh/glusto-tests/blob/master/tests/functional/nfs_ganesha/root-squash/test_nfs_ganesha_root_squash.py
https://github.com/srivickynesh/glusto-tests/blob/master/tests/functional/nfs_ganesha/root-squash/test_nfs_ganesha_rootsquash_multiple_client.py
https://github.com/srivickynesh/glusto-tests/blob/master/tests/functional/nfs_ganesha/root-squash/test_root_squash_with_glusterd_restart.py
https://github.com/srivickynesh/glusto-tests/blob/master/tests/functional/nfs_ganesha/root-squash/test_root_squash_with_volume_restart.py

Additional info:
test_root_squash_enable
test_rootsquash_enable
test_root_squash_enable
test_root_squash_enable

ERROR level logs should reflect only real errors

There are several cases in which logs are logged with ERROR level, but don't necessarily reflect real errors.
E.g:

2020-08-09 12:16:22,630 INFO (run) root@vm1 (cp): gluster volume info | egrep "^Brick[0-9]+" | grep -v "ss_brick"
2020-08-09 12:16:22,630 DEBUG (_get_ssh_connection) Retrieved connection from cache: root@vm1
2020-08-09 12:16:22,743 INFO (_log_results) �[34;1mRETCODE (root@vm1): 1�[0m
2020-08-09 12:16:22,744 INFO (_log_results) �[31;1mSTDERR (root@vm1)...
No volumes present
�[0m
2020-08-09 12:16:22,744 ERROR (get_servers_used_bricks_dict) error in getting bricklist using gluster v info
2020-08-09 12:16:22,744 INFO (run) root@vm1 (cp): gluster volume info testvol_distributed --xml
2020-08-09 12:16:22,744 DEBUG (_get_ssh_connection) Retrieved connection from cache: root@vm1
2020-08-09 12:16:22,917 INFO (_log_results) �[34;1mRETCODE (root@vm1): 0�[0m
2020-08-09 12:16:22,918 DEBUG (_log_results) �[47;30;1mSTDOUT (root@vm1)...

This log doesn't terminate the run, but will actually trigger volume creation (some lines later from the same log):

2020-08-09 12:16:22,918 INFO (run) root@vm1 (cp): gluster volume create testvol_distributed        vm1:/mnt/disk2/testvol_distributed_brick0 vm2:/mnt/disk2/testvol_distributed_brick1 vm3:/mnt/disk2/testvol_distributed_brick2 --mode=script force
...
2020-08-09 12:16:26,261 INFO (run) root@vm1 (cp): gluster vol status testvol_distributed  
2020-08-09 12:16:26,261 DEBUG (_get_ssh_connection) Retrieved connection from cache: root@vm1
2020-08-09 12:16:26,373 INFO (_log_results) �[34;1mRETCODE (root@vm1): 0�[0m
2020-08-09 12:16:26,373 INFO (_log_results) �[47;30;1mSTDOUT (root@vm1)...
Status of volume: testvol_distributed
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick vm1:/mnt/disk2/testvol_distributed_br
ick0                                        49152     0          Y       19881
Brick vm2:/mnt/disk2/testvol_distributed_br
ick1                                        49152     0          Y       15478
Brick vm3:/mnt/disk2/testvol_distributed_br
ick2                                        49152     0          Y       15516
 
Task Status of Volume testvol_distributed
------------------------------------------------------------------------------
There are no active volume tasks

Error logs should only reflect failed operation which causes termination of the current test that's being run or the framework itself.

Reduce runtime and enable concurrent test execution

Based on discussions regarding the performance of Glusto-test it was noted that it takes several days to complete a full regression run. This performance does not meet the development team's requirement of completing a full run in no more than a day, so functional regression could be tested on a daily basis.

In addition, the framework does not allow parallel execution of tests. Not only this limitation undoubtedly hinders performance but it also adversely affect the comprehensiveness of the testing itself, as no concurrent testing is being conducted.

The framework needs to be modified in such a way so performance meets the developments team requirements and in addition allows parallel execution of tests (both on a single setup and in a containerized environment).

Parallelization should be done in 2 ways:

Run different tests in parallel
For the same tests, run on different volume types in parallel

Some tests cannot be parallelized as they perform operation which disrupts the overall system (such as kill glusterd), these tests should be marked appropriatly as non-parallelizeable and be run sequentially.

snapshot suite is faling with "list index out of range"

When we ran snapshot suite, its failing with below error.

___ ERROR collecting tests/functional/snapshot/test_validate_snapshot_256.py ___
tests/functional/snapshot/test_validate_snapshot_256.py:33: in
class SanpCreate256(GlusterBaseClass):
/usr/lib/python2.7/site-packages/glusto/carteplex.py:93: in call
updated_axis_names.append(self.axis_names[i])
E IndexError: list index out of range

Cleanup Issue when tearing down the cluster

Description of problem:
_ ERROR at teardown of TestAddIdenticalBrick_cplex_distributed_glusterfs.test_add_identical_brick _

cls = <class 'functional.glusterd.test_add_identical_brick_new_node.TestAddIdenticalBrick_cplex_distributed_glusterfs'>

@classmethod
def doClassCleanups(cls):
    if (GlusterBaseClass.error_or_failure_exists or
               cls._is_error_or_failure_exists()):

E TypeError: _is_error_or_failure_exists() missing 1 required positional argument: 'self'

cls = <class 'functional.glusterd.test_add_identical_brick_new_node.TestAddIdenticalBrick_cplex_distributed_glusterfs'>

/root/toolbox/venv/gpy3/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/gluster_base_class.py:1098: TypeError

Library needs fix:
doCleanups
doClassCleanups

How reproducible:
Everytime

test_expanding_volume_when_io_in_progress fails intermittently

See:

https://ci.centos.org/view/Gluster/job/gluster_glusto/311/
https://ci.centos.org/view/Gluster/job/gluster_glusto/309/

Please let's figure out what's causing this and solve this sooner than later. Intermittent failures are unavoidable when dealing with end-to-end testing, so this is most likely due to a timeout of some sort.

[RFE] Add option to control Glustr's log level through gluso-test config file

Currently, the "log_level" field in the config file only applies to the test framework itself.
An option needs to be added which will allow to run Gluster with the specified log level.
E.g:
glusto_log_level - The log level in which glusto will run
gluster_log_level - The log level in which gluster will run

dispersed_glusterfs and replicated_glusterfs _get_hashrange error when do dht's some test case

for example, do command " glusto -c /root/glusto-tests/tests/test.yml --pytest='-v -s dht/test_volume_start_stop_while_rebalance_in_progress.py::RebalanceValidation_cplex_dispersed_glusterfs::test_volume_start_stop_while_rebalance_is_in_progress' " will get hash error. The reason is that the dispersed and replicated volume's mountpoint xattr don't have hashrange when we do command getfattr -dm . -e hex mountpoint.

Add issue template for github

Problem:
There have been a couple of instances where we have received issues which don't have a clear description,
making it difficult for glusto-tests developers to fix it. One such example is [1].

Solution:
Adding a ISSUE_TEMPLATE for github issues so that the reporter of the issue follows a given format for the issue.

Links:
[1] #19

glusto-tests/tests/functional/dht/test_negative_exercise_add_brick_command.py is failing with ImportError

It looks like test case is using "form_bricks_list_to_add_brick" function which is not present/merged in glustolibs.gluster.volume_libs

Error:

ImportError while importing test module 'glusto-tests/tests/functional/dht/test_negative_exercise_add_brick_command.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/functional/dht/test_negative_exercise_add_brick_command.py:22: in
from glustolibs.gluster.volume_libs import (form_bricks_list_to_add_brick,
E ImportError: cannot import name form_bricks_list_to_add_brick

allow even disperse_data count as an input parameter for creating disperse volumes

allow even disperse_data count as an input parameter for creating disperse volumes
currently only disperse_count and redundancy_count are to be entered as input param
however, enduser can even mention data_count, and hence we need to allow that too

Tests fail with ImportError: cannot import name get_subvols

11:01:12 ImportError while importing test module '/root/glusto-tests/tests/functional/bvt/test_basic.py'.
11:01:12 Hint: make sure your test modules/packages have valid Python names.
11:01:12 Traceback:
11:01:12 functional/bvt/test_basic.py:22: in <module>
11:01:12     from glustolibs.gluster.gluster_base_class import GlusterBaseClass
11:01:12 ../glustolibs-gluster/glustolibs/gluster/gluster_base_class.py:31: in <module>
11:01:12     from glustolibs.gluster.volume_libs import (setup_volume, cleanup_volume,
11:01:12 ../glustolibs-gluster/glustolibs/gluster/volume_libs.py:40: in <module>
11:01:12     from glustolibs.gluster.brick_libs import are_bricks_online, get_all_bricks
11:01:12 ../glustolibs-gluster/glustolibs/gluster/brick_libs.py:23: in <module>
11:01:12     from glustolibs.gluster.volume_libs import (get_subvols, is_tiered_volume,
11:01:12 E   ImportError: cannot import name get_subvols

See: https://ci.centos.org/view/Gluster/job/gluster_glusto/276/console

Glusto stuck in a loop

Scenario:
1 - Set up a couple of nodes (e.g. 3)
2 - start gluster on all of these node ('glusterd')
3 - Run glusto tests

Result:
The test framework is stuck in an infinite loop trying to restart the glustrd process on the nodes.

Console output from the node running the tests:
glusto -c "/root/glusto-tests/tests/gluster_basic_config.yml" --pytest='/root/glusto-tests/tests/functional/dht/test_dht_create_dir.py'
Log glustomain created as /tmp/glustomain.log with log level DEBUG
Starting glusto via main()
clients: [barak-pc]
clients_info:
barak-pc: {host: barak-pc}
gluster:
volume_create_force: true
volume_types:
arbiter: {arbiter_count: 1, replica_count: 3, transport: tcp, type: arbiter}
dispersed: {disperse_count: 6, redundancy_count: 2, transport: tcp, type: dispersed}
distributed: {dist_count: 3, transport: tcp, type: distributed}
distributed-arbiter: {arbiter_count: 1, dist_count: 2, replica_count: 3, transport: tcp,
type: distributed-arbiter}
distributed-dispersed: {disperse_count: 6, dist_count: 2, redundancy_count: 2,
transport: tcp, type: distributed-dispersed}
distributed-replicated: {dist_count: 1, replica_count: 3, transport: tcp, type: distributed-replicated}
replicated: {arbiter_count: 1, replica_count: 3, transport: tcp, type: replicated}
log_file: /var/log/tests/gluster_tests.log
log_level: DEBUG
servers: [vm1, vm2, vm3]
servers_info:
vm1:
brick_root: /mnt/disk2
devices: [/dev/vdb]
host: vm1
vm2:
brick_root: /mnt/disk2
devices: [/dev/vdb]
host: vm2
vm3:
brick_root: /mnt/disk2
devices: [/dev/vdb]
host: vm3

pytest: /root/glusto-tests/tests/functional/dht/test_dht_create_dir.py
=================================================================================================================================================================================== test session starts ====================================================================================================================================================================================
platform linux -- Python 3.8.0, pytest-6.0.0rc1, py-1.9.0, pluggy-0.13.1
rootdir: /root/glusto-tests/tests/functional/dht
collected 2 items

../root/glusto-tests/tests/functional/dht/test_dht_create_dir.py ^C

===================================================================================================================================================================================== warnings summary =====================================================================================================================================================================================
test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_directory
test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_directory
test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_directory
test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_directory
test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_directory
test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_directory
/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/peer_ops.py:293: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_directory
test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_directory
test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_directory
test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_directory
test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_directory
test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_directory
/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/peer_ops.py:296: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

test_dht_create_dir.py: 168 warnings
/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/volume_ops.py:707: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

test_dht_create_dir.py: 168 warnings
/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/volume_ops.py:713: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

test_dht_create_dir.py: 168 warnings
/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/volume_ops.py:716: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

test_dht_create_dir.py: 504 warnings
/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/volume_ops.py:719: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

test_dht_create_dir.py: 504 warnings
/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/volume_ops.py:727: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

test_dht_create_dir.py: 80 warnings
/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/volume_ops.py:820: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

-- Docs: https://docs.pytest.org/en/stable/warnings.html
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
/usr/lib64/python3.8/selectors.py:415: KeyboardInterrupt
(to show a full traceback on KeyboardInterrupt use --full-trace)
============================================================================================================================================================================ 1607 warnings in 158.15s (0:02:38) ============================================================================================================================================================================
Ending glusto via main()

Glusto log is also attached.
glustomain.log

Test framework fault tolerance needs to be improved

As recent patch that was submitted has shown, the glusto-test framework is not sufficiently fault-tolerant.

The patch that was submitted for review caused the test to run an unreasonably long operation, which caused the test framework to hang in wait for operation to finish.
As a test framework is used to test and verify ongoing work, it is to be expected that faulty patches will be submitted to the system, and the system should handle these gracefully. A testing framework should possess a very high fault-tolerance.
The case that is described here poses a security risk as it opens the way for a complete shutdown of the CI system if a large number of long-running tests are submitted which will cause the test machines to hang (Similar to a DDoS attack).

Ideally, the test framework should set a timeout for each test and if a test continues to run longer than it's allocated time, the test framework will forcefully terminate the long-running test, end the regression gracefully and mark the run as a failure, without hanging.

dht/test_dht_create_dir.py::test_create_directory code is not correct

In dht/test_dht_create_dir.py::test_create_directory::134 according to the unit test the code gfid = get_fattr(brick_tuple[0], m_point + '/direc', 'trusted.gfid') is not correct, so we can change it to gfid = get_fattr(brick_tuple[0], brick_path + '/' + direc, 'trusted.gfid').

Reset of volume options after a TC ends

Description of problem: Current flow doesn't reset the cluster options in gluster after the TC ends which causes certain TC's to fail.

Operating system: ALL

Python Version: ALL

Terminal dump of the issue observed::

Exact glusto-tests log where the issue was observed::

Setup info:

Number of nodes:

Number of clients: NA

Number of servers: NA

Testcase or library: This affects all the TCs

How reproducible: For instance take the TC wherein the default value of Quorum value is being verified. If the Quorum option is set in a previous TC then it'll be carried on to the current TC.

Additional info:

Glust-test framework cleanup

Currently we have huge amounts of code duplication in the framework - In many tests generic and basic operations such as lookup are defined as part of the code of the test itself.

The proper way to define generic and common functionality is by having an "ops" library in which these operations are defined. It does seem like glsto-test has such libraries, but they are incomplete, and instead of adding functionality to these libraries, all the functionality is being defined in the tests themselves.

This issue is created for the following:

Add the missing functionality to the common "ops" libraries.
Clean up existing tests so they will use the libraries instead of implementing the functionality themselves.
Currently error handling for various operations is done in the scope of the test, this is very problematic as error handling is not
consistent (error message differ) and can even be forgotten (e.g.https://review.gluster.org/#/c/glusto-tests/+/25223/). Error handling for all library functionality should be done in the scope of the library and not in the test.

Glusto fails to run tests due to error in getfattr operation

Running tests on Centos8 machines (client, servers and the machine running the tests - All Centos8), results in the following errors:

glusto -c "/root/glusto-tests/tests/gluster_basic_config.yml" --pytest='/root/glusto-tests/tests/functional/dht/test_dht_create_dir.py'
Log glustomain created as /tmp/glustomain.log with log level DEBUG
Starting glusto via main()
clients: [barak-pc]
clients_info:
barak-pc: {host: barak-pc}
gluster:
volume_create_force: true
volume_types:
arbiter: {arbiter_count: 1, replica_count: 3, transport: tcp, type: arbiter}
dispersed: {disperse_count: 6, redundancy_count: 2, transport: tcp, type: dispersed}
distributed: {dist_count: 3, transport: tcp, type: distributed}
distributed-arbiter: {arbiter_count: 1, dist_count: 2, replica_count: 3, transport: tcp,
type: distributed-arbiter}
distributed-dispersed: {disperse_count: 6, dist_count: 2, redundancy_count: 2,
transport: tcp, type: distributed-dispersed}
distributed-replicated: {dist_count: 1, replica_count: 3, transport: tcp, type: distributed-replicated}
replicated: {arbiter_count: 1, replica_count: 3, transport: tcp, type: replicated}
log_file: /var/log/tests/gluster_tests.log
log_level: DEBUG
servers: [vm1, vm2, vm3]
servers_info:
vm1:
brick_root: /mnt/disk2
devices: [/dev/vdb]
host: vm1
vm2:
brick_root: /mnt/disk2
devices: [/dev/vdb]
host: vm2
vm3:
brick_root: /mnt/disk2
devices: [/dev/vdb]
host: vm3

../root/glusto-tests/tests/functional/dht/test_dht_create_dir.py F. [100%]

self = <functional.dht.test_dht_create_dir.TestDhtClass_cplex_distributed_glusterfs testMethod=test_create_directory>

def test_create_directory(self):

    m_point = self.mounts[0].mountpoint
    command = 'mkdir -p ' + m_point + '/root_dir/test_dir{1..3}'
    ret, _, _ = g.run(self.mounts[0].client_system, command)
    self.assertEqual(ret, 0, "directory creation failed on %s"
                     % self.mounts[0].mountpoint)
    command = 'ls ' + m_point + '/root_dir'
    ret, out, _ = g.run(self.mounts[0].client_system, command)
    self.assertEqual(ret, 0, "ls failed on parent directory:root_dir")
    g.log.info("ls on parent directory: successful")

    command = 'touch ' + m_point + \
        '/root_dir/test_file{1..5} ' + m_point + \
        '/root_dir/test_dir{1..3}/test_file{1..5}'
    ret, _, _ = g.run(self.mounts[0].client_system, command)
    self.assertEqual(ret, 0, "File creation: failed")
    command = 'ls ' + m_point + '/root_dir'
    ret, out, _ = g.run(self.mounts[0].client_system, command)
    self.assertEqual(ret, 0, "Failed to list the created directories")
    list_of_files_and_dirs = out.split('\n')
    flag = True
    for x_count in range(3):
        dir_name = 'test_dir%d' % (x_count+1)
        if dir_name not in list_of_files_and_dirs:
            flag = False
    for x_count in range(5):
        file_name = 'test_file%d' % (x_count+1)
        if file_name not in list_of_files_and_dirs:
            flag = False
    self.assertTrue(flag, "ls command didn't list all the "
                    "directories and files")
    g.log.info("Creation of files at multiple levels successful")

    command = 'cd ' + m_point + ';find root_dir -type d -print'
    ret, out, _ = g.run(self.mounts[0].client_system, command)
    self.assertEqual(ret, 0, "Creation of directory list failed")
    list_of_all_dirs = out.split('\n')
    del list_of_all_dirs[-1]

    flag = validate_files_in_dir(self.clients[0],
                                 m_point + '/root_dir',
                                 test_type=k.TEST_LAYOUT_IS_COMPLETE)
    self.assertTrue(flag, "Layout has some holes or overlaps")
    g.log.info("Layout is completely set")

    brick_list = get_all_bricks(self.mnode, self.volname)
    for direc in list_of_all_dirs:
        list_of_gfid = []
        for brick in brick_list:
            # the partition function returns a tuple having 3 elements.
            # the host address, the character passed i.e. ':'
            # , and the brick path
            brick_tuple = brick.partition(':')
            brick_path = brick_tuple[2]

          **gfid = get_fattr(brick_tuple[0], brick_path + '/' + direc,

                             'trusted.gfid')**

/root/glusto-tests/tests/functional/dht/test_dht_create_dir.py:127:

/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/glusterfile.py:113: in get_fattr
rcode, rout, rerr = g.run(host, command)
/usr/local/lib/python3.8/site-packages/glusto-0.72-py3.8.egg/glusto/connectible.py:132: in run
stdout, stderr = proc.communicate()
/usr/lib64/python3.8/subprocess.py:1024: in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
/usr/lib64/python3.8/subprocess.py:1904: in _communicate
stdout = self._translate_newlines(stdout,

self = <subprocess.Popen object at 0x7f22b4e2f490>, data = b'\xber\t\nO\xebO\xee\xa4\x9c\xc4L\xac\x1cj\xd5', encoding = 'UTF-8', errors = 'strict'

def _translate_newlines(self, data, encoding, errors):

  data = data.decode(encoding, errors)

E UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbe in position 0: invalid start byte

/usr/lib64/python3.8/subprocess.py:901: UnicodeDecodeError
===================================================================================================================================================================================== warnings summary =====================================================================================================================================================================================
test_dht_create_dir.py: 16 warnings
/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/peer_ops.py:293: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

test_dht_create_dir.py: 16 warnings
/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/peer_ops.py:296: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

test_dht_create_dir.py: 113 warnings
/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/volume_ops.py:707: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

test_dht_create_dir.py: 113 warnings
/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/volume_ops.py:713: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

test_dht_create_dir.py: 113 warnings
/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/volume_ops.py:716: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

test_dht_create_dir.py: 339 warnings
/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/volume_ops.py:719: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

test_dht_create_dir.py: 339 warnings
/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/volume_ops.py:727: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_directory
test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_directory
test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_directory
test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_link_for_directory
test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_link_for_directory
test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_link_for_directory
/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/volume_ops.py:453: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

test_dht_create_dir.py: 100 warnings
/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/volume_ops.py:820: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_directory
test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_link_for_directory
/usr/local/lib/python3.8/site-packages/glustolibs_gluster-0.22-py3.8.egg/glustolibs/gluster/peer_ops.py:345: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.

-- Docs: https://docs.pytest.org/en/stable/warnings.html
================================================================================================================================================================================= short test summary info ==================================================================================================================================================================================
FAILED ../root/glusto-tests/tests/functional/dht/test_dht_create_dir.py::TestDhtClass_cplex_distributed_glusterfs::test_create_directory - UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbe in position 0: invalid start byte
================================================================================================================================================================== 1 failed, 1 passed, 1157 warnings in 117.31s (0:01:57) ==================================================================================================================================================================
Ending glusto via main()

have a single IO tool for generating all kinds of IOs

need a single io tool for generating all kinds of IOs as below
char file create
block file create
truncate
fallocate (and with hole punch)
different sized files,
different chunk sizes to same file(ie varied block size) to same file
overlapping writes
appends
creates
deletes
pipes
soft/hardlinks

also IOs where i can have below operations performed
taking locks on files and holding to them
opening a file using an fd

Need contributor guide

How to contribute to glusto-tests?

Please update about contributing to glusto-tests in the top level README either inline (or) in separate contributing.md

glustolibs.gluster.brick_libs.bring_bricks_offline takes volname as a parameter that's unused

In brick_libs.bring_bricks_offline, volname is the first parameter and it remains unused throughout the function.