redhatqe / wrapanapi Goto Github PK

Error:

googleapiclient.discovery_cache - WARNING - file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth
Traceback (most recent call last):
  File "/home/jitendrayejare/Desktop/Learnings/Python/pyVirEnv/robottelo3/lib64/python3.7/site-packages/googleapiclient/discovery_cache/__init__.py", line 36, in autodetect
    from google.appengine.api import memcache
ModuleNotFoundError: No module named 'google.appengine'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jitendrayejare/Desktop/Learnings/Python/pyVirEnv/robottelo3/lib64/python3.7/site-packages/googleapiclient/discovery_cache/file_cache.py", line 33, in <module>
    from oauth2client.contrib.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.contrib.locked_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jitendrayejare/Desktop/Learnings/Python/pyVirEnv/robottelo3/lib64/python3.7/site-packages/googleapiclient/discovery_cache/file_cache.py", line 37, in <module>
    from oauth2client.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.locked_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jitendrayejare/Desktop/Learnings/Python/pyVirEnv/robottelo3/lib64/python3.7/site-packages/googleapiclient/discovery_cache/__init__.py", line 41, in autodetect
    from . import file_cache
  File "/home/jitendrayejare/Desktop/Learnings/Python/pyVirEnv/robottelo3/lib64/python3.7/site-packages/googleapiclient/discovery_cache/file_cache.py", line 41, in <module>
    'file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth')
ImportError: file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth
2019-08-29 14:38:57 - googleapiclient.discovery_cache - WARNING - file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth
Traceback (most recent call last):
  File "/home/jitendrayejare/Desktop/Learnings/Python/pyVirEnv/robottelo3/lib64/python3.7/site-packages/googleapiclient/discovery_cache/__init__.py", line 36, in autodetect
    from google.appengine.api import memcache
ModuleNotFoundError: No module named 'google.appengine'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jitendrayejare/Desktop/Learnings/Python/pyVirEnv/robottelo3/lib64/python3.7/site-packages/googleapiclient/discovery_cache/file_cache.py", line 33, in <module>
    from oauth2client.contrib.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.contrib.locked_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jitendrayejare/Desktop/Learnings/Python/pyVirEnv/robottelo3/lib64/python3.7/site-packages/googleapiclient/discovery_cache/file_cache.py", line 37, in <module>
    from oauth2client.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.locked_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jitendrayejare/Desktop/Learnings/Python/pyVirEnv/robottelo3/lib64/python3.7/site-packages/googleapiclient/discovery_cache/__init__.py", line 41, in autodetect
    from . import file_cache
  File "/home/jitendrayejare/Desktop/Learnings/Python/pyVirEnv/robottelo3/lib64/python3.7/site-packages/googleapiclient/discovery_cache/file_cache.py", line 41, in <module>
    'file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth')
ImportError: file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth

pyvcloud version we are using since #231 is frozen and contains a critical vulnerability.

Fix in pyvcloud is already present: vmware-archive/pyvcloud#266

Hello, I am trying to run some automation tests that require setting up VMware provider on CFME appliance and creating a test VM on top of it. The task seems to be failing somewhere in wrapanapi. Log follows:

2017-07-05 12:57:47,300 [I] ----------------------- test_html5_vm_console[vsphere55] ----------------------- (fixtures/log.py:21)
2017-07-05 12:57:47,301 [I] Running rails command '"File.open('/tmp/yam_dump.yaml', 'w') {|f| f.write(Settings.to_hash.deep_stringify_keys.to_yaml) }"' (utils/ssh.py:319)
2017-07-05 12:57:47,301 [I] Running command 'cd /var/www/miq/vmdb; bin/rails runner "File.open('/tmp/yam_dump.yaml', 'w') {|f| f.write(Settings.to_hash.deep_stringify_keys.to_yaml) }"' (utils/ssh.py:240)
2017-07-05 12:57:55,721 [I] Running command 'cat /tmp/yam_dump.yaml' (utils/ssh.py:240)
2017-07-05 12:57:56,084 [I] Running command 'ip address show' (utils/ssh.py:240)
2017-07-05 12:57:56,213 [I] [DB] db_url is postgresql://root:[email protected]:5432/vmdb_production (utils/db.py:196)
2017-07-05 12:57:56,455 [I] Running command 'cat /var/www/miq/vmdb/GUID' (utils/ssh.py:240)
2017-07-05 12:57:56,602 [I] Running rails command '"File.open('/tmp/yam_dump.yaml', 'w') {|f| f.write(Settings.to_hash.deep_stringify_keys.to_yaml) }"' (utils/ssh.py:319)
2017-07-05 12:57:56,602 [I] Running command 'cd /var/www/miq/vmdb; bin/rails runner "File.open('/tmp/yam_dump.yaml', 'w') {|f| f.write(Settings.to_hash.deep_stringify_keys.to_yaml) }"' (utils/ssh.py:240)
2017-07-05 12:58:04,673 [I] Running command 'cat /tmp/yam_dump.yaml' (utils/ssh.py:240)
2017-07-05 12:58:05,218 [I] Running command 'test -s /etc/init.d/merkyl' (utils/ssh.py:240)
2017-07-05 12:58:05,345 [I] Checking appliance's /etc/hosts for its own hostname (cfme/fixtures/base.py:57)
2017-07-05 12:58:05,345 [I] Running command 'grep $(hostname) /etc/hosts' (utils/ssh.py:240)
2017-07-05 12:58:05,528 [I] Running rails command '"File.open('/tmp/yam_dump.yaml', 'w') {|f| f.write(Settings.to_hash.deep_stringify_keys.to_yaml) }"' (utils/ssh.py:319)
2017-07-05 12:58:05,528 [I] Running command 'cd /var/www/miq/vmdb; bin/rails runner "File.open('/tmp/yam_dump.yaml', 'w') {|f| f.write(Settings.to_hash.deep_stringify_keys.to_yaml) }"' (utils/ssh.py:240)
2017-07-05 12:58:14,035 [I] Running command 'cat /tmp/yam_dump.yaml' (utils/ssh.py:240)
2017-07-05 12:58:15,049 [I] (rest-api) [RESTAPI] GET https://10.8.198.137:443/api {} (/home/kkulkarn/git/cfme_venv/lib/python2.7/site-packages/manageiq_client/api.py:84)
2017-07-05 12:58:15,320 [I] (rest-api) [RESTAPI] GET https://10.8.198.137/api/providers {'filter[]': [u'name = "vSphere 5.5 (nested)"']} (/home/kkulkarn/git/cfme_venv/lib/python2.7/site-packages/manageiq_client/api.py:84)
2017-07-05 12:58:15,464 [I] (rest-api) [RESTAPI] GET https://10.8.198.137/api/providers/2 {} (/home/kkulkarn/git/cfme_venv/lib/python2.7/site-packages/manageiq_client/api.py:84)
2017-07-05 12:58:15,603 [I] Running command 'date --iso-8601=seconds -u' (utils/ssh.py:240)
2017-07-05 12:58:15,787 [I] Getting ready to deploy VM/instance test-scat-zfxr from template damnSmallLinux on provider vSphere 5.5 (nested) (utils/virtual_machines.py:69)
2017-07-05 12:58:16,449 [I] Provisioning progress damnSmallLinux->test-scat-zfxr: Picked datastore iSCSI-Datastore-1 (/home/kkulkarn/git/cfme_venv/lib/python2.7/site-packages/wrapanapi/virtualcenter.py:579)
2017-07-05 12:58:16,470 [I] Provisioning progress damnSmallLinux->test-scat-zfxr: Picked resource pool Resources (/home/kkulkarn/git/cfme_venv/lib/python2.7/site-packages/wrapanapi/virtualcenter.py:579)
2017-07-05 12:58:16,503 [I] Provisioning progress damnSmallLinux->test-scat-zfxr: Picked folder vm (/home/kkulkarn/git/cfme_venv/lib/python2.7/site-packages/wrapanapi/virtualcenter.py:579)
2017-07-05 12:58:16,575 [I] Provisioning progress damnSmallLinux->test-scat-zfxr: running (/home/kkulkarn/git/cfme_venv/lib/python2.7/site-packages/wrapanapi/virtualcenter.py:579)
2017-07-05 12:58:20,623 [I] Provisioning progress damnSmallLinux->test-scat-zfxr: error (/home/kkulkarn/git/cfme_venv/lib/python2.7/site-packages/wrapanapi/virtualcenter.py:579)
2017-07-05 12:58:20,655 [E] Could not provisioning VM/instance test-scat-zfxr (AttributeError: 'vmodl.LocalizableMessage[]' object has no attribute 'message') (utils/virtual_machines.py:77)
2017-07-05 12:58:21,051 [E] Could not retrieve VM/Instance status: VMInstanceNotFound: Could not find a VM/instance named test-scat-zfxr. (utils/virtual_machines.py:20)
2017-07-05 12:58:21,051 [I] Attempting cleanup on VM/instance test-scat-zfxr (utils/virtual_machines.py:21)
2017-07-05 12:58:21,478 [E] @pytest.fixture(scope="function")
def vm_obj(request, provider, setup_provider, small_template, vm_name):
vm_obj = VM.factory(vm_name, provider, template_name=small_template)

    @request.addfinalizer
    def _delete_vm():
        try:
            vm_obj.delete_from_provider()
        except Exception:
            logger.warning("Failed to delete vm `{}`.".format(vm_obj.name))

  vm_obj.create_on_provider(timeout=2400, find_in_cfme=True, allow_skip="default")

cfme/tests/cloud_infra_common/test_html5_vm_console.py:34:

---

cfme/common/vm.py:731: in create_on_provider
deploy_template(self.provider.key, self.name, self.template_name, **kwargs)
utils/virtual_machines.py:73: in deploy_template
vm_name = provider_crud.mgmt.deploy_template(timeout=timeout, **deploy_args)
../cfme_venv/lib/python2.7/site-packages/wrapanapi/virtualcenter.py:700: in deploy_template
self.clone_vm(template, destination, **kwargs)

---

def clone_vm(self, source, destination, resourcepool=None, datastore=None, power_on=True,
             sparse=False, template=False, provision_timeout=1800, progress_callback=None,
             allowed_datastores=None, cpu=None, ram=None, **kwargs):
    """Clone a VM"""
    try:
        vm = self._get_obj(vim.VirtualMachine, name=destination)
        if vm and vm.name == destination:
            raise Exception("VM already present!")
    except VMInstanceNotFound:
        pass

    if progress_callback is None:
        progress_callback = partial(self._progress_log_callback, self.logger,
            source, destination)

    source_template = self._get_vm(source)

    vm_clone_spec = vim.VirtualMachineCloneSpec()
    vm_reloc_spec = vim.VirtualMachineRelocateSpec()
    # DATASTORE
    if isinstance(datastore, six.string_types):
        vm_reloc_spec.datastore = self._get_obj(vim.Datastore, name=datastore)
    elif isinstance(datastore, vim.Datastore):
        vm_reloc_spec.datastore = datastore
    elif datastore is None:
        if allowed_datastores is not None:
            # Pick a datastore by space
            vm_reloc_spec.datastore = self._pick_datastore(allowed_datastores)
        else:
            # Use the same datastore
            datastores = source_template.datastore
            if isinstance(datastores, (list, tuple)):
                vm_reloc_spec.datastore = datastores[0]
            else:
                vm_reloc_spec.datastore = datastores
    else:
        raise NotImplementedError("{} not supported for datastore".format(datastore))
    progress_callback("Picked datastore `{}`".format(vm_reloc_spec.datastore.name))

    # RESOURCE POOL
    if isinstance(resourcepool, vim.ResourcePool):
        vm_reloc_spec.pool = resourcepool
    else:
        vm_reloc_spec.pool = self._get_resource_pool(resourcepool)
    progress_callback("Picked resource pool `{}`".format(vm_reloc_spec.pool.name))

    vm_reloc_spec.host = None
    if sparse:
        vm_reloc_spec.transform = vim.VirtualMachineRelocateTransformation().sparse
    else:
        vm_reloc_spec.transform = vim.VirtualMachineRelocateTransformation().flat

    vm_clone_spec.powerOn = power_on
    vm_clone_spec.template = template
    vm_clone_spec.location = vm_reloc_spec
    vm_clone_spec.snapshot = None

    if cpu is not None:
        vm_clone_spec.config.numCPUs = int(cpu)
    if ram is not None:
        vm_clone_spec.config.memoryMB = int(ram)

    try:
        folder = source_template.parent.parent.vmParent
    except AttributeError:
        folder = source_template.parent
    progress_callback("Picked folder `{}`".format(folder.name))

    task = source_template.CloneVM_Task(folder=folder, name=destination, spec=vm_clone_spec)

    def _check(store=[task]):
        try:
            if hasattr(store[0].info, 'progress') and store[0].info.progress is not None:
                progress_callback("{}/{}%".format(store[0].info.state, store[0].info.progress))
            else:
                progress_callback("{}".format(store[0].info.state))
        except AttributeError:
            pass
        if store[0].info.state not in {"queued", "running"}:
            return True
        else:
            store[0] = self._get_updated_obj(store[0])
            return False

    wait_for(_check, num_sec=provision_timeout, delay=4)

    if task.info.state != 'success':
        if hasattr(task.info.error, 'localizedMessage'):
            message = str(task.info.error.localizedMessage)
        elif hasattr(task.info.error, 'faultMessage'):

          message = str(task.info.error.faultMessage.message)

E AttributeError: 'vmodl.LocalizableMessage[]' object has no attribute 'message'

../cfme_venv/lib/python2.7/site-packages/wrapanapi/virtualcenter.py:684: AttributeError (fixtures/log.py:70)
2017-07-05 12:58:21,492 [I] Begin delete_from_provider (cfme/common/vm.py:700)
2017-07-05 12:58:21,921 [I] Cleaning up VM test-scat-zfxr on provider vsphere55 (cfme/fixtures/vm_name.py:15)
2017-07-05 12:58:22,326 [W] Failed to clean up VM test-scat-zfxr on provider vsphere55 (cfme/fixtures/vm_name.py:19)
2017-07-05 12:58:22,343 [I] ---------------- test_html5_vm_console[vsphere55] result: error ---------------- (fixtures/log.py:57)
2017-07-05 12:58:22,343 [I] ------------------------ test_html5_vm_console[rhevm36] ------------------------ (fixtures/log.py:21)

The azurermclient.get_vm(name=host_ft.name) is failing even though the VM is completely provisioned and visible from Azure UI.

This affects RedHat Satellite Azure provisioning test cases in failing, so need to look into it on priority!

def rack_stats(self, ...):
  return evaluate_lambdas(hash_with_lambdas, keys, lambda_args)

def rack_inventory(self, ...):
  return evaluate_lambdas(hash_with_lambdas, keys, lambda_args)

def chassis_stats(self, ...):
  return evaluate_lambdas(hash_with_lambdas, keys, lambda_args)

def chassis_inventory(self, ...):
  return evaluate_lambdas(hash_with_lambdas, keys, lambda_args)

(of course, you need to provide correct values for hash_with_lambdas, keys and lambda_args)

Because we're doing the same thing all the time, evaluating correct lambdas... Maybe we could put evaluate_lambdas in a Mixin even to make it reusable.

Originally posted by @miha-plesko in #335

In SCVMMSystem class list_clusters, list_host, list_networks throwing error of PowerShellScriptError:

PowerShellScriptError: Script returned 1!: Get- : The term 'Get-' is not recognized as the name of a cmdlet, function, 
script file, or operable program. Check the spelling of the name, or if a path 
was included, verify that the path is correct and try again.

The list_vm and/or all_vm methods for the Google module do not filter instances by region, but return instances from all regions.

The API should filter instances based on the region/zone configured for the GoogleCloudSystem object.

The MIQ/Integration_tests/scripts/cleanup_old_vms.py script uses vm_creation_time, and is encountering the following exception when parsing for vm creation time on the CFME QE azure account:

azure: Exception getting creation time for test-pwr-c-ppod
Traceback (most recent call last):
File "scripts/cleanup_old_vms.py", line 163, in scan_vm
vm_creation_time = provider_mgmt.vm_creation_time(vm_name)
File "/home/mshriver/repos/integration_tests/.cfme_tests/lib/python2.7/site-packages/wrapanapi/azure.py", line 324, in vm_creation_time
creation_time = parse(str(vhd_last_modified))
File "/home/mshriver/repos/integration_tests/.cfme_tests/lib/python2.7/site-packages/dateutil/parser.py", line 1168, in parse
return DEFAULTPARSER.parse(timestr, **kwargs)
File "/home/mshriver/repos/integration_tests/.cfme_tests/lib/python2.7/site-packages/dateutil/parser.py", line 559, in parse
raise ValueError("Unknown string format")
ValueError: Unknown string format

I am trying out SCVMM list vms method, but when i am running the code getting
File "/Users/rohsing2/.pyenv/versions/3.7.7/lib/python3.7/site-packages/winrm/transport.py", line 268, in _send_message_request raise InvalidCredentialsError("the specified credentials were rejected by the server") winrm.exceptions.InvalidCredentialsError: the specified credentials were rejected by the server

I am able to get information running command directly by importing winrm module stand alone
#ps_script = """ Get-SCVirtualMachine -All -VMMServer localhost """ #s = winrm.Session("http://x.x.x.x.:5985/wsman", auth=('test', 'test$@#123'), transport='ntlm',server_cert_validation='ignore') #r = s.run_ps(ps_script) #print(str(r.std_out))
my code is straightforward
system = SCVMMSystem(hostname='x.x.x.x', username='test', password="test$@#123",domain=somevalue", provisioning="xxxx") system.list_vms()
Do we need domain value , and provisioning value as standalone winrm code is able to get information without it.
value for domain is known but what will be value for provisioning field.

https://github.com/Azure/azure-sdk-for-python/wiki/Direct-access-to-%22models%22-is-deprecated

The vm_creation_time methods use differrent timestamp formats.

For example:
Azure uses some string:
https://github.com/ManageIQ/wrapanapi/blob/master/wrapanapi/azure.py#L319

Google uses an ISO format:
https://github.com/ManageIQ/wrapanapi/blob/master/wrapanapi/google.py#L543

Standardize these so they can be easily parsed when dealing with multiple provider types.

Evaluate a migration to ovirtsdk4.

Red Hat Cloud QE, using this library to create VM's and add disks to them, is hitting an authentication/authorization bug against RHV4.1.

This bug is not reproducable under v4 of the API, and the migration may resolve the bugs we're seeing.

Evaluate the impact of an ovirtsdk upgrade to a newer API version and refactor wrapanapi/rhevm.py if the migration is feasible.

i just noticed current Github Release is not matching pip.
Is this behavior intentional or bug as relevant GH Action didn't run @mshriver ?

WrapanAPI needs methods to list all the unused resources for following:

• Nics
• Disks
• PIPs

For now it has methods to list the free resources attached to VM(vm_name). But this use case is for list all unused/unattended resources left after the VMs are deleted.

Issue: from time to time we get VMs named not by common practice 'nick','blablblba','justsometext' etc and they can be forgotten by the owner for some time. is this case our cleanup jobs will miss them
Idea: Stop all VM which are not in whitelist/don't have dnd/ etc (Do we have a requirement for VM to stay for a long period except adding IRC nickname into vm_name?)
Stopped VM won't cost anything for us. Cleanup them after N days/weeks/ or if they are forgotten/unused?
@mpusater, @mshriver , @izapolsk

delete_vm() is returning False erroneously on vsphere 5.5, 6.0, and 6.5.

The failures are split between failing during power off action (actually does power off, method fails/times out though), and delete status checking (vm is deleted, failure is returned immediately instead of waiting for success message)

Currently wrapanapi is not supporting ability to read notes on a VM. We need to add that as it would be required for the test in the PR https://github.com/ManageIQ/integration_tests/pull/9463/files
Currently it is added as a TODO to fix the test case once this wrapanapi issue is resolved.

It will be helpful for new people like me to understand more about mgmtsystem if we have some documentations.

Can you add documents like,

• What can we do with this project
• How to contribute on this project
• Where should I start?
• How to use this module in my code with few samples.

after python 3.7 release async is now treat as reserved keyword. In our project, we have one async.

References:
https://bugzilla.redhat.com/show_bug.cgi?id=1622043
oVirt/ovirt-engine-sdk@7f75f47#diff-1540af24608edbd48a0ef0357e473a14

This snippet, provided by @gshefer, demonstrates the desired data structure:

def pods_per_ready_status(self):
        """Grabing the Container Statuses Summary of the pods from API"""
        #  TODO: Add later this logic to wrapanapi
        entities_j = self.mgmt.api.get('pod')[1]['items']
        out = {}
        for entity_j in entities_j:
            out[entity_j['metadata']['name']] = {
                condition['type']: eval_strings([condition['status']]).pop()
                for condition in entity_j['status'].get('conditions', [])
            }
        return out

WrapanAPI needs methods to list all the unused resources in EC2 those are:

• Nics
• Disks
• IPs

For now it has methods to list unused resources but not for deleting them. But this use case will handle that for each resource cleanup and combining all in one def.

Currently we use ENV vars to define whether to use mock or non-mock data in test_openshift.
We should putting the mocked/non-mocked variants into plugin objects.