Attempting to run add_provider.yml with an Openshift provider and getting the following error:

"msg": "Failed to add provider. Error: APIException("Api::BadRequestError: Could not create the new provider - unknown attribute 'certificate_authority' for Endpoint.",)"

The yml looks like this:

• hosts: localhost
  tasks:

  manageiq_provider: 
    hawkular_hostname: <hostname>
    hawkular_port: 443
    metrics: "True"
    miq_password: 
    miq_url: <url>
    miq_username: 
    miq_verify_ssl: "False"
    name: "CI OSE"
    provider_api_auth_token: <token>
    provider_api_hostname: <hostname>
    provider_api_port: 8443
    provider_type: openshift-origin
    state: present
  name: "Add Openshift Containers Provider to ManageIQ"
  register: result
  

  • debug: var=result

More info here:
http://pastebin.test.redhat.com/466033

When I ran Ansible script with an incorrect password I got the crash message:

fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_name": "manageiq_provider"}, "module_stderr": "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:821: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html\n InsecureRequestWarning)\nTraceback (most recent call last):\n File "/tmp/ansible_J_hZLz/ansible_module_manageiq_provider.py", line 368, in \n main()\n File "/tmp/ansible_J_hZLz/ansible_module_manageiq_provider.py", line 350, in main\n manageiq = ManageIQ(module, miq_url, miq_username, miq_password, verify_ssl, ca_bundle_path)\n File "/tmp/ansible_J_hZLz/ansible_module_manageiq_provider.py", line 149, in init\n self.client = MiqApi(self.api_url, (self.user, self.password), verify_ssl=verify_ssl, ca_bundle_path=ca_bundle_path)\n File "/usr/lib/python2.7/site-packages/miqclient/api.py", line 41, in init\n self._load_data()\n File "/usr/lib/python2.7/site-packages/miqclient/api.py", line 44, in _load_data\n data = self.get(self._entry_point)\n File "/usr/lib/python2.7/site-packages/miqclient/api.py", line 88, in get\n return self._result_processor(data)\n File "/usr/lib/python2.7/site-packages/miqclient/api.py", line 65, in _result_processor\n "{}: {}".format(result["error"]["klass"], result["error"]["message"]))\nmiqclient.api.APIException: MiqException::MiqEVMLoginError: Authentication failed\n", "module_stdout": "", "msg": "MODULE FAILURE", "parsed": false}

Should be more informative regarding the source of failure

I used this template to run add_provider script but received: "msg": "unsupported parameter for module: hawkular_hostname"
Here's my add_provider.yml:

-hosts: localhost
tasks:
	-manageiq_provider: {
		hawkular_hostname: myhawkular.redhat.com,
		hawkular_port: '443',
		metrics: 'True',
		metrics_type: hawkular,
		miq_password: password,
		miq_url: 'https://0.0.0.0',
		miq_username: admin,
		miq_verify_ssl: false,
		name: cm-env1,
		provider_api_auth_token: mytoken
		provider_api_hostname: myopenshift.redhat.com,
		provider_api_port: 8443,
		provider_type: openshift-origin,
		provider_verify_ssl: false,
		state: present
	}
name: Add Openshift Containers Provider to ManageIQ
register: result -
	{
		debug: var = result
	}

Looking at

It's funny that a class called "ManageIQUser" has a constructor argument "user - the username in manageiq" that's just the MIQ API username...
This constructor is same across all modules, right?
The obvious next step is a shared superclass but I'm not sure that's right. The classes would mix 2 responsibilities — inherited responsibility of constructing MiqApi + per-module ansible actions.

I'm thinking we should rather construct MiqApi in main(), via shared helper function(s). Something like:

    module = AnsibleModule(
        argument_spec=dict(
            somewhere.miq_api_argument_spec(),
            userid=...,
            ...  # without miq_url, miq_username, miq_password, verify_ssl, ca_bundle_path
    ...
    
    userid         = module.params['userid']
    ...  # without miq_url, miq_username, miq_password, verify_ssl, ca_bundle_path
    
    client = somewhere.miq_api(module.params)
    
    manageiq_user = ManageIQUser(module, client)
    if state == "present":
        res_args = manageiq_user.create_or_update_user(userid, username, password,
                                                       group, email)
    if state == "absent":
        res_args = manageiq_user.delete_user(userid)

I ran the following yml against my appliance:

- hosts: localhost
  tasks:
  - manageiq_tag_assignment:
      miq_password: password
      miq_url: https://0.0.0.0
      miq_username: admin
      miq_verify_ssl: false
      resource: user
      resource_name: admin
      state: present
      tags:
      - {category: environment, name: qa}
      - {category: department, name: accounting}
    name: Create a tag in ManageIQ
    register: result
  - {debug: var=result}

And got the following error:
"msg": "Failed to assign tag: admin user does not exist in manageiq"
The admin user definitely exists in manageiq and I also tried with another user I created and had the same problem

I know it is a hassle to get the error messages right with ansible. Well.

I have tried to add a provider trough a http like this:

ansible-playbook -vvv add_provider.yml --extra-vars "name=oshift01 type=openshift-origin state=present zone=default miq_url=http://X.Y.Z.56 miq_username=admin miq_password=smartvm hostname=A.B.C.8 port=8443 token="$TOKEN" metrics=True verify_ssl=False hawkular_hostname=A.B.C.8 hawkular_port=443"

There is a HTTP redirection to https port:

`--> curl -v -u admin:smartvm -k http://A.B.C.138/api/providers/ 
> GET /api/providers/ HTTP/1.1
< HTTP/1.1 302 Found
...
<p>The document has moved <a href="https://10.8.168.138/api/providers/">here</a>.</p>

Which the miq-client-python doesn't seem to follow. Then the ansible module fails horribly:

fatal: [localhost]: FAILED! => {"changed": false, "failed": true, ... blah blah...
"msg": "Failed to add provider. Error: 'results'"}

When you try to remove the provider by setting the state=absent, the error message will make your day:

"msg": "Failed to delete oshift01 provider. Error: 'success'"

Similar to #48, the ManageIQ REST API won't tell us the passwords of users,
so we don't compare it in user_update_required(), so if you try to change only the password the module will do nothing.
[untested, based on looking at code only]

The only solution I see is always assume user_update_required() == True and do the POST.

• Would be nice if ManageIQ API at least told us if anything changed after the fact, so we could report "changed": true / "changed": false appropriately...

cc @abraverm who asked whether this works.

ansible-doc command should show documentation on Ansible modules.
It fails on all modules with a useless error message.

We should at least:

• check the yaml validity

• add top level description (in addition to the existing short_descriptions)

cc @cben

After running an Ansible Playbook to update a provider I am getting the following message:

ansible-playbook update_provider_pavel.yml 
 [WARNING]: provided hosts list is empty, only localhost is available


PLAY [localhost] ***************************************************************

TASK [setup] *******************************************************************
ok: [localhost]

TASK [Add Openshift Containers Provider to ManageIQ] ***************************
changed: [localhost]

TASK [debug] *******************************************************************
ok: [localhost] => {
    "result": {
        "changed": true, 
        "msg": "Failed to validate provider Openshift 3 after update. Authentication: {'bearer': (u'Valid', u'Ok')}", 
        "provider_id": 5, 
        "updates": {
            "Added": {}, 
            "Removed": {}, 
            "Updated": {
                "default": {
                    "hostname": "something_different.redhat.com"
                }
            }
        }
    }
}

PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=1    unreachable=0    failed=0   

The provider itself was updated in UI

Running the same against an older code doesn't yield such error

Tried to run the script with a random token and the action was successful.
Is there a plan to add token validation?

https://docs.ansible.com/ansible/dev_guide/developing_modules_in_groups.html#before-you-start-coding

• pass pep8 --ignore=E402 --max-line-length=160 cleanly

• support Python 2.6 and Python 3.5+

  • I believe we use 2.7 dict comprehension syntax

• GPLv3 license => #60

• Shared code can be placed into lib/ansible/module_utils/ (under BSD license) — will this apply as soon as we extract shared code (#36)?

• Shared documentation (for example describing common arguments) can be placed in lib/ansible/utils/module_docs_fragments/— nice!

• With great power comes a duty to keep modules up to date

• Although not required, unit and/or integration tests are strongly recommended

  • we don't have any integration tests (against an actual MIQ). #38 could be a start...

• Naming Convention — I think this will apply when we actually contribute into ansible repo?

https://docs.ansible.com/ansible/dev_guide/developing_modules_checklist.html

summary of top-level bold bullets:

• The shebang must always be #!/usr/bin/python. This allows ansible_python_interpreter to work

• Modules must have ANSIBLE_METADATA section

• Documentation: Make sure it exists — we have it, but there is looong checklist there...

  • Verify that a GPL 3 License header is included.

• Predictable user interface — IMHO we're not bad on guidelines there

• Informative responses

  • Please note, that for >= 2.0, it is required that return data to be documented.
  • Return diff if in diff mode.

• Code — we're not bad but:

  • Modules should not do the job of other modules, that is what roles are for. Less magic is more.
    • manageiq_provider + validate + refresh perhaps violates this
  • Support check mode.

• Exceptions: The module must handle them.

• then follows flat soup of many more things:

  • The module must not use sys.exit() –> use fail_json() from the module object.

  • Import custom packages in try/except and handled with fail_json() in main() e.g.

  • The return structure should be consistent, even if NA/None are used for keys normally returned under other options.

  • Are module actions idempotent? If not document in the descriptions or the notes. — we're striving, but some modules have "complicated idempotency" worth documenting

  • Import ansible.module_utils code in the same place as you import other libraries. In older code, this was done at the bottom of the file but that’s no longer needed.

  • Do not use wildcards for importing other python modules (ex: from ansible.module_utils.basic import *).

  • The module must have a main function that wraps the normal execution.

  • Call your main() from a conditional so that it would be possible to import them into unittests

  • Try to normalize parameters with other modules, you can have aliases for when user is more familiar with underlying API name for the option — we're trying

  • Being pep8 compliant is nice, but not a requirement.

  • Avoid ‘action/command‘, they are imperative and not declarative, there are other ways to express the same thing

  • Do not add list or info state options to an existing module - create a new _facts module.

    • facts modules must return facts in the ansible_facts field of the result dictionary
    • modules that are purely about fact gathering need to implement check_mode.

  • Return values must be able to be serialized as json via the python stdlib json library.

  • When fetching URLs, please use either fetch_url or open_url from ansible.module_utils.urls rather than urllib2; urllib2 does not natively verify TLS certificates and so is insecure for https.

    • I wonder how API client libs are normally handled? We're so far on the border, we could drop the lib and use HTTP correctly with not much effort. But I hope they're not flat against using libs...

  • Basic auth: module_utils.api has some helpers ... — not applicable

    • ... may want to fallback on environment variables for default values — we do.

• Windows modules checklist — irrelevant

I noticed that when attempting to add a Provider I can't get the token added with the rest of the Provider's details.
I am attaching you my yaml, please check locally to see if it works for you. I can see the provider added but nothing gets populated under the Provider because the Token doesn't get in.
I see no error in Ansible reply as well.

YAML:

---
- hosts: localhost

  tasks:
  - name: Add Openshift Containers Provider to ManageIQ
    manageiq_provider:
      name: 'Openshift 3'
      provider_type: 'openshift-origin'
      state: 'present'
      provider_api_hostname: <My Hostname>
      provider_api_port: '8443'
      provider_api_auth_token: <Token>
      metrics: True
      hawkular_hostname: <My Hawkular Hostname>
      hawkular_port: '443'
      miq_url: <My MIQ URL>
      miq_username: <My User>
      miq_password: <My Pass>
      verify_ssl: False
    register: result

  - debug: var=result

Here's the reply I get:

[pavel@hades manageiq_ansible_module]$ ansible-playbook add_provider_test.yml 
 [WARNING]: provided hosts list is empty, only localhost is available


PLAY [localhost] ***************************************************************

TASK [setup] *******************************************************************
ok: [localhost]

TASK [Add Openshift Containers Provider to ManageIQ] ***************************
changed: [localhost]

TASK [debug] *******************************************************************
ok: [localhost] => {
    "result": {
        "changed": true, 
        "msg": "Successfuly added Openshift 3 provider", 
        "provider_id": 5
    }
}

PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=1    unreachable=0    failed=0   

manageiq_provider module doesn't compare token param to current token of existing provider, so if you change only token it will decide everything is up-to-date and not actually update the token.

The problem is that manageiq API, for security reasons, can't tell us what the current token is.

• Perhaps we could unconditionally do the POST with desired token, when all else is equal? That should still be idempotent on ManageIQ side, ManageIQ should do nothing if the token didn't change.

  • We won't know whether it changed or was already good, but that's minor problem compared to current behavior.
  • We won't know whether we should wait for validation. This could be fixed on ManageIQ side by doing validation synchronously, or keeping it async but immediately switching to temp "Validating" status.

• Another option we discussed was adding a param eg. force_token_update that would let user force a POST — essentially doing exactly the above, only not by default.

I hope we can do it unconditionally though, without requiring user to be aware of a corner case.

We should add the requirement file in order to track other packages dependencies:

https://pip.pypa.io/en/stable/user_guide/#requirements-files

cc @dkorn

This is tracking / announce issue. This repo is becoming unmaintained, we're in process of upstreaming these modules into Ansible itself. But the new modules have interface and sometimes behavior changes.

Anybody using any modules from this repo, subscribe to this issue to hear what's changing; better yet tell us where your playbooks are and we'll help transition :-)

cc @yaacov @theute @twiest @blrm @itewk @zgalor @JaryN @pavelzag @dtschan @ldomb. (some of you found by github search)
https://github.com/ManageIQ/integration_tests/tree/master/utils/ansible_conf
https://github.com/openshift/openshift-tools/blob/prod/ansible/playbooks/adhoc/manageiq/miq-setup.yml
https://github.com/appuio/ansible-role-openshift-zabbix-monitoring/blob/master/vendor/openshift-tools/ansible/playbooks/adhoc/manageiq/miq-setup.yml
https://github.com/ldomb/rhsummit2017/blob/master/roles/buildcfme/tasks/main.yml
https://github.com/strategicdesignteam/cfme-ansible/blob/master/cfme-ose-provider/tasks/main.yml

Mixed usage?

• Untested: can one use new modules from Ansible together with unported ones from this repo?

Upstreamed modules

New code: in https://github.com/ansible/ansible/search?q=manageiq

• manageiq_provider.py [released in Ansible 2.4]
  https://docs.ansible.com/ansible/devel/manageiq_provider_module.html
  • Removed auth validation and queueing of first refresh (pending better MIQ API).
  • Changed endpoint params (hostname etc)
• manageiq_user.py [released in Ansible 2.4]
  https://docs.ansible.com/ansible/devel/manageiq_user_module.html
  • name param became userid, fullname became name.
• manageiq_policy_assignment.py
  https://docs.ansible.com/ansible/devel/manageiq_policies_module.html
  • module renamed manageiq_policies
  • resource param renamed resource_type
  • as of this writing, only supports assignment of policy profiles to resources (e.g. a provider)
  • added state: list support
• manageiq_tag_assignment.py
  https://docs.ansible.com/ansible/devel/manageiq_tags_module.html
  • module renamed manageiq_tags
  • resource param renamed resource_type
  • added state: list support
• manageiq_alert.py — Significantly changed, split into 2 modules:
  https://docs.ansible.com/ansible/devel/module_docs/manageiq_alerts_module.html,
  https://docs.ansible.com/ansible/devel/module_docs/manageiq_alert_profiles_module.html
• manageiq_custom_attributes.py
  Not sure if planned, please write here if you want this.

All new modules specify how to connect to manageiq (api url, user etc) differently.

Handling of sensitive fields (user or provider password/token) changed — now always attempting update, always reporting changed: True.

I am running an add_provider.yml and I am getting the following error:

ansible-playbook add_provider.yml

[WARNING]: Host file not found: /etc/ansible/hosts

[WARNING]: provided hosts list is empty, only localhost is available

PLAY [localhost] ***************************************************************

TASK [setup] *******************************************************************
ok: [localhost]

TASK [Add Openshift Containers Provider to ManageIQ] ***************************
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Failed to Validate provider authentication after addition. details: {'bearer': (u'Valid', u'Ok'), 'hawkular': (u'Error', u'\n \n <meta name="viewport" content="width=device-width, initial-scale=1">\n <style type="text/css">\n /*!\n * Bootstrap v3.3.5 (http://getbootstrap.com)\\n * Copyright 2011-2015 Twi...')}"}
to retry, use: --limit @/var/lib/jenkins/workspace/cfme-5.7-ose-3.4-provider-test-dev/cfme_tests/utils/ansible_conf/add_provider.retry

PLAY RECAP *********************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=1

Here's my yaml:

[
{
"tasks": [
{
"register": "result",
"manageiq_provider": {
"provider_api_port": "8443",
"hawkular_hostname": "something.redhat.com",
"hawkular_port": "443",
"miq_username": "admin",
"miq_url": "https://x.x.x.x",
"state": "present",
"provider_api_hostname": "something.redhat.com",
"provider_api_auth_token": "token",
"name": "OSE",
"metrics": "True",
"miq_password": "password",
"verify_ssl": "False",
"provider_type": "openshift-origin"
},
"name": "Add Openshift Containers Provider to ManageIQ"
},
{
"debug": "var=result"
}
],
"hosts": "localhost"
}
]

I simply had no manageiq running on localhost:3000. The error I get is:

TASK [Add Openshift Containers Provider to ManageIQ] **************************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=3000): Max retries exceeded with url: /api (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fd0cc7f6050>: Failed to establish a new connection: [Errno 111] Connection refused',))
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_9_D4GC/ansible_module_manageiq_provider.py\", line 601, in <module>\n    main()\n  File \"/tmp/ansible_9_D4GC/ansible_module_manageiq_provider.py\", line 575, in main\n    manageiq = ManageIQProvider(module, miq_url, miq_username, miq_password, miq_verify_ssl, ca_bundle_path)\n  File \"/tmp/ansible_9_D4GC/ansible_module_manageiq_provider.py\", line 210, in __init__\n    self.client        = MiqApi(self.api_url, (self.user, self.password), verify_ssl=miq_verify_ssl, ca_bundle_path=ca_bundle_path)\n  File \"/home/bpaskinc/manageiq-api-client-python/src/manageiq_client/api.py\", line 41, in __init__\n    self._load_data()\n  File \"/home/bpaskinc/manageiq-api-client-python/src/manageiq_client/api.py\", line 44, in _load_data\n    data = self.get(self._entry_point)\n  File \"/home/bpaskinc/manageiq-api-client-python/src/manageiq_client/api.py\", line 83, in get\n    partial(self._session.get, url, params=get_params))\n  File \"/home/bpaskinc/manageiq-api-client-python/src/manageiq_client/api.py\", line 78, in _sending_request\n    raise last_connection_exception\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=3000): Max retries exceeded with url: /api (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fd0cc7f6050>: Failed to establish a new connection: [Errno 111] Connection refused',))\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 0}

I think this is a simple common error for which we should return a short json without traceback.