leapfrogonline / ansible-merge-vars Goto Github PK

View Code? Open in Web Editor NEW

104.0 104.0 20.0 91 KB

An Ansible action plugin to explicitly merge inventory variables

License: Other

Makefile 3.09% Python 96.91%

ansible-merge-vars's People

Contributors

Stargazers

Watchers

ansible-merge-vars's Issues

Test against multiple versions of Ansible

We should have CI run all of the tests against multiple versions of Ansible so we can be notified if the plugin needs to be updated with newer versions.

Missing interpreter line

Hello,

when running my playbook:
ansible-playbook -vvv --diff -i inventory/dev.ini playbook/env_dev.yml --limit merge-test

i just got :

task path: /home/hindikah/Projects/Environments/env_dev/ansible/roles/ufw/tasks/20_configuration.yml:2
Friday 29 May 2020  16:02:43 +0200 (0:00:01.507)       0:00:03.445 ************ 
Using module file /home/ybenoit/Projects/Environments/env_dev/ansible/library/merge_vars
fatal: [ka-test1]: FAILED! => {
    "msg": "module (merge_vars) is missing interpreter line"
}
META: cleared host errors
META: ending play

my role that call the merge_vars plugin:

- name: Merge basics rules
  merge_vars:
    suffix_to_merge: ufw_rules_basics
    merged_var_name: merged_ufw_rules_basics
    expected_type: 'list'

-> tree -L 3
.
├── ansible
│   ├── action_plugins
│   │   └── merge_vars.py
│   ├── inventory
│   │   └── core
│   ├── library
│   │   ├── merge_vars
│   ├── playbook
│   │   ├── v1.yml
│   ├── README.md
│   ├── roles
│   │   ├── ufw
├── ansible.cfg
├── ansible.log
├── inventory
│   ├── dev.ini
│   ├── group_vars
│   │   ├── merge-test-vm
│   └── host_vars
│       ├── merge-test
├── playbook
│   └── env_dev.yml
└── README.md

-> cat ansible.cfg 

[defaults]
nocows = 1
library = ansible/library
inventory = inventory
log_path = ansible.log
roles_path = ansible
vault_password_file = .vault_pass
retry_files_enabled = False
host_key_checking = False
callback_whitelist = profile_tasks
retries = 3
timeout = 30
gather_timeout = 60
local_tmp = ~/.ansible/tmp
action_plugins = ansible/action_plugins

gathering = smart

remote_user = ansible

# Default privilege escalation
[privilege_escalation]
become = true
become_method = sudo
become_user = root

[ssh_connection]
pipelining = True

cat ansible/action_plugins/merge_vars.py 
from ansible_merge_vars import ActionModule

As i saw in certain thread i tried to change the path(s), without any success, copy/paste the module itself in the merge_vars.py file, still without sucess !

Environment
Ansible Version: 2.9.7
Python Version: 2.7

Include in Ansible Community collection

Hi Guys,

Thank you for this awesome plugin! It really transformed the way I am able to use Ansible! :-)

Have you ever considered getting the module included in the Ansible Community Collection? This would allow a much greater audience to enjoy the features this module brings.

Love to hear from you! <3

ModuleNotFoundError: No module named 'ansible_merge_vars'

Hello.
I'm having problems using the ansible-merge-vars plugin.
Please, help and thank for your time.

Describe the Bug

$ pip install ansible_merge_vars

$ python -c "for dist in __import__('pkg_resources').working_set:print dist.project_name.replace('Python', '')"
wsgiref
... skipped
ansible
ansible-vault
ansible-playbook-debugger
ansible-merge-vars
wheel
six
... skipped
cryptography
asn1crypto

$ grep -B1 -A1 action ansible.cfg 
[defaults]
action_plugins = ./plugins/action_plugins

$ mkdir -p ./plugins/action_plugins

$ echo "from ansible_merge_vars import ActionModule" > ./plugins/action_plugins/merge_vars.py

$ ansible-doc -t module -l | grep ansible
add_host        add a host (and alternatively a group) to the ansible-playbook in-memory inventory                                                     
set_stats         Set stats for the current ansible run

$ cat playbook-library/test_merge_vars/test_merge_vars.yaml
- hosts: all
  vars:
    test_01_test__to_merge:
      - 123
      - 456
    test_02_test__to_merge:
      - 234
      - 567

  tasks:
  - name: Merge vars
    merge_vars:
      suffix_to_merge: test__to_merge
      merged_var_name: merged_test_vars
      expected_type: 'list'

$ ansible-playbook playbook-library/test_merge_vars/test_merge_vars.yaml -i inventories/test_merge_vars/inventory 

PLAY [all] **************************************************************************************************************************************************************************************

TASK [Gathering Facts] **************************************************************************************************************************************************************************
ok: [192.168.95.2]
ERROR! Unexpected Exception, this is probably a bug: No module named 'ansible_merge_vars'
to see the full traceback, use -vvv

$ ansible-playbook playbook-library/test_merge_vars/test_merge_vars.yaml -i inventories/test_merge_vars/inventory -vvv
ansible-playbook 2.6.2
  config file = /home/oldman/repos/my/ansible.cfg
  configured module search path = ['/home/oldman/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/oldman/.local/lib/python3.6/site-packages/ansible
  executable location = /home/oldman/.local/bin/ansible-playbook
  python version = 3.6.6 (default, Sep 12 2018, 18:26:19) [GCC 8.0.1 20180414 (experimental) [trunk revision 259383]]
Using /home/oldman/repos/my/ansible.cfg as config file
Parsed /home/oldman/repos/my/inventories/test_merge_vars/inventory inventory source with ini plugin

PLAYBOOK: test_merge_vars.yaml ******************************************************************************************************************************************************************
1 plays in playbook-library/test_merge_vars/test_merge_vars.yaml

PLAY [all] **************************************************************************************************************************************************************************************

TASK [Gathering Facts] **************************************************************************************************************************************************************************
task path: /home/oldman/repos/my/playbook-library/test_merge_vars/test_merge_vars.yaml:1
... skipped
ok: [192.168.95.2]
META: ran handlers
ERROR! Unexpected Exception, this is probably a bug: No module named 'ansible_merge_vars'
the full traceback was:

Traceback (most recent call last):
  File "/home/oldman/.local/bin/ansible-playbook", line 118, in <module>
    exit_code = cli.run()
  File "/home/oldman/.local/lib/python3.6/site-packages/ansible/cli/playbook.py", line 122, in run
    results = pbex.run()
  File "/home/oldman/.local/lib/python3.6/site-packages/ansible/executor/playbook_executor.py", line 159, in run
    result = self._tqm.run(play=play)
  File "/home/oldman/.local/lib/python3.6/site-packages/ansible/executor/task_queue_manager.py", line 289, in run
    play_return = strategy.run(iterator, play_context)
  File "/home/oldman/.local/lib/python3.6/site-packages/ansible/plugins/strategy/linear.py", line 247, in run
    action = action_loader.get(task.action, class_only=True)
  File "/home/oldman/.local/lib/python3.6/site-packages/ansible/plugins/loader.py", line 378, in get
    self._module_cache[path] = self._load_module_source(name, path)
  File "/home/oldman/.local/lib/python3.6/site-packages/ansible/plugins/loader.py", line 357, in _load_module_source
    module = imp.load_source(full_name, path, module_file)
  File "/usr/lib/python3.6/imp.py", line 172, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/oldman/repos/my/plugins/action_plugins/merge_vars.py", line 1, in <module>
    from ansible_merge_vars import ActionModule
ModuleNotFoundError: No module named 'ansible_merge_vars'

Ansible Playbook

- hosts: all
  vars:
    test_01_test__to_merge:
      - 123
      - 456
    test_02_test__to_merge:
      - 234
      - 567

  tasks:
  - name: Merge vars
    merge_vars:
      suffix_to_merge: test__to_merge
      merged_var_name: merged_test_vars
      expected_type: 'list'

Environment
Ansible Version: 2.6.2
Python Version: 2.7.15rc1
Python3 Version: 3.6.6
OS: Ubuntu 18.04.1 LTS

Unable to merge lists referencing other lists

Hello,

I use your plugin every day, which usually does great job.

But I have "templated variables" which looks like this:

template_esx:
  - switchport mode access
  - switchport access vlan 10
interfaces__to_merge:
  - int g1/1: >
    {{ [
    u'desc ESX 1'
    ] + template_esx
    }}
  - int g1/2: >
    {{ [
    u'desc ESX 2'
    ] + template_esx
    }}

It fails to merge them as it doesn't interpretate jinja, and gives me errors like this:
FAILED! => {"failed": true, "msg": "template error while templating string: expected token ',', got 'string'. String: {{ [ u'desc ESX1' ] + template_esx }}\n"}

I guess it would be possible to add something to your code to make it render jinja templates when he finds them. But I'm not good enough in Python coding to see exactly what to do and where.

Could you help me ?

No longer works with 2.5

Traceback:

TASK [merge_vars] ***********************************************************************
task path: /data/test.yml:43
Merging vars in this order: [u'1_attributes__to_merge',u'2_attributes__to_merge']
ERROR! Unexpected Exception, this is probably a bug: update expected at most 1 arguments, got 2
the full traceback was:

Traceback (most recent call last):
File "/usr/bin/ansible-playbook", line 118, in
exit_code = cli.run()
File "/usr/lib/python2.7/site-packages/ansible/cli/playbook.py", line 122, in run
results = pbex.run()
File "/usr/lib/python2.7/site-packages/ansible/executor/playbook_executor.py", line 159, in run
result = self._tqm.run(play=play)
File "/usr/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py", line 290, in run
play_return = strategy.run(iterator, play_context)
File "/usr/lib/python2.7/site-packages/ansible/plugins/strategy/linear.py", line 291, in run
results += self._wait_on_pending_results(iterator)
File "/usr/lib/python2.7/site-packages/ansible/plugins/strategy/init.py", line 680, in _wait_on_pending_results
results = self._process_pending_results(iterator)
File "/usr/lib/python2.7/site-packages/ansible/plugins/strategy/init.py", line 117, in inner
results = func(self, iterator, one_pass=one_pass, max_passes=max_passes)
File "/usr/lib/python2.7/site-packages/ansible/plugins/strategy/init.py", line 612, in _process_pending_results
self._variable_manager.set_host_facts(target_host, result_item['ansible_facts'].copy())
File "/usr/lib/python2.7/site-packages/ansible/vars/manager.py", line 585, in set_host_facts
self._fact_cache.update(host.name, facts)
TypeError: update expected at most 1 arguments, got 2

How this can be done with standard ansible

Hello,
after revisiting the whole dict merge deprecation mess in ansible again - I think it's possible with the default "combine" ansible module by now - here's an example:

- name: mergedicts
  hosts: localhost
  vars:
    v0a_iptables__to_merge:
      a:
        x: default
        y: default
      b: default
      c: default
    v1a_iptables__to_merge:
      a:
        y: patch1
        z: patch1
      b: patch1
    v2a_iptables__to_merge:
      a:
        w: patch2
        z: patch2
      c: patch2

  tasks:
    - name: varnames iptables__to_merge
      ansible.builtin.debug:
        msg: "{{ lookup('vars', item) }}"
      loop: "{{ lookup('ansible.builtin.varnames', '.+iptables__to_merge$', wantlist=True) }}"

    - name: merge varnames with suffix iptables__to_merge
      set_fact:
        iptables_dict: "{{ iptables_dict | default({}) | ansible.builtin.combine(lookup('vars', item), recursive=True) }}"
      loop: "{{ lookup('ansible.builtin.varnames', '.+iptables__to_merge$', wantlist=True) }}"

    - name: iptables_dict
      ansible.builtin.debug: msg="{{ iptables_dict }}"

As the order of merging dicts is important - I use the convention:

v0: group_vars
v1: roles
v2: host_vars

Run with ansible-playbook playbook_name.yaml

TASK [varnames iptables__to_merge] ********************************************************************************************
ok: [localhost] => (item=v0a_iptables__to_merge) => {
    "msg": {
        "a": {
            "x": "default",
            "y": "default"
        },
        "b": "default",
        "c": "default"
    }
}
ok: [localhost] => (item=v1a_iptables__to_merge) => {
    "msg": {
        "a": {
            "y": "patch1",
            "z": "patch1"
        },
        "b": "patch1"
    }
}
ok: [localhost] => (item=v2a_iptables__to_merge) => {
    "msg": {
        "a": {
            "w": "patch2",
            "z": "patch2"
        },
        "c": "patch2"
    }
}

TASK [merge varnames with suffix iptables__to_merge] **************************************************************************
ok: [localhost] => (item=v0a_iptables__to_merge)
ok: [localhost] => (item=v1a_iptables__to_merge)
ok: [localhost] => (item=v2a_iptables__to_merge)

TASK [iptables_dict] **************************************************************************************************
ok: [localhost] => {
    "msg": {
        "a": {
            "w": "patch2",
            "x": "default",
            "y": "patch1",
            "z": "patch2"
        },
        "b": "patch1",
        "c": "patch2"
    }
}

Anyway - thanks for the module - it has helped me a lot.

Module fails with "zero length field in format" error at "Merging vars in this order"

Describe the Bug
Ansible playbook fails with the following traceback:

The full traceback is:
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ansible/executor/task_executor.py", line 138, in run
res = self._execute()
File "/usr/lib/python2.6/site-packages/ansible/executor/task_executor.py", line 576, in _execute
result = self._handler.run(task_vars=variables)
File "/usr/lib/python2.6/site-packages/ansible_merge_vars.py", line 58, in run
display.v("Merging vars in this order: {}".format(keys))
ValueError: zero length field name in format

Ansible Playbook
If possible, include the smallest valid Ansible playbook that exhibits the bug
Tested using included example playbook "merg_dicts_playbook" from the examples folder.

name: Example of merging dicts
hosts: localhost
gather_facts: false # Speed up the example
vars: # Note that these could be definied anywhere in inventory
group1_settings__to_merge:
some_setting: "foo"
another_setting: "bar"
group2_settings__to_merge:
some_setting: "foo"
yet_another_setting: "baz"
group3_settings__to_merge:
some_setting: "foo"
a_totally_different_setting: "qux"
group4_some_other_var: # This won't get merged
what: ever
tasks:
- name: Merge settings vars
  merge_vars:
  suffix_to_merge: settings__to_merge
  merged_var_name: merged_settings
  expected_type: dict
- debug:
  var: merged_settings

Environment
Ansible Version: 2.6.2
Python Version: 2.6.6
Centos 7 system

Does the suffix_to_merge work with facts instead of variables?

Hello,

I am trying to merge 2 dictionaries that are facts, would the suffix_to_merge work with them?

final_json_fact3__to_merge
{
        "description": "",
        "variables": {
            "unid": "EEE8021627ED4554882580C10060C26F",
            "network_interface": "1.2.3.4"
        },
        "enabled": "true",
        "instance_id": "",
        "inventory": "null",
        "name": "host02"
    },

final_json_fact2__to_merge
{
        "description": "",
        "variables": {
            "unid": "EEE8021627ED4554882580C10060C26F",
            "network_interface": "1.2.3.4"
        },
        "enabled": "true",
        "instance_id": "",
        "inventory": "null",
        "name": "host01"
    },

my playbook task:

name: "Build the final_json_fact3 dictionary"
merge_vars:
suffix_to_merge: final_json_fact__to_merge, final_json_fact2__to_merge
merged_var_name: merge
dedup: false
recursive_dict_merge: false
expected_type: 'dict'
when: final_json_fact2__to_merge.keys()|length > 0

Ditch Travis CI

Travis CI is no longer providing unlimited free builds to open source projects:
https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing

We've gotta move CI for this project to something else free (for now). I'm going to try to move it to Github Actions.

Merge variables with the same name

Hi.

Is it possible to merge variables (dict, list) with the same name which exists in a different group groups? As for now, the plugin takes one of them

Let's me explain my use case.

[serviceX-servers]
node1
node2

[serviceX-clients]
node1
node2
node3

and group vars

# serviceX-servers file #1
serviceX_type: 
- server

# serviceX-clients file #2
serviceX_type: 
- client

And in a playbook I want to know which service type(s) I have on a server. I'd prerfer this way cause it more clean that have 2 (or may be even more) variables:

is_serviceX_client: false
is_serviceX_server: true

Is it a kind of misusage of your plugin or you can implement this option?

Thank you

Deploy to PyPI

Some folks are actually using this now :)

We should deploy it to pypi so that we packages changes into proper versions and folks can control what changes they pick up and when.

Make pip-installable
Add deployment to pypi from tagged commits to TravisCI config

WARNING on merging the same dict data

Describe the Bug
I get warning when the data in the {} is exactly the same. Even empty. This is the simplest bug report I could share.

I am using ansible-merge-vars==3.0.0, I tried 4.0.0 but it didn't change anything.

What am I doing wrong?

TASK [routing : Merge Aggregates] ****************************************************************************************************************************
task path: /Users/yeled/src/gh/network/ansible/roles/routing/tasks/main.yml:1
Friday 23 November 2018  12:52:57 +0000 (0:00:00.518)       0:00:10.563 *******
 [WARNING]: aggregates is already defined, are you sure you want to overwrite it?
The contents of aggregates are: {}
Merging vars in this order: []
ok: [leaf-1-lab.net.cp1-iad.github.net] => {
    "ansible_facts": {
        "aggregates": {}
    },
    "ansible_facts_cacheable": false,
    "changed": false
}

Ansible Playbook

- name: Merge Aggregates
  merge_vars:
    suffix_to_merge: __aggregates__to_merge
    merged_var_name: aggregates
    expected_type: 'dict'
    recursive_dict_merge: yes
  when:
    - not unmerged_aggregates is defined


- name: "Routing Aggregates"
  template: src={{os}}/aggregates.j2 dest=../../tmp/fragments/{{inventory_hostname}}/aggregates.conf
  when:
    - aggregates is defined
    - inventory_hostname_short is match("bdr.*|!ssw.*|!spine.*")

- name: "Default VRF route"
  template: src={{os}}/default-vrf.j2 dest=../../tmp/fragments/{{inventory_hostname}}/iproute-default-vrf.conf
  when:
    - inventory_hostname_short is match("^leaf.*")

Environment

ansible 2.7.0
  config file = /Users/yeled/src/gh/network/ansible/etc/ansible.cfg
  configured module search path = [u'/Users/yeled/src/gh/network/ansible/env/lib/python2.7/site-packages/napalm_ansible']
  ansible python module location = /Users/yeled/src/gh/network/ansible/env/lib/python2.7/site-packages/ansible
  executable location = /Users/yeled/src/gh/network/ansible/env/bin/ansible
  python version = 2.7.15 (default, Jun 17 2018, 12:46:58) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)]

No module named plugins.action

I wrote a simple task to try out the variable merging, and am seeing the following:

Traceback (most recent call last):
  File "/usr/local/bin/ansible-playbook", line 324, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/usr/local/bin/ansible-playbook", line 264, in main
    pb.run()
  File "/Library/Python/2.7/site-packages/ansible/playbook/__init__.py", line 348, in run
    if not self._run_play(play):
  File "/Library/Python/2.7/site-packages/ansible/playbook/__init__.py", line 789, in _run_play
    if not self._run_task(play, task, False):
  File "/Library/Python/2.7/site-packages/ansible/playbook/__init__.py", line 497, in _run_task
    results = self._run_task_internal(task, include_failed=include_failed)
  File "/Library/Python/2.7/site-packages/ansible/playbook/__init__.py", line 439, in _run_task_internal
    results = runner.run()
  File "/Library/Python/2.7/site-packages/ansible/runner/__init__.py", line 1466, in run
    p = utils.plugins.action_loader.get(self.module_name, self)
  File "/Library/Python/2.7/site-packages/ansible/utils/plugins.py", line 218, in get
    self._module_cache[path] = imp.load_source('.'.join([self.package, name]), path)
  File "/Users/wicaeed/Projects/wicaeed-ansible-playbooks/action_plugins/merge_vars.py", line 8, in <module>
    from ansible.plugins.action import ActionBase
ImportError: No module named plugins.action

Are there any minimum requirements for Ansible? I'm currently on 1.9.6

Posibility to install it directly from git

I like my ansible stuff to be as most self including as possible.

On the other hand I do never use that out-of-band package managers like pip or bundler or others that create more security holes as they close and pulling stuff that I don't like to have in multiple versions.

Then I usually use ansible in a pull way. So all libraries that are needed must come with the system or are not usable.

For that I like to use git-subtrees to install stuff inside the environment where it is used. Unfortunatelly the need to write a python script that includes stuff from an out-of-band directories does not play well in this respects. (Added with my restricted knowledge of python to maybe work around such restrictions.)

Maybe you could include such a boot strap plugin that could be included into ansible playbook git to add the functionality.

leapfrogonline / ansible-merge-vars Goto Github PK

ansible-merge-vars's People

Contributors

Stargazers

Watchers

Forkers

ansible-merge-vars's Issues

Recommend Projects

Recommend Topics

Recommend Org