ISSUE TYPE

• code question

SOFTWARE VERSIONS

python:

Python 3.8.13 (default, Mar 17 2022, 00:00:00)

hier_config:

hier-config==2.0.1
nornir-hier-config==0.1.0

SUMMARY

I would like to generate a remediation file for a Cisco Nexus device. I found that some lines seem to be omited when you choose nxos os.

STEPS TO REPRODUCE

Create two files:
running-config.txt:

tacacs-server deadtime 3
tacacs-server host 192.168.1.99 key 7 "Test12345"

template.txt:

tacacs-server host 192.168.1.98 key 0 Test135 timeout 3
tacacs-server host 192.168.100.98 key 0 test135 timeout 3

the script when i set nxos from a host.

from hier_config.host import Host
from hier_config import HConfig
host = Host(hostname='test', os='nxos')
running = HConfig(host=host)
running.load_from_file('running-config.txt')
compiled = HConfig(host=host)
compiled.load_from_file('template.txt')
rem = running.config_to_get_to(compiled)
for line in rem.all_children():
    print(line.cisco_style_text())

EXPECTED RESULTS

no tacacs-server deadtime 3
no tacacs-server host 192.168.1.99 key 7 "Test12345"
tacacs-server host 192.168.1.98 key 0 Test135 timeout 3
tacacs-server host 192.168.100.98 key 0 test135 timeout 3

ACTUAL RESULTS

tacacs-server host 192.168.1.98 key 0 Test135 timeout 3
tacacs-server host 192.168.100.98 key 0 test135 timeout 3

When I set host = Host(hostname='test', os='nxos') to host = Host(hostname='test', os='ios')

EXPECTED RESULTS

no tacacs-server deadtime 3
no tacacs-server host 192.168.1.99 key 7 "Test12345"
tacacs-server host 192.168.1.98 key 0 Test135 timeout 3
tacacs-server host 192.168.100.98 key 0 test135 timeout 3

ACTUAL RESULTS

no tacacs-server deadtime 3
no tacacs-server host 192.168.1.99 key 7 "Test12345"
tacacs-server host 192.168.1.98 key 0 Test135 timeout 3
tacacs-server host 192.168.100.98 key 0 test135 timeout 3

Exclude tags appears to omit some remediation steps. In the output below the commands:

interface Vlan3
  description switch_mgmt_10.0.3.0/24
interface Vlan4
  description switch_mgmt_10.0.4.0/24

should be present, but the interface parent commands are missing from the description commands.

>>> from hier_config import Host
>>> host = Host(hostname="aggr-example.rtr", os="ios")
>>> host.load_tags_from_file("tags_ios.yml")
>>> host.load_running_config_from_file("running_config.conf")
>>> host.load_generated_config_from_file("generated_config.conf")
>>> print(host.remediation_config_filtered_text(include_tags=[], exclude_tags=["manual"]))
vlan 3
  name switch_mgmt_10.0.3.0/24
vlan 4
  name switch_mgmt_10.0.4.0/24
  description switch_mgmt_10.0.3.0/24
  description switch_mgmt_10.0.4.0/24

Does anyone have knowledge of others extending this project to Juniper or Nokia or other devices?

hier_config has the ability to load many instances of remediation into a single object tree. This provides the basis for creating reports over large sets of hier_config remediation configs e.g.:

• How many instances of line vty 0 4 are there?
• Provide a merged report of all remediations by tag

Create documentation for this feature set.

• Find a way to use a single idempotency rule for the following:

client 10.1.1.1 server-key
client 10.2.2.2 server-key 

#e.g.
idempotent_commands:
- lineage:
  - re_search: ^client {client_ip} server-key 

• See what we can do to clarify this or improve HConfig behavior
  For the best results, idempotent command lineage rules should be a single match per rule e.g.

idempotent_commands:
- lineage:
  - startswith: hardware access-list tcam region
- lineage:
  - startswith: hardware profile tcam region

If you were to lump them together, an idempotent command check for hardware profile tcam region X would be True if hardware access-list tcam region Y were in the compiled_config. Much of the time, this is still okay but there are cases where it is not. In certain cases, this behavior can shadow command negations.

• Restructure Idempotent rules to not require the word lineage over and over

idempotent_commands:
- - startswith: aaa authentication attempts
- - startswith: aaa authentication login
- - startswith: aaa authentication enable
- - startswith: interface # Support for deep rules
  - startswith: description

CodeFactor found an issue: Redefining built-in 'file'

It's currently on:
hier_config\host.py:100
Commit f44aedb

EOS uses 3 spaces instead of 2 for indenting children.

CodeFactor found an issue: Trailing spaces

It's currently on:
.travis.yml:25

CodeFactor found an issue: init method from base class 'HConfigChild' is not called

It's currently on:
hier_config_init_.py:52

Currently, we have the capability to compare configurations with the goal of building remediation commands to get from one configuration state to another. It would be useful to provide a diff functionality that understands the hierarchy of network device configurations in a way that the diff command does not.

+ ntp server 192.168.1.1
- ntp server 192.168.1.2
interface fa0/1
  + ip address 172.16.1.1 255.255.255.0
  - ip address  172.16.2.1 255.255.255.0

I'm trying to ensure that an ACL gets created before any other commands. It seems that by lowering the order ensures that it will get created before everything else which has a value of 500. (As a test I've also changed it so it's a greater number than 500 and still makes no difference)

I've set the below in my options.yml file but it doesn't take affect, the output file always shows that the ACL is going to be applied in the same place.

ordering:
- lineage:
  - startswith:
    - ip access-list
  order: 1

Create a dynamic remediation that makes working with access-lists hitless.

CodeFactor found an issue: Unnecessary "else" after "return"

It's currently on:
hier_config\hc_child.py:38

CodeFactor found an issue: Use of unsafe yaml load. Allows instantiation of arbitrary objects. Consider yaml.safe_load().

It's currently on:
hier_config\host.py:225
Commit f44aedb

ref #14

Add support for IOS-XR configuration comments

router isis backbone
 set-overload-bit on-startup 180
 is-type level-2-only
 ! ISIS network number should be encoded with 0-padded loopback IP
 ! example:
 ! example: 
 net 49.0001.1921.2022.0222.00

I'm thinking the comments should be a property of the configuration line just below the comment lines.

How to remove comments:
https://supportforums.cisco.com/t5/xr-os-and-platforms/cisco-ios-xr-remove-comment/td-p/2546154

I do not get a match under ordering when I use a nested lineage (interface subcommands); but it works when I only match the parent line.

Here is my config data:
Running:

interface GigabitEthernet1/0/18
 ip access-group TEST in
!
ip access-list extended TEST
 permit tcp host 10.1.1.1 host 10.1.1.2 eq 443
!
end

Intended:

interface GigabitEthernet1/0/18
 ip access-group TESTING in
!
ip access-list extended TESTING
 permit tcp host 10.1.1.2 host 10.1.1.1 eq 443
!
end

Here is a non-working ordering

Options:

ordering:
- lineage:
  - startswith:
    - ip access-list
    - access-list
  order: 100
- lineage:
  - startswith: interface
  - startswith: ip access-group
  order: 200
- lineage:
  - startswith:
    - no ip access-list
    - no access-list
  order: 300

idempotent_commands:
- lineage:
  - startswith: interface
  - startswith:
    - ip access-group

Result:

"ip access-list extended TESTING",
"  10 permit tcp host 10.1.1.2 host 10.1.1.1 eq 443",
"no ip access-list extended TEST",    ## This should have a weight of 300 (it actually does based on teh next results) ##
"interface GigabitEthernet1/0/18",   ## The next two lines should have a weight of 200 (does not since the above is 300) ##
"  ip access-group TESTING in"

I tested without using a nested lineage and I get the expected results

Options:

ordering:
- lineage:
  - startswith:
    - ip access-list
    - access-list
  order: 100
- lineage:
  - startswith: interface      ## Only matching the parent line ##
  order: 200
- lineage:
  - startswith:
    - no ip access-list
    - no access-list
  order: 300

idempotent_commands:
- lineage:
  - startswith: interface
  - startswith:
    - ip access-group

Result:

"ip access-list extended TESTING",
"  10 permit tcp host 10.1.1.2 host 10.1.1.1 eq 443",
"interface GigabitEthernet1/0/18",
"  ip access-group TESTING in"
"no ip access-list extended TEST",

https://pythonwheels.com/

CodeFactor found an issue: The if statement can be replaced with 'return bool(test)'

It's currently on:
hier_config\hc_child.py:814

We need a fast/simple method of loading a deep series of children e.g.

deepest_child = config.add_children_deep([
    'router ospf 123',
    'area 0',
    'interface bundle-e1',
    'cost 100'])

deepest_child.text
> 'cost 100'

router ospf 123
  area 0
    interface bundle-e1
      cost 100

This would be handy for setting up tests.

Load default options, that can be extended upon with the hconfig_options.

• We can create a set of default options, by os, that are known good settings
• It would make it easier for users to create their own options of just the things they care about, rather than having to create a set of key, value pairs that may never get used.

Python 3.7
On Windows 10
Fresh git Fork and Clone of netdevops / hier_config

When I run the Unit Tests I get one failure

Testing started at 22:19 ...
C:\Users\Rich\.virtualenvs\hier_config-UnGuqua-\Scripts\python.exe "C:\Program Files\JetBrains\PyCharm Community Edition 2018.2.2\helpers\pycharm\_jb_unittest_runner.py" --path C:/Users/Rich/PycharmProjects/hier_config/tests
Launching unittests with arguments python -m unittest discover -s C:/Users/Rich/PycharmProjects/hier_config/tests -t C:\Users\Rich\PycharmProjects\hier_config\tests in C:\Users\Rich\PycharmProjects\hier_config\tests

C:\Users\Rich\.virtualenvs\hier_config-UnGuqua-\lib\site-packages\pep8.py:110: FutureWarning: Possible nested set at position 1
  EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]')

Error
Traceback (most recent call last):
  File "C:\Program Files (x86)\Python37-32\Lib\unittest\case.py", line 59, in testPartExecutor
    yield
  File "C:\Program Files (x86)\Python37-32\Lib\unittest\case.py", line 615, in run
    testMethod()
  File "C:\Users\Rich\PycharmProjects\hier_config\tests\test_hier_config.py", line 69, in test_load_from_file
    hier.load_from_file(myfile.name)
  File "C:\Users\Rich\PycharmProjects\hier_config\hier_config\hconfig.py", line 141, in load_from_file
    with open(file_path) as f:
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\Rich\\AppData\\Local\\Temp\\tmpv_hmd0_7'

C:\Users\Rich\PycharmProjects\hier_config\tests\test_host.py:32: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  cls.hconfig_tags = yaml.load(open(cls.tags_file))
C:\Users\Rich\PycharmProjects\hier_config\tests\test_host.py:32: ResourceWarning: unclosed file <_io.TextIOWrapper name='C:\\Users\\Rich\\PycharmProjects\\hier_config\\tests\\files\\test_tags_ios.yml' mode='r' encoding='cp1252'>
  cls.hconfig_tags = yaml.load(open(cls.tags_file))
C:\Users\Rich\PycharmProjects\hier_config\tests\test_host.py:33: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  cls.hconfig_options = yaml.load(open(cls.options_file))
C:\Users\Rich\PycharmProjects\hier_config\tests\test_host.py:33: ResourceWarning: unclosed file <_io.TextIOWrapper name='C:\\Users\\Rich\\PycharmProjects\\hier_config\\tests\\files\\test_options_ios.yml' mode='r' encoding='cp1252'>
  cls.hconfig_options = yaml.load(open(cls.options_file))


Ran 60 tests in 0.073s

FAILED (errors=1)

Process finished with exit code 1

The temp File is created and the contents are written to it, I am unable to open it manually, The security permissions show as full access for my user. If I copy the file, I can open the copy and inspect the contents. It appears to be a lock issue, unique to Windows. It is discussed here:
https://bugs.python.org/issue14243

The Fix would be to close the file before calling load_from_file, then manually clean up

CodeFactor found an issue: Unnecessary "else" after "return"

It's currently on:
hier_config\hc_child.py:814

When the below empty banner is defined, _load_from_string_lines() does not recognize the end of the banner resulting in AssertionError: we are still in a banner for some reason.

banner motd ##

If PyYaml is required, then this project should not support Python 3.4

https://pyyaml.org/wiki/PyYAML states:
Requirements
PyYAML requires Python 2.7 or Python 3.5+.

• loading running config
• loading compiled/ideal config
• building remediation

CodeFactor found an issue: Use of unsafe yaml load. Allows instantiation of arbitrary objects. Consider yaml.safe_load().

It's currently on:
tests\test_hier_config.py:41

CodeFactor found an issue: Unnecessary "else" after "return"

It's currently on:
hier_config\hc_child.py:198

There is currently no documentation besides some comments in the files them selves. We need to expand in the docs.

CodeFactor found an issue: The if statement can be replaced with 'return bool(test)'

It's currently on:
hier_config\hc_child.py:47

CodeFactor found an issue: Unnecessary "else" after "return"

It's currently on:
hier_config\hc_child.py:764

CodeFactor found an issue: Unnecessary "else" after "return"

It's currently on:
hier_config\hc_child.py:205

CodeFactor found an issue: The if statement can be replaced with 'return bool(test)'

It's currently on:
hier_config\hc_child.py:129

Test full configuration workflows in a circular way that ensures correctness of hier_config's remediation and future config generating features.

• Instances
  • One set of test files for each major supported OS - IOS, EOS, NXOS, IOSXR
• Testing flow
  • Load a full running config
    • Assert that loaded running config matches the file
  • Load a full generated config
    • Assert that loaded generated config matches the file
  • Generate the remediation config
    • Assert that remediation config matches the file
  • Generate the future config running.future(remediation)
    • Assert the future config equals the generated config
  • Generate the rollback config
    • Assert that the rollback config matches the file
  • Generate the rollback_future config future.future(rollback)
    • Assert the rollback_future equals the running config
• Config files
  • Generated (aka future)
  • Running (aka rollback_future)
  • Remediation
  • Rollback

Environment

• hier_config: v2.2.0

Proposed Functionality

As a network automation engineer, I'd appreciate the ability to modify the remediation configuration lines.

Use Case

This is useful for situations where a negation line needs to be in a certain format for the network OS to accept it. For example, removing an SNMP user on Cisco Nexus requires the line to be truncated after the username and will error if the full line is presented.

Create remediation fix up functions for IOS/NXOS/EOS/IOS-XR that safely remove unused objects such as:

• ACLs
• Prefix lists/sets
• object-groups
• VRFs
• AS Path Sets
• IPv6 general prefixes
• Routemaps
• Classmaps

CodeFactor found an issue: Unnecessary "else" after "return"

It's currently on:
hier_config\hc_child.py:47

CodeFactor found an issue: Unnecessary "else" after "return"

It's currently on:
hier_config\hc_child.py:129

CodeFactor found an issue: Use of unsafe yaml load. Allows instantiation of arbitrary objects. Consider yaml.safe_load().

It's currently on:
tests\test_hier_config.py:38

CodeFactor found an issue: Unnecessary "else" after "return"

It's currently on:
hier_config\hc_child.py:762

I am adding functionality for HPE Comware switches. However, as VLAN configuration mixes multiple vlans into the same configuration statement things are getting a bit complicated. I have written a pre/post processor for dealing with it. However from architectural standpoints this is a mess.

Could you please advise on a more generic approach to implement it?

See my request for comments branch on this: https://github.com/empi89/hier_config/tree/rfc-pre-and-postprocessing

As a network automation engineer, I want the minimum required package versions to be bumped to the latest versions to resolve conflicts with other automation platform packages.

Tagging rules can be written in such a way as to produce stranded children in the remediation config for a particular tag. While this is infrequent, it can be painful to track down.

• Under the HConfigChild class, populate the tag of a trunk HConfigChild object by recursively examining the tags of the leaf HConfigChild objects.

class HConfigChild(object):

@property
def tags(self):
    my_tags = set(self._tags)
    for child in self.children:
        my_tags.update(child.tags)
    return mytags

• Update add_tags, deep_append_tags, etc. as needed

Exaggerated example

- lineage
  - startswith: interface
  add_tag: interfaces_with_children
- lineage
  - startswith: interface
  - anything: null
  remove_tags: interfaces_with_children

Remediation config with interfaces_with_children tag:

interface fa0/1
interface fa0/2

CodeFactor found an issue: Redefining built-in 'file'

It's currently on:
hier_config\host.py:181
Commit f44aedb

In reference to #11, our team has been discussing Python 2.7 support, in addition to the Python 3.6 support that the code is currently written to. We certainly appreciate the contributions, but go back and forth on whether it makes sense to walk back code to support 2.7.

Our main concerns with supporting 2.7 are:

• Soon to be deprecated.

  • Python 2.7 will no longer be supported by the community after January 1, 2020
  • At least Redhat has announced that their next LTS release of RHEL will drop support for Python 2. Others will likely follow soon.
  • By providing 2.7 support, we provide less incentive for others to update their code to support 3.x

• Ability to take advantage of Python 3.x enhancements:

  • Undoing enhancements that are specific to Python 3. It seems counter intuitive to roll back Python 3 specific items to support a version of Python that will soon be deprecated.
  • Ability to use f and r strings make core more efficient and easier to read.
  • We've been looking at implementing type hinting. We'd like be able to take advantage of enforcing hints. In Python 2.7, we'd be limited to comment based type hinting, which would limit our ability to enforce hints.

Given this, we'd like to be able to support the community. We'd like to propose some options:

1. Continue supporting Python 3 only support out of the box.
2. Commit to supporting Python 2.7 and Python 3 until Jun 1, 2019. At which time, hier_config would likely be converted to Python 3 only. However, Python 2.7 versions would remain on pypi and the last release supporting Python 2.7 would be preserved in a python2 branch.
3. Create a fork of hier_config to specifically support Python 2.7.

While generating the remediation commands, also generate rollback configuration. The rollback should be tied to the remediation in such a way as to receive the same tags. Rollback config can be generated today by reversing the commands_to_get_to() operation.

Running config

hostname router1.dfw
interface fa1
  ip address 192.168.1.1 255.255.255.0

Compiled config

hostname router1-new.dfw
interface fa1
  ip address 192.168.2.1 255.255.255.0
interface fa2
  ip address 192.168.3.1 255.255.255.0

Remediation config

hostname router1-new.dfw
interface fa1
  ip address 192.168.2.1 255.255.255.0
interface fa2
  ip address 192.168.3.1 255.255.255.0

Rollback config

hostname router1.dfw
interface fa1
  ip address 192.168.1.1 255.255.255.0
no interface fa2

Maybe I'm missing something but the banner check will not work on Nexus devices. The format of my banner is this:

banner motd #
Multiline
text
here.
#

No matter how I run it or what tweaks I make, it has this result:

File "/home//lib64/python3.8/site-packages/hier_config/root.py", line 402, in _load_from_string_lines
assert not in_banner, "we are still in a banner for some reason"
AssertionError: we are still in a banner for some reason

Obviously it thinks it's still in the banner and not detecting the "#" as the end of the banner. I've tried on multiple Nexus devices with the same result. It works fine on IOS devices. I can provide more information if needed. If there's a set work around for this then maybe I've missed it.

More info:
hier-config==2.2.0
I'm also making sure I'm setting the style to "nxos".

Thanks!

Since going from hier-config 1.4.0 to 1.4.1 and using the very latest hconfig-remediation-ansible hier-config breaks with the following error:

ansible-playbook -i inventory test.pb.yml -v
Using /etc/ansible/ansible.cfg as config file

PLAY [all] *******************************************************************************************************************************

TASK [BUILD REMEDIATION CONFIG WITHOUT TAGS] *********************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: load_tags() got an unexpected keyword argument 'file'
fatal: [router1]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_3zgHIj/ansible_module_hconfig_remediation.py\", line 189, in <module>\n    main()\n  File \"/tmp/ansible_3zgHIj/ansible_module_hconfig_remediation.py\", line 167, in main\n    host.load_tags(hier_tags, file=False)\nTypeError: load_tags() got an unexpected keyword argument 'file'\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1}
        to retry, use: --limit @/storage1/ansible-hconfig/hconfig-remediation-ansible-master/example/test.pb.retry

PLAY RECAP *******************************************************************************************************************************
router1                    : ok=0    changed=0    unreachable=0    failed=1

Downgrading back to hier-config 1.4.0 everything works again.
Looks like the codefactor_fixes broke things