networktocode / network-importer Goto Github PK

The network importer is a tool/library to analyze and/or synchronize an existing network with a Network Source of Truth (SOT), it's designed to be idempotent and by default it's only showing the difference between the running network and the remote SOT.

License: Other

Makefile 0.39% Python 99.14% Dockerfile 0.09% Jinja 0.38%

network diffsync batfish source-of-truth nornir netbox

network-importer's Introduction

Network Importer

The main use cases for the network importer:

Import an existing network into a SOT (Nautobot or NetBox) as a first step to automate a brownfield network
Check the differences between the running network and the Source of Truth

This application is intended to run outside of Nautobot.

Quick Start

Questions

For any questions or comments, please feel free to open an issue or swing by the networktocode slack channel. (channel #networktocode)

network-importer's People

Contributors

Stargazers

Watchers

network-importer's Issues

Network Importer Fails to import VLANs with same id at same site

Environment

Python version: 3.7.5
Network Importer version: Early

Steps to Reproduce

In netbox create a site with 2 vlans with the same ID, different prefixes
Run network importer to import that site

Expected Behavior

Both VLANs should be imported

Observed Behavior

Traceback (most recent call last):
  File "/usr/local/bin/network-importer", line 5, in <module>
    main()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/source/network_importer/cli.py", line 103, in main
    ni.init(limit=limit)
  File "/source/network_importer/performance.py", line 83, in timed
    result = method(*args, **kw)
  File "/source/network_importer/main.py", line 156, in init
    self.sot.init()
  File "/source/network_importer/adapters/netbox_api/adapter.py", line 97, in init
    self.import_netbox_vlan(site)
  File "/source/network_importer/adapters/netbox_api/adapter.py", line 153, in import_netbox_vlan
    self.add(vlan)
  File "/source/dsync/__init__.py", line 461, in add
    raise ObjectAlreadyExist(f"Object {uid} already present")
dsync.exceptions.ObjectAlreadyExist: Object site1__1000 already present

'NoneType' object has no attribute 'remote_id'

Environment

Python version: 3.7.9
Network Importer version: develop-2.0

I'm running network-importer across numerous sites, 1 site at a time.

Steps to Reproduce

run: network-importer --check --update-configs --limit="site=${site}"
followed by: network-importer --apply --update-configs --limit="site=${site}"

Expected Behavior

I expect all differences to be rectified in NetBox

Observed Behavior

About 50% of the time the apply errors with the following stack trace:

Traceback (most recent call last):
  File "/usr/local/bin/network-importer", line 5, in <module>
    main()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/source/network_importer/cli.py", line 113, in main
    ni.sync()
  File "/source/network_importer/main.py", line 168, in sync
    self.sot.sync_from(self.network, diff_class=NetworkImporterDiff)
  File "/source/dsync/__init__.py", line 398, in sync_from
    self._sync_from_diff_element(child, flags=flags, logger=log)
  File "/source/dsync/__init__.py", line 487, in _sync_from_diff_element
    self._sync_from_diff_element(child, flags=flags, parent_model=obj, logger=logger)
  File "/source/dsync/__init__.py", line 448, in _sync_from_diff_element
    obj = object_class.create(dsync=self, ids=element.keys, attrs={key: diffs[key]["src"] for key in diffs})
  File "/source/network_importer/adapters/netbox_api/models.py", line 135, in create
    nb_params = item.translate_attrs_for_netbox(attrs)
  File "/source/network_importer/adapters/netbox_api/models.py", line 114, in translate_attrs_for_netbox
    nb_params["lag"] = parent_interface.remote_id
AttributeError: 'NoneType' object has no attribute 'remote_id'

Add unit test for Netbox Inventory

Update pynetbox to version 5.x

Environment

Python version: 3.7.5
Network Importer version: 2.0.0-dev

Steps to Reproduce

Running apply for a specific site.

Expected Behavior

Netbox not to through an exception

Observed Behavior

The following exception occurred with the network importer:

Traceback (most recent call last):
  File "/usr/local/bin/network-importer", line 5, in <module>
    main()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/source/network_importer/cli.py", line 109, in main
    ni.sync()
  File "/source/network_importer/main.py", line 169, in sync
    self.sot.sync(self.network)
  File "/source/dsync/__init__.py", line 138, in sync
    self.sync_element(child)
  File "/source/dsync/__init__.py", line 161, in sync_element
    self.sync_element(child)
  File "/source/dsync/__init__.py", line 156, in sync_element
    self.create_object(object_type=element.type, keys=element.keys, params=element.source_attrs)
  File "/source/dsync/__init__.py", line 272, in create_object
    self._crud_change(action="create", keys=keys, object_type=object_type, params=params)
  File "/source/dsync/__init__.py", line 315, in _crud_change
    item = getattr(self, f"{action}_{object_type}")(keys=keys, params=params)
  File "/source/network_importer/adapters/netbox_api/adapter.py", line 431, in create_interface
    intf = self.netbox.dcim.interfaces.create(**nb_params)
  File "/usr/local/lib/python3.7/site-packages/pynetbox/core/endpoint.py", line 287, in create
    ).post(args[0] if args else kwargs)
  File "/usr/local/lib/python3.7/site-packages/pynetbox/core/query.py", line 381, in post
    return self._make_call(verb="post", data=data)
  File "/usr/local/lib/python3.7/site-packages/pynetbox/core/query.py", line 274, in _make_call
    raise RequestError(req)
pynetbox.core.query.RequestError: The request failed with code 500 Internal Server Error but more specific details were not returned in json. Check the NetBox Logs or investigate this exception's error attribute.

Migrate project to Poetry

Would like to migrate the project to Poetry to help with packaging and dependancies management.
https://python-poetry.org/

NI move interfaces from non master VC devices to the master

Environment

Python version: 3.7.7
Network Importer version: 2.0.0

Steps to Reproduce

Create 2 (or more) device Types - Ex: C9200L-48T-4X (with interfaces GigabitEthernet1/0/1 etc...) and C9200L-48T-4X_VC2 (with interfaces GigabitEthernet2/0/1 etc...)
Create each member of the VC by selecting the correct DeviceType depending of the node position in the VC
Each physical node will host is own interfaces and All interfaces will go back visible to the master device
- Scripts will updates interfaces from the master device
Interfaces updated by network-importer will be detached from the secondary unit and attached to the master device

Expected Behavior

Interface should stay on the initial device

Observed Behavior

Interfaces are moved to the master unit of the VC

Refactor Netbox Inventory to use pynetbox

Network importer fails with duplicate IP when utilizing a device from the onboarding plugin

Environment

Python version: 3.7.9
Network Importer version: 2.0.0-beta3

When running network-importer on a device which was added by the onboarding plugin the IP address fails to be added by network importer as it has already been added. I'm not sure if this is a bug with the onboarding plugin or network importer, but the interface was added by the onboarding plugin as vlan 6 but network importer wants to add the IP to a device Vlan6 this fails as it has already been added. Changing the name of the interface from the onboarding plugin from vlan 6 to Vlan6 is a workaround for the moment, but the 2 plugins should be consistent. The following error is from Network Importer:

2020-12-26 19:47:16,275 - network-importer - INFO - Created interface Vlan6 (126) in NetBox
2020-12-26 19:47:16,275 - diffsync.helpers - ERROR - 2020-12-26 19:47.16 [error    ] Failed to create ip_address {'address': '10.12.48.28/24'} - it already exists! [diffsync.helpers] action=create diffs={'+': {'interface_name': 'Vlan6', 'device_name': 'ciscosw2'}} dst=<Netbox> flags=<DiffSyncFlags.NONE: 0> model=ip_address src=<Network> status=error unique_id=192.168.48.2/24

Steps to Reproduce

Import device with Network Importer (Cisco SG300 in my case)
Device gets added with IP on vlan 6
Run network-importer apply, network importer fails with IP already exists.

Expected Behavior

The IP should not be re-added by network importer

Observed Behavior

IP added by the onboarding plugin was for interface vlan 6 but network-importer wants to add the IP address to a new interface it creates Vlan6

Add documentation to explain how to extend the default Adapters

Environment

Network Importer version: 2.0.0-dev

Proposed Functionality

Add documentation to explain how to extend the default adapters

Upgrade Nornir to 3.x

Environment

Python version: 3.7.7
Network Importer version: 2.0.0-beta2

Proposed Functionality

Upgrade Nornir and the inventory to the latest major version : 3.x

Use Case

Dependency management, house keeping

Configuration Changes

Add support for import_vlan in Netbox 2.9

Environment

Python version: 3.7.7
Network Importer version: 2.0.0-dev

Proposed Functionality

Add support for Netbox 2.9 and specifically the new format for tags in the API

Use Case

Support the latest version of netbox for all features
Currently Network-importer is reading/updating the tags on the vlan to keep track of the device<>vlan relationship which means that import_vlan won't work with 2.9

Configuration Changes

We need to introduce a flag in the configuration to identify the version of Netbox

Validate Port-Channel interface parsing

This is only to mark this issue as "to verify"

Customer had a "Port-Channel" (uppercase C letter) network-imported, however Cisco device lists it with interface Port-channel (smallercase C).

This is track this issue and validate a reason of this discrepancy .

Add support for Prefixes

Proposed Functionality

Currently the network-importer supports VLAN and IPs but not Prefixes.
The proposal is to add support for Prefixes as well.

To be consistent with VLANs, Prefixes should be managed per site for now.

Configuration Changes

Need to add a flag in the configuration to turn this feature on/off, should be off by default

AttributeError: 'NoneType' object has no attribute 'id'

Environment

Run in Docker based on docker-compose of the project
Python version: "3.7.7"
Network Importer version: (2, 0, "0-beta3")
Netbox version : v2.9.7

Steps to Reproduce

Configuration inventory section filters on one device (tried one in VC and no VC same error)
network-importer inventory --config /source/config/network_importer.toml

Behavior

[root@mtlbmepvlxcdkr01 network-importer]# docker exec -ti network_importer network-importer inventory --config /source/config/network_importer.toml
Traceback (most recent call last):
  File "/usr/local/bin/network-importer", line 5, in <module>
    main()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/local/network_importer/cli.py", line 185, in inventory
    ni.build_inventory(limit=limit)
  File "/local/network_importer/performance.py", line 67, in timed
    result = method(*args, **kw)
  File "/local/network_importer/main.py", line 107, in build_inventory
    "conn_timeout": config.SETTINGS.network.conn_timeout,
  File "/usr/local/lib/python3.7/site-packages/nornir/init_nornir.py", line 78, in InitNornir
    **conf.inventory.options,
  File "/usr/local/lib/python3.7/site-packages/nornir/core/deserializer/inventory.py", line 129, in deserialize
    deserialized = cls(*args, **kwargs)
  File "/local/network_importer/inventory.py", line 133, in __init__
    if dev.id != dev.virtual_chassis.master.id:
AttributeError: 'NoneType' object has no attribute 'id'

Physical interface reported as Virtual

In some cases, we found some 40G interfaces reported as virtual by the NI, need to investigate further
On this device, only the 40G interfaces are impacted (FortyGigabitEthernet1/1/1) but the other interfaces are fine (GigabitEthernet1/1/1 / TenGigabitEthernet1/1/1)

Device: C9300-24P
OS: ios

Update project to align with latest ntc standard for python

Environment

Python version: 3.7.7
Network Importer version: 2.0.0-dev

Proposed Functionality

We need to update this project to use invoke and the latest linters from our standard python project.
I think we should wait until the update of the cookiecutter-ntc-python project is done

Use Case

Standardization

Add support for SSL controls

The [netbox] configuration should provide options to:

disable SSL verify
provide SSL certificate file path

Also control with environment variables.

Add support for generate_hostvars in 2.0

Environment

Network Importer version: 2.0.0-dev

Proposed Functionality

In 1.x, network importer could generate a variable file per device that contains all information extracted by Batfish (BGP, ACL etc .. )
This feature is not currently available in 2.0 and should be added back at some point.

Use Case

Expose all data parsed by Batfish in a simple format that can easily integrate with other tools (ansible, Nornire etc ..)
Backward compatibility

Configuration Changes

Add some flags in the configuration to control when and where the variable files should be created

Parametrize the Dockerfile

Environment

Network Importer version: 0.5.0-dev

Proposed Functionality

Right now the version of python is hardcoded in the Dockerfile, it would be useful to convert that as an argument to be able to build and test the project with multiple version of Python

Use Case

Support for version of python with minimal effort

Configuration Changes

No changes

Add support for Netbox 2.7

Would be the perfect opportunity to create a second backend SOT. Netbox2.6 and Netbox2.7.
It will force s to refactor the code properly to support multiple backend

Add unit test around NI config management

Ensure default values are properly working
Ensure user defined, env var and config inputs are properly processed
...

Multiple Filters only filters last filter

Environment

Python version: 3.7.7
Network Importer version: 2.0.0

Steps to Reproduce

Install network-importer
Set the filter to site=msp-1,site=msp-2
Only the site msp-2 devices would be filtered to

Expected Behavior

Expected based on docs to get both sites in the filter list

Observed Behavior

Only the items at msp-2 (whichever is last) would be used.

Add support for Batfish session parameters

From the pybatfish project

host: The host of the batfish service
port_v1: The port batfish service is running on (9997 by default)
port_v2: The additional port of batfish service (9996 by default)
ssl: Whether to use SSL when connecting to Batfish (False by default)
api_key: Your API key

This will allow to run multiple version of Batfish on the same host for testing and it will add the support for Batfish Enterprise

ER: support NetBox device status assignment based on outcome

When the network-importer (NI) tool processes a given device there are three general outcomes that could be used to assign the device state in NetBox:

import process was successful
import process failed to connect to device, therefore cannot import
import process failed during the processing of the device

This ER proposes a change to the NI configuration file so that the User can identify the device status value (slug) that NI will assign upon those outcomes. The following is an example.

Note: the variable names (when_device_unreachable) are provided for example purposes only and could be different in the eventual implementation.

[netbox]
when_import_unreachable = "Offline"
when_import_fail = "Failed"
when_import_pass = "Active"

If any of these when variables are not provided, then the NI process will not modify the device status value; event when the import completes without error.

Set return code on --check

Set an appropriate return code based on the number of updates needed, or just a 1 if there is a difference between the local and remote device.

Purpose

Ability to use as a system check inside of a CI tool, so if a build fails that the "audit" would fail

When network importer tries to create an interface in NetBox an exception is thrown

Environment

Python version: 3.7
Network Importer version: develop-2.0 branch

When an interface is missing in NetBox and network-importer tries to create it I see this exception:

Traceback (most recent call last):
  File "/usr/local/bin/network-importer", line 5, in <module>
    main()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/source/network_importer/cli.py", line 113, in main
    ni.sync()
  File "/source/network_importer/main.py", line 168, in sync
    self.sot.sync_from(self.network, diff_class=NetworkImporterDiff)
  File "/source/dsync/__init__.py", line 393, in sync_from
    self._sync_from_diff_element(child, flags=flags, logger=log)
  File "/source/dsync/__init__.py", line 479, in _sync_from_diff_element
    self._sync_from_diff_element(child, flags=flags, parent_model=obj, logger=logger)
  File "/source/dsync/__init__.py", line 443, in _sync_from_diff_element
    obj = object_class.create(dsync=self, ids=element.keys, attrs={key: diffs[key]["src"] for key in diffs})
  File "/source/network_importer/adapters/netbox_api/models.py", line 134, in create
    item = super().create(**ids, dsync=dsync, **attrs)
TypeError: create() got an unexpected keyword argument 'name'

Steps to Reproduce

Remove an interface from NetBox
Run apply which should create an interface in NetBox

Expected Behavior

The interface should be created

Observed Behavior

Add integration tests based on example projects

Environment

Network Importer version: 2.0.0-dev

Proposed Functionality

It would be really helpful to have some integration tests to validate the end-to-end workflow as part of the CI pipeline to validate the integration with Batfish and NetBox.

Currently there are 2 example projects in the examples directory that could be used for this, the workflow could be as follow:

Start Batfish container
Start Netbox-Docker
Run ansible playbook to create inventory in netbox
Run network-importer in apply mode
Run network-importer in check mode and ensure the diff is clean

Steps 3 to 5 should be repeated per project

Use Case

Improve the test coverage and test some part of the code that are not been tested today.
Ensure the network importer is working as expected with different version of Netbox and/or Batfish

Configuration Changes

deepcopy fails in models.py occasionally

Environment

Python version: 3.7.9
Network Importer version: develop-2.0

Occasionally when running apply network-importer throughs a stack trace which looks like it comes form network_importer/models.py line 148 and is caused by a deepcopy.

Steps to Reproduce

Run apply on various sites until it happens

Expected Behavior

No exception to be thrown

Observed Behavior

Occasionally when running apply we see:

Traceback (most recent call last):
  File "/usr/local/bin/network-importer", line 5, in <module>
    main()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/source/network_importer/cli.py", line 113, in main
    ni.sync()
  File "/source/network_importer/main.py", line 168, in sync
    self.sot.sync_from(self.network, diff_class=NetworkImporterDiff)
  File "/source/dsync/__init__.py", line 398, in sync_from
    self._sync_from_diff_element(child, flags=flags, logger=log)
  File "/source/dsync/__init__.py", line 448, in _sync_from_diff_element
    obj = object_class.create(dsync=self, ids=element.keys, attrs={key: diffs[key]["src"] for key in diffs})
  File "/source/network_importer/adapters/netbox_api/models.py", line 477, in create
    item = super().create(ids=ids, dsync=dsync, attrs=attrs)
  File "/source/dsync/__init__.py", line 177, in create
    return cls(**ids, dsync=dsync, **attrs)
  File "/source/network_importer/models.py", line 148, in __init__
    new_kwargs = copy.deepcopy(kwargs)
  File "/usr/local/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.7/copy.py", line 241, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.7/copy.py", line 281, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.7/copy.py", line 241, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.7/copy.py", line 281, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.7/copy.py", line 241, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.7/copy.py", line 281, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.7/copy.py", line 221, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/local/lib/python3.7/copy.py", line 221, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/local/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.7/copy.py", line 241, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.7/copy.py", line 307, in _reconstruct
    value = deepcopy(value, memo)
  File "/usr/local/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.7/copy.py", line 281, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.7/copy.py", line 221, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/local/lib/python3.7/copy.py", line 221, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/local/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.7/copy.py", line 241, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.7/copy.py", line 307, in _reconstruct
    value = deepcopy(value, memo)
  File "/usr/local/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.7/copy.py", line 281, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.7/copy.py", line 221, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/local/lib/python3.7/copy.py", line 221, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/local/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.7/copy.py", line 241, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.7/copy.py", line 281, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.7/copy.py", line 241, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.7/copy.py", line 281, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.7/copy.py", line 241, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.7/copy.py", line 281, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.7/copy.py", line 241, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.7/copy.py", line 169, in deepcopy
    rv = reductor(4)
TypeError: can't pickle _thread.lock objects

Migrate CLI to Click

Environment

Python version: 3.7.7
Network Importer version: 0.5.0dev

Proposed Functionality

Currently the CLI is build on argparse, in order to align with the other project at NTC we should migrate the CLI to Click

Use Case

The main use case is to align with the other projects at NTC to simplify the maintenance moving forward

Configuration Changes

No config change

After apply still reporting diff in description with trailing whitespace

Environment

Python version: 3.7.9
Network Importer version: develop-2.0

After running apply, then a followup check, some interfaces still show differences only by a trailing whitespace:

site
device
  device:junosdevice.name
    interface
      interface: interfacename.0
        description   Network(DESCRIPTIONTEXT )   Netbox(DESCRIPTIONTEXT)
      interface: interfacename
        description   Network(DESCRIPTIONTEXT )   Netbox(DESCRIPTIONTEXT)

Steps to Reproduce

Run check
Run apply
Run check

NOTE: Only happens on some interfaces on some junos devices

Expected Behavior

In this case the description is off by 1 trailing whitespace character, I'm not sure where this is coming from, but I believe they are equal and should not be reported as diffs.

Observed Behavior

Interface description flagged as diff.

Add support for Link Discovery based on LLDP/CDP

Add support for Python 3.6

Environment

Python version: 3.7
Network Importer version: 2.0.0-beta2

Proposed Functionality

With #145, network-importer can run on Python 3.7 and 3.8 but we are still missing 3.6.
After doing some research, I think the 2 main dependencies that are preventing this project to support 3.6 are:

Diffsync, currently it requires 3.7+ but the next version should support 3.6 as well
Pydantic, doesn't work with 3.6 by default but it's possible to support 3.6 if we install dataclasses as well.

Use Case

Support more environments

Configuration Changes

Getting configuration file error when using BATFISH_ADDRESS in environment variable

Environment

Python version: python:3.7.5
Network Importer version: 0.5.0 per network-importer --version

Steps to Reproduce

Start docker-compose, with env vars set
Exec into importer
Try running network-importer

Expected Behavior

Execution

Observed Behavior

Getting the error:

# network-importer --check --update-configs
Found 1 error(s) in the configuration file (network_importer.toml)
  True is not of type 'string' in batfish/address

Config File:

[main]
import_ips = true
import_prefixes = true
import_intf_status = true


# Vlans (name, vlan-id) can be imported from the configuration, from the CLI or both
# - "config" will import the vlans just from the configuration
# - "cli" will import the vlans from the cli using the action `get_vlans`
# - "true" will import vlans from both cli and config
# - no or false will not import any vlans
# The association between interface and vlans will always be derived from the configuration.
import_vlans = false         # Valid options are ["cli", "config", "no", true, false]

# Cabling can be imported from LLDP, CDP or the configuration (for some point to point links)
# - "lldp" or "cdp" will import the vlans from the cli using the action `get_neighbors`
# - "config" will import the neighbors from the configuration (for point to point links)
# - "true" will import neighbors from both cli and config
# - no or false will not import any neighbors
import_cabling = "lldp"         # Valid options are ["lldp", "cdp", "config", "no", true, false]
excluded_platforms_cabling = ["cisco_asa"]

# Number of nornir tasks to execute at the same tim
nbr_workers= 25

# Directory where the configuration can be find, organized in Batfish format
configs_directory = "configs"

[netbox]
supported_platforms = [ "cisco_ios", "cisco_nxos" ]

[network]
# Connection parameters for Netmiko
global_delay_factor = 5
banner_timeout = 15
conn_timeout = 5

# List of valid FQDN that can be found in the network,
# The FQDNs in this list will be automatically removed from all neighbords discovered from LLDP/CDP
fqdns = [ ]

[batfish]
network_name = "network-importer"
snapshot_name = "latest"
port_v1 = 9997
port_v2 = 9996
use_ssl = false


[inventory]
# The default method is to use the primary IP defined in netbox
# As an alternative it's possible to use the name of the device and provide your own FQDN.
use_primary_ip = true
fqdn = []

# Optional filter to limit the scope of the inventory, takes a comma separated string of key value pair"
#filter = "site=XXX,site=YYY,status=active"    # Alternative Env Variable : INVENTORY_FILTER

# Configure what Inventory will be loaded bu the network importer.
inventory_class = "network_importer.inventory.NetboxInventory"


[logs]
# Control the level of the logs printed to the console.
level = "info"        # "debug", "info", "warning"

# For each run, a performance log can be generated to capture how long
# some functions took to execute

Import device with management IP

Environment

Python version: python3.7
Network Importer version: 2.0b2 (Docker latest )

When i am importing to netbox it cant delete management IP address from netbox but it try to recreate it and failed

2020-12-07 21:18:48,479 - network-importer - INFO - Updated Prefix 192.168.5.0/24 (1) in NetBox
2020-12-07 21:18:48,649 - network-importer - WARNING - Unable to delete IP Address 192.168.5.200/24 on leaf1-a-lab, because it's currently the management IP address
2020-12-07 21:18:48,660 - diffsync - WARNING - 2020-12-07 21:18.48 [warning ] Non-fatal failure encountered [diffsync] action=delete diffs={'src': {}, 'dst': {'interface_name': 'vme', 'device_name': 'leaf1-a-lab'}} dst= flags=<DiffSyncFlags.NONE: 0> model=ip_address src= status=failure unique_id=192.168.5.200/24
2020-12-07 21:18:48,660 - diffsync - WARNING - 2020-12-07 21:18.48 [warning ] Not syncing children [diffsync] action=delete diffs={'src': {}, 'dst': {'interface_name': 'vme', 'device_name': 'leaf1-a-lab'}} dst= flags=<DiffSyncFlags.NONE: 0> model=ip_address src= unique_id=192.168.5.200/24
2020-12-07 21:18:48,661 - diffsync - ERROR - 2020-12-07 21:18.48 [error ] Failed to create ip_address {'address': '192.168.5.200/24'} - it exists! [diffsync] action=create diffs={'src': {'interface_name': 'vme.0', 'device_name': 'leaf1-a-lab'}, 'dst': {}} dst= flags=<DiffSyncFlags.NONE: 0> model=ip_address src= status=error unique_id=192.168.5.200/24
Traceback (most recent call last):
File "/usr/local/bin/network-importer", line 5, in
main()
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/local/network_importer/cli.py", line 111, in apply
ni.sync()
File "/local/network_importer/main.py", line 169, in sync
self.sot.sync_from(self.network, diff_class=NetworkImporterDiff)
File "/usr/local/lib/python3.7/site-packages/diffsync/init.py", line 484, in sync_from
self._sync_from_diff_element(child, flags=flags, logger=log)
File "/usr/local/lib/python3.7/site-packages/diffsync/init.py", line 577, in _sync_from_diff_element
self._sync_from_diff_element(child, flags=flags, parent_model=obj, logger=logger)
File "/usr/local/lib/python3.7/site-packages/diffsync/init.py", line 577, in _sync_from_diff_element
self._sync_from_diff_element(child, flags=flags, parent_model=obj, logger=logger)
File "/usr/local/lib/python3.7/site-packages/diffsync/init.py", line 533, in _sync_from_diff_element
raise ObjectNotCreated(f"Failed to create {object_class.get_type()} {element.keys} - it exists!")
diffsync.exceptions.ObjectNotCreated: Failed to create ip_address {'address': '192.168.5.200/24'} - it exists!

Steps to Reproduce

Have device with management ip in netbox and try to import it

Expected Behavior

It should ignore this prefix if it cant create it

Observed Behavior

It throw a exception and exit

Add support for device<>vlan association in 2.x version

Environment

Python version: 3.7.7
Network Importer version: 2.0.0-dev

Proposed Functionality

Currently in the 2.x branch, network-importer is not capturing the relationship between device and vlan like it's doing in 1.x
In 1.x we are using one tag per device on each Vlan to capture the relationship.
We need to support the same feature in 2.x for Netbox 2.8 & 2.9

Use Case

Feature parity with 1.x

Configuration Changes

Cisco NXOS "show interface transceiver" fails to parse

The Cisco NXOS show interface transceivers fails to parse the following types of output:

Interface missing a `type` value

Ethernet2/3
    transceiver is present
    type is
    name is Arista Networks
    part number is QSFP-40G-LR4-AR
    revision is --
    serial number is GTZBAL00780
    nominal bitrate is 10300 MBit/sec
    Link length supported for SMF fiber is 10 km
    cisco id is --
    cisco extended id number is 196

Interface without a transceiver

Ethernet102/1/44
    transceiver is not applicable

Add --version option to command

Please add a --version option to the CLI.

Add support for changelog in 2.0

Environment

Python version: 3.7.7
Network Importer version: 2.0.0-dev

Proposed Functionality

In 1.x network importer could generate a changelog that will contains all changes made to the SOT for traceability.
This feature is not currently available in 2.0 and should be added back at some point.

One option would be to leverage the structured logging system supported by DiffSync.

Use Case

Track all changes made to the SOT for traceability.

Configuration Changes

Add latest tag to `make build`

Documentation around inventory/fqdn is misleading

Environment

Python version: 3.7.7
Network Importer version: 2.0.0-beta3

Observed Behavior

Documentation "Inventory section" let thinks it's a list expected but looks like to be a string
fqdn = []

[inventory]
# The default method is to use the primary IP defined in netbox 
# As an alternative it's possible to use the name of the device and provide your own FQDN.
use_primary_ip = true  
fqdn = []

Error

docker exec -ti network_importer network-importer inventory --config /source/config/network_importer.toml
Configuration not valid, found 1 error(s)
  inventory/fqdn | str type expected (type_error.str)

Change default import parameters

Environment

Network Importer version: 2.0.0-beta2

Proposed Functionality

Currently the network importer will import all interfaces, ip addresses, vlans and cables by default.
I think these default are probably too large, especially the cabling that can generate some messages that can be confusing for someone not familiar with this tool.
The proposal is to change the default parameters to only import interfaces and Ip addresses by default and deactivate vlans and cabling by default. These options remains available in the configuration file of course.

Use Case

Improve the user experience for new users,
The first time someone run the network importer the experience should be as smooth as possible.
Once the import of the interfaces and Ip addresses is working, it's possible to enable more import in the configuration.

Configuration Changes

Create list of failed devices to easily re-execute NI only on these devices

The idea is to provide something similar to what Ansible is providing with the .retry file.

KeyError: 'access_vlan' during apply

Environment

Python version: 3.7.9
Network Importer version: develop-2.0

I'm running network-importer across numerous sites, 1 site at a time.

Steps to Reproduce

run: network-importer --check --update-configs --limit="site=${site}"
followed by: network-importer --apply --update-configs --limit="site=${site}"

Expected Behavior

I expect all differences to be rectified in NetBox

Observed Behavior

About 50% of the time the apply errors with the following stack trace:

Traceback (most recent call last):
  File "/usr/local/bin/network-importer", line 5, in <module>
    main()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/source/network_importer/cli.py", line 113, in main
    ni.sync()
  File "/source/network_importer/main.py", line 168, in sync
    self.sot.sync_from(self.network, diff_class=NetworkImporterDiff)
  File "/source/dsync/__init__.py", line 398, in sync_from
    self._sync_from_diff_element(child, flags=flags, logger=log)
  File "/source/dsync/__init__.py", line 487, in _sync_from_diff_element
    self._sync_from_diff_element(child, flags=flags, parent_model=obj, logger=logger)
  File "/source/dsync/__init__.py", line 454, in _sync_from_diff_element
    obj = obj.update(attrs={key: diffs[key]["src"] for key in diffs})
  File "/source/network_importer/adapters/netbox_api/models.py", line 168, in update
    nb_params = self.translate_attrs_for_netbox(attrs)
  File "/source/network_importer/adapters/netbox_api/models.py", line 92, in translate_attrs_for_netbox
    if "mode" in attrs and attrs["mode"] in ["TRUNK", "ACCESS"] and attrs["access_vlan"]:
KeyError: 'access_vlan'

Add command to display the inventory in CLI

Environment

Python version: 3.7.7
Network Importer version: 2.0.0-dev

Proposed Functionality

Add a cli command to provide more visibility into the current inventory and the status of the device in Batfish.

What devices are present,
what is the platform,
is the device reachable,
is the config available,
is the device properly recognized in Batfish
and probably more

Use Case

Provide more visibility into the current inventory.
Currently it can be hard to understand why NI can't connect to a device.

Configuration Changes

Remove local copy of DSync

Environment

Python version: 3.7.7
Network Importer version: 2.0.0-dev

Proposed Functionality

Once DSync has been published, we need to remove the local copy from this repo

Upgrade Batfish to v2020.10.08

Environment

Network Importer version: 2.0.0-dev

Proposed Functionality

A new version of Batfish has been released recently, we should update this project to use v2020.10.08

Docker Compose Does Not Build Without Tweaks

Environment

Network Importer version: pre-release

Steps to Reproduce

Clone repo
Issue command docker-compose up -d
Work through issues

Expected Behavior

Standard build

Observed Behavior

Issues within docker-compose in relation to Dockerfile

Automatically create parent LAG if it does not exist

Environment

Python version: 3.7.5
Network Importer version:

Proposed Functionality

Today when a LAG member is imported, if the LAG does not exist it creates a warning and moves on. This requires network-importer to run 1 time to create the LAG interfaces and potentially a second time to add the LAG members to the LAG

Use Case

Any LAG interface that does not exist in netbox but does on the network will require network-importer be run 2x.

Configuration Changes

netbox_api adapter - IP address creation not working with Netbox 2.9

Environment

Python version: 3.7.7
Network Importer version: 2.0.0-dev

Steps to Reproduce

run network-importer --apply with some changes that includes a new IP address
run network-importer --check to ensure that all changes have been properly applied

Expected Behavior

The diff should return clean and all changes should have been made to the SOT in the first run

Observed Behavior

All IP addresses are still showing as missing in the Netbox.
After investigation the IP addresses have been created but they haven't been properly attached to the interface.
The Rest API to associate an Ip address with an interface changes in 2.9.

networktocode / network-importer Goto Github PK

network-importer's Introduction

Network Importer

Quick Start

Questions

network-importer's People

Contributors

Stargazers

Watchers

Forkers

network-importer's Issues

Environment

Steps to Reproduce

Expected Behavior

Observed Behavior

Environment

Steps to Reproduce

Expected Behavior

Observed Behavior

Environment

Steps to Reproduce

Expected Behavior

Observed Behavior

Environment

Steps to Reproduce

Expected Behavior

Observed Behavior

Environment

Steps to Reproduce

Expected Behavior

Observed Behavior

Environment

Proposed Functionality

Environment

Proposed Functionality

Use Case

Configuration Changes

Environment

Proposed Functionality

Use Case

Configuration Changes

Proposed Functionality

Configuration Changes

Environment

Steps to Reproduce

Behavior

Environment

Proposed Functionality

Use Case

Environment

Proposed Functionality

Use Case

Configuration Changes

Environment

Proposed Functionality

Use Case

Configuration Changes

Environment

Steps to Reproduce

Expected Behavior

Observed Behavior

Purpose

Environment

Steps to Reproduce

Expected Behavior

Observed Behavior

Environment

Proposed Functionality

Use Case

Configuration Changes

Environment

Steps to Reproduce

Expected Behavior

Observed Behavior

Environment

Proposed Functionality

Use Case

Configuration Changes

Environment

Steps to Reproduce

Interface missing a `type` value