Giter Club home page Giter Club logo

netbox-agent's Introduction

Netbox agent Build Status

This project aims to create hardware automatically into Netbox based on standard tools (dmidecode, lldpd, parsing /sys/, etc).

The goal is to generate an existing infrastructure on Netbox and have the ability to update it regularly by executing the agent.

Features

  • Create virtual machines, servers, chassis and blade through standard tools (dmidecode)
  • Create physical, bonding and vlan network interfaces with IPs (IPv4 & IPv6)
  • Create IPMI interface if found
  • Create or get existing VLAN and associate it to interfaces
  • Generic ability to guess datacenters and rack location through drivers (cmd and file and custom ones)
  • Update existing Device and Interface
  • Handle blade moving (new slot, new chassis)
  • Handle blade GPU expansions
  • Automatic cabling (server's interface to switch's interface) using lldp
  • Local inventory using Inventory Item for CPU, GPU, RAM, RAID cards, physical disks (behind raid cards)
  • PSUs creation and power consumption reporting (based on vendor's tools)

Requirements

Inventory requirement

  • hpassacli
  • storcli
  • omreport

Installation

# pip3 install netbox-agent

Usage

The agent can be run from a shell and get its configuration from either the configuration file or environment variables.

Configuration values are overridden based on the following precedence: command line arguments (might include config file) > environment variables > default config file > defaults.

# netbox_agent -c /etc/netbox_agent.yaml --register
INFO:root:Creating chassis blade (serial: QTFCQ574502EF)
INFO:root:Creating blade (serial: QTFCQ574502D2) myserver on chassis QTFCQ574502EF
INFO:root:Setting device (QTFCQ574502D2) new slot on Slot 9 (Chassis QTFCQ574502EF)..
INFO:root:Interface a8:1e:84:f2:9e:6a not found, creating..
INFO:root:Creating NIC enp1s0f1 (a8:1e:84:f2:9e:6a) on myserver
INFO:root:Interface 02:42:7a:89:cf:a4 not found, creating..
INFO:root:Creating NIC br-07ea1e4a2f0e (02:42:7a:89:cf:a4) on myserver
INFO:root:Create new IP 172.19.0.1/16 on br-07ea1e4a2f0e
INFO:root:Interface a8:1e:84:f2:9e:69 not found, creating..
INFO:root:Creating NIC enp1s0f0 (a8:1e:84:f2:9e:69) on myserver
INFO:root:Create new IP 42.42.42.42/24 on enp1s0f0
INFO:root:Create new IP fe80::aa1e:84ff:fef2:9e69/64 on enp1s0f0
INFO:root:Interface a8:1e:84:cd:9d:d6 not found, creating..
INFO:root:Creating NIC IPMI (a8:1e:84:cd:9d:d6) on myserver
INFO:root:Create new IP 10.191.122.10/24 on IPMI

If you need, you can update only specific informations like:

  • Network
  • Inventory
  • Location
  • PSUs
# ip a add 42.42.42.43/24 dev enp1s0f1
# netbox_agent -c /etc/netbox_agent.yaml --update-network
INFO:root:Create new IP 42.42.42.43/24 on enp1s0f1
# netbox_agent --update-inventory
INFO:root:Creating Disk Samsung SSD 850 S2RBNX0K101698D

Configuration

# Netbox configuration
netbox:
 url: 'http://netbox.internal.company.com'
 token: supersecrettoken
 # uncomment to disable ssl verification
 # ssl_verify: false
 # uncomment to use the system's CA certificates
 # ssl_ca_certs_file: /etc/ssl/certs/ca-certificates.crt

# Network configuration
network:
  # Regex to ignore interfaces
  ignore_interfaces: "(dummy.*|docker.*)"
  # Regex to ignore IP addresses
  ignore_ips: (127\.0\.0\..*)
  # enable auto-cabling by parsing LLDP answers
  lldp: true

#
# You can use these to change the Netbox roles.
# These are the defaults.
#
#device:
# chassis_role: "Server Chassis"
# blade_role: "Blade"
# server_role: "Server"
# tags: server, blade, ,just a comma,delimited,list
# custom_fields: field1=value1,field2=value2#
#
# Can use this to set the tenant
#
#tenant:
# driver: "file:/tmp/tenant"
# regex: "(.*)"

## Enable virtual machine support
# virtual:
#   # not mandatory, can be guessed
#   enabled: True
#   # see https://netbox.company.com/virtualization/clusters/
#   cluster_name: my_vm_cluster

# Enable datacenter location feature in Netbox
datacenter_location:
 driver: "cmd:cat /etc/qualification | tr [A-Z] [a-z]"
 regex: "datacenter: (?P<datacenter>[A-Za-z0-9]+)"
# driver: 'cmd:lldpctl'
# regex: 'SysName: .*\.([A-Za-z0-9]+)'
#
# driver: "file:/tmp/datacenter"
# regex: "(.*)"

# Enable rack location feature in Netbox
rack_location:
# driver: 'cmd:lldpctl'
# match SysName: sw-dist-a1.dc42
# regex: 'SysName:[ ]+[A-Za-z]+-[A-Za-z]+-([A-Za-z0-9]+)'
#
# driver: "file:/tmp/datacenter"
# regex: "(.*)"

# Enable local inventory reporting
inventory: true

Specific workflow

Blades

Each vendor class has a is_blade method which is later used for Device creation using the Netbox parent/child feature.

The get_blade_slot method return the name of the Device Bay.

Certain vendors don't report the blade slot in dmidecode, so we can use the slot_location regex feature of the configuration file.

Some blade servers can be equipped with additional hardware using expansion blades, next to the processing blade, such as GPU expansion, or drives bay expansion. By default, the hardware from the expnasion is associated with the blade server itself, but it's possible to register the expansion as its own device using the --expansion-as-device command line parameter, or by setting expansion_as_device to true in the configuration file.

Drives attributes processing

It is possible to process drives extended attributes such as the drive's physical or logical identifier, logical drive RAID type, size, consistency and so on.

Those attributes as set as custom_fields in Netbox, and need to be registered properly before being able to specify them during the inventory phase.

As the custom fields have to be created prior being able to register the disks extended attributes, this feature is only activated using the --process-virtual-drives command line parameter, or by setting process_virtual_drives to true in the configuration file.

The custom fields to create as DCIM > inventory item Text are described below.

NAME            LABEL                      DESCRIPTION
mount_point     Mount point                Device mount point(s)
pd_identifier   Physical disk identifier   Physical disk identifier in the RAID controller
vd_array        Virtual drive array        Virtual drive array the disk is member of
vd_consistency  Virtual drive consistency  Virtual disk array consistency
vd_device       Virtual drive device       Virtual drive system device
vd_raid_type    Virtual drive RAID         Virtual drive array RAID type
vd_size         Virtual drive size         Virtual drive array size

In the current implementation, the disks attributes ore not updated: if a disk with the correct serial number is found, it's sufficient to consider it as up to date.

To force the reprocessing of the disks extended attributes, the --force-disk-refresh command line option can be used: it removes all existing disks to before populating them with the correct parsing. Unless this option is specified, the extended attributes won't be modified unless a disk is replaced.

It is possible to dump the physical/virtual disks map on the filesystem under the JSON notation to ease or automate disks management. The file path has to be provided using the --dump-disks-map command line parameter.

Anycast IP

The default behavior of the agent is to assign an interface to an IP. So two servers with anycasted IPs, running update mode, would only trigger IP's interface assignement in a loop.

In order to handle this case, user need to set Netbox IP's mode to Anycast so that the agent will create another one if it's present on another server.

Hardware

Tested on:

Virtual Machines

  • Hyper-V
  • VMWare
  • VirtualBox
  • AWS
  • GCP

Blades

  • PowerEdge MX7000
  • PowerEdge M1000e (your DeviceType should have slots named Slot 01 and so on)
  • PowerEdge MX740c
  • PowerEdge M640
  • PowerEdge M630
  • PowerEdge M620
  • PowerEdge M610

Pizzas

  • DSS7500

Blades

  • HP BladeSystem c7000 Enclosure G2 / G3 (your DeviceType should have slots named Bay 1 and so on)
  • HP ProLiant BL460c Gen8
  • HP ProLiant BL460c Gen9
  • HP ProLiant BL460c Gen10
  • HP ProLiant BL460c Gen10 Graphics Exp its expansion HP ProLiant BL460c Graphics Expansion Blade
  • HP Moonshot 1500 Enclosure (your DeviceType should have slots batch create with Bay c[1-45n1]) with HP ProLiant m750, m710x, m510 Server Cartridge

Pizzas

  • ProLiant DL380p Gen8
  • ProLiant SL4540 Gen8
  • ProLiant SL4540 Gen9
  • ProLiant XL450 Gen10

Blades

  • SBI-* and SBA-* should be supported, but I need dmidecode output example to support automatic blade location

Pizzas

  • SSG-6028R
  • SYS-6018R

Blades

  • QuantaMicro X10E-9N

Pizzas

  • Nothing ATM, feel free to send me a dmidecode or make a PR!

Known limitations

  • The project is only compatible with Linux. Since it uses ethtool and parses /sys/ directory, it's not compatible with *BSD distributions.
  • Netbox >=2.6.0,<=2.6.2 has a caching problem ; if the cache lifetime is too high, the script can get stale data after modification. We advise to set CACHE_TIME to 0.

Developing

If you want to run the agent while adding features or just for debugging purposes

# git clone https://github.com/Solvik/netbox-agent.git
# cd netbox-agent
# python3 -m netbox_agent.cli --register

On a personal note, I use the docker image from netbox-community/netbox-docker

# git clone https://github.com/netbox-community/netbox-docker
# cd netbox-docker
# docker-compose pull
# docker-compose up

netbox-agent's People

Contributors

adaichendt-tv1 avatar aluxima avatar bds-congnguyen avatar cyrinux avatar deathrabbit679 avatar freberkivra avatar gardar avatar geektophe avatar illes avatar kivrachristoffer avatar markd69 avatar morihaya avatar ramnes avatar renovate-bot avatar renovate[bot] avatar solvik avatar strus38 avatar sylr avatar thomasadavis avatar varriount avatar wolfman2g1 avatar ypid avatar z0nker avatar zagrebelsky avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

netbox-agent's Issues

Handle IPMI (bmc) as management

If ipmitool is available, we should parse its output to:

  • create the IPMI IP
  • associate it to a "IPMI" Interface on the Device
  • set it as management

Supermicro - divice mis-identification

the agent is recognizing Supermicro SYS-1029U-E1CRTP-2-LL09G as a chassis. This should be a 1U service not a chassis. Is there a specific override that I can set in the config file? I don't have any chassis in my infra.

Also identifying any servers with motherboard X11SPW-TF as a blade.

Error when running script on SuperMicro

sh# netbox_agent --debug -c /etc/netbox_agent.yaml

Traceback (most recent call last):
File "/usr/local/bin/netbox_agent", line 11, in
load_entry_point('netbox-agent==0.5.1.dev3+gbb7c8c8', 'console_scripts', 'netbox_agent')()
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev3+gbb7c8c8-py3.7.egg/netbox_agent/cli.py", line 39, in main
return run(config)
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev3+gbb7c8c8-py3.7.egg/netbox_agent/cli.py", line 21, in run
manufacturer = dmidecode.get_by_type('Chassis')[0].get('Manufacturer')
TypeError: get_by_type() missing 1 required positional argument: 'type_id'

dmideode reports:
root@inet-xen-215-13:/netbox/netbox-agent/netbox_agent# dmidecode -t chassis

dmidecode 3.2

Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.

Handle 0x0003, DMI type 3, 22 bytes
Chassis Information
Manufacturer: Supermicro
Type: Desktop
Lock: Not Present
Version: 0123456789
Serial Number: 0123456789
Asset Tag: To Be Filled By O.E.M.
Boot-up State: Safe
Power Supply State: Safe
Thermal State: Safe
Security Status: None
OEM Information: 0x00000000
Height: Unspecified
Number Of Power Cords: 1
Contained Elements: 0
SKU Number: To be filled by O.E.M.

netbox_agent bails out with error on lshw

Traceback (most recent call last):
File "/usr/local/bin/netbox_agent", line 11, in
load_entry_point('netbox-agent==0.5.1.dev3+gbb7c8c8.d20200216', 'console_scripts', 'netbox_agent')()
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev3+gbb7c8c8.d20200216-py3.7.egg/netbox_agent/cli.py", line 40, in main
return run(config)
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev3+gbb7c8c8.d20200216-py3.7.egg/netbox_agent/cli.py", line 35, in run
server.netbox_update(config)
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev3+gbb7c8c8.d20200216-py3.7.egg/netbox_agent/server.py", line 336, in netbox_update
self.inventory = Inventory(server=self)
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev3+gbb7c8c8.d20200216-py3.7.egg/netbox_agent/inventory.py", line 56, in init
self.lshw = LSHW()
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev3+gbb7c8c8.d20200216-py3.7.egg/netbox_agent/lshw.py", line 18, in init
self.hw_info = json.loads(data)
File "/usr/lib/python3.7/json/init.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.7/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 25 column 8 (char 679)

i am using the master branch with the manual applied fixes from the "fix" branch

Hardware inventory update

Screenshot from 2019-08-28 12-00-10

So, memory, disk, network, CPU all nice and neat.. NVME is also partially included - I have another node that has this..

It doesn't use dmi to do this, it uses the output of lshw to collect it.

I think I need to make the asset tags unique, so I was going to make it 'hostname:part'

thomas

Handle device update

A server can see its attributes change over the time, it's a normal lifecycle

A few examples are:

  • It can move from a rack to another rack or from a chassis to another one
  • new hostname
  • have a new NIC cards, remove old ones
  • change IPs on current cards
  • have more RAM, more disk (see #8)

Netbox agent crash on server register

As identified in #149, the agent crashed with the following output

File "/tmp/netbox/netbox_agent/cli.py", line 39, in run
   server.netbox_create_or_update(config)
 File "/tmp/netbox/netbox_agent/server.py", line 306, in netbox_create_or_update
   if server.name != self.get_hostname():
AttributeError: 'NoneType' object has no attribute 'name'

With Netbox v2.8.6

agent configuration help

I'm pretty embarrassed to ask this but I've spent hours trying to sort this out with no luck.

I get
ValueError: get() returned more than one result. Check that the kwarg(s) passed are valid for this endpoint or use filter() or all() instead.
when running python3 -m netbox_agent.cli -c ./agent.yaml --register

Here is the /etc/qualification file ( where I suspect the real issue is)

DATACENTER:

id: dmin-1 # I've also tried setting this to the id number in the data base (1)

agent.yaml

 # Netbox configuration

netbox:

url: 'http://netbox'

token: xxxx

# Network configuration

network:

 # Regex to ignore interfaces  

 ignore_interfaces: "(dummy.*|docker.*)"

 # Regex to ignore IP addresses

 ignore_ips: (127\.0\.0\..*)

 # enable auto-cabling by parsing LLDP answers

 lldp: true

## Enable virtual machine support  

# virtual:

#   # not mandatory, can be guessed

#   enabled: True

#   # see https://netbox.company.com/virtualization/clusters/

#   cluster_name: my_vm_cluster

# Enable datacenter location feature in Netbox

datacenter_location:

driver: "cmd:cat /etc/qualification | tr [a-z] [A-Z]"

regex: "DATACENTER: (?P<datacenter>[A-Za-z0-9]+)"

# driver: 'cmd:lldpctl'

# regex: 'SysName: .*\.([A-Za-z0-9]+)'

#

# driver: "file:/tmp/datacenter"

# regex: "(.*)"

# Enable rack location feature in Netbox

rack_location:

# driver: 'cmd:lldpctl'

# match SysName: sw-dist-a1.dc42

# regex: 'SysName:[ ]+[A-Za-z]+-[A-Za-z]+-([A-Za-z0-9]+)'

#

# driver: "file:/tmp/datacenter"

# regex: "(.*)"

# Enable local inventory reporting  

inventory: true

Cdrom size error

Hello,

The script seem to detect cdrom as a disk but as there is no size for it, i get the following error :

Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/tmp/netbox-agent/netbox_agent/cli.py", line 48, in
main()
File "/tmp/netbox-agent/netbox_agent/cli.py", line 44, in main
return run(config)
File "/tmp/netbox-agent/netbox_agent/cli.py", line 39, in run
server.netbox_create_or_update(config)
File "/tmp/netbox-agent/netbox_agent/server.py", line 296, in netbox_create_or_update
self.inventory.create_or_update()
File "/tmp/netbox-agent/netbox_agent/inventory.py", line 447, in create_or_update
self.do_netbox_disks()
File "/tmp/netbox-agent/netbox_agent/inventory.py", line 385, in do_netbox_disks
disks = self.get_hw_disks()
File "/tmp/netbox-agent/netbox_agent/inventory.py", line 326, in get_hw_disks
d['Size'] = '{} GB'.format(int(size / 1024 / 1024 / 1024))
TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'

Rework configuration & cli

We should try to make sure configuration's file arguments can be passed as CLI arguments like Kubernetes binaries

We should also enable specific behaviors like:

  • --create-missing: create missing blade chassis, network NICs, VLAN, IP, etc
  • --update-all
  • --update-{rack,datacenter,network,inventory}

Guess primary address

Netbox has the ability to flag an IP as Primary address for a Device

Since it's tied to the user workflow, I'd say we could enable this guess with a config argument and use the following workflow

  • parse the route table
  • find the source ip of the default route
  • flag this IP as Primary adresss

Handle Netbox API versions

At some point, Netbox versions will depreciate or change some behavior.
The project has been developed with Netbox 2.6

One way to change the behavior depending of the version will be to match the HTTP header API-Version:

โžœ  ~ curl -i https://netbox.company.com/api/
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 29 Aug 2019 15:58:27 GMT
Content-Type: application/json
Content-Length: 325
Connection: keep-alive
Vary: Accept, Cookie, Origin
Allow: GET, HEAD, OPTIONS
API-Version: 2.6
X-Frame-Options: SAMEORIGIN
P3P: CP="ALL DSP COR PSAa PSDa OUR NOR ONL UNI COM NAV"

Packaging

It would be great at some point if we could create packages for this project so it's easier for lazy sysadmins to install the agent

I assume that not everybody is a fan of installing packages and dependencies through pip

If Debian/RPM/etc packages were available it'd be a nice addition!

lshw output UnicodeDecodeError

  File "/usr/share/python/netbox-agent/lib/python3.4/site-packages/netbox_agent/lshw.py", line 16, in __init__
    'lshw -quiet -json'
  File "/usr/lib/python3.4/subprocess.py", line 736, in getoutput
    return getstatusoutput(cmd)[1]
  File "/usr/lib/python3.4/subprocess.py", line 717, in getstatusoutput
    data = check_output(cmd, shell=True, universal_newlines=True, stderr=STDOUT)
  File "/usr/lib/python3.4/subprocess.py", line 609, in check_output
    output, unused_err = process.communicate(inputdata, timeout=timeout)
  File "/usr/lib/python3.4/subprocess.py", line 947, in communicate
    stdout = _eintr_retry_call(self.stdout.read)
  File "/usr/lib/python3.4/subprocess.py", line 491, in _eintr_retry_call
    return func(*args)
  File "/usr/share/python/netbox-agent/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 164: ordinal not in range(128)```

Netbox Datacenter: None

Traceback (most recent call last):
File "/usr/local/bin/netbox_agent", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.5/dist-packages/netbox_agent/cli.py", line 44, in main
return run(config)
File "/usr/local/lib/python3.5/dist-packages/netbox_agent/cli.py", line 36, in run
server.print_debug()
File "/usr/local/lib/python3.5/dist-packages/netbox_agent/server.py", line 299, in print_debug
print('Netbox Rack:', self.get_netbox_rack())
File "/usr/local/lib/python3.5/dist-packages/netbox_agent/server.py", line 91, in get_netbox_rack
site_id=self.get_netbox_datacenter().id,

please advise whats wrong here.
he finds out datacetner correctly
and the rack as well.

netbox-2.7

Working with the beta, the interfaces types changed.. So that is some that needs to be fixed. Not sure what else is broken.

KeyError: 'interface:type'

Hi

Running on virtualbox.

Traceback (most recent call last): File "/usr/local/bin/netbox_agent", line 11, in <module> sys.exit(main()) File "/usr/local/lib/python3.6/dist-packages/netbox_agent/cli.py", line 44, in main return run(config) File "/usr/local/lib/python3.6/dist-packages/netbox_agent/cli.py", line 39, in run server.netbox_create_or_update(config) File "/usr/local/lib/python3.6/dist-packages/netbox_agent/virtualmachine.py", line 76, in netbox_create_or_update self.network.create_or_update_netbox_network_cards() File "/usr/local/lib/python3.6/dist-packages/netbox_agent/network.py", line 399, in create_or_update_netbox_network_cards interface = self.create_netbox_nic(nic) File "/usr/local/lib/python3.6/dist-packages/netbox_agent/network.py", line 246, in create_netbox_nic type = self.get_netbox_type_for_nic(nic) File "/usr/local/lib/python3.6/dist-packages/netbox_agent/network.py", line 162, in get_netbox_type_for_nic return self.dcim_choices['interface:type']['Virtual'] KeyError: 'interface:type'

Handle anycast IPs

Right now, the agent doesn't handle anycast IPs

If you two servers run the agent in update mode, the IP will move from one server to another in a loop.
So we'd require the user to manually create the IP in Netbox with the Anycast role so that the agent behave like this:

  • If IP exists and is in Anycast, create a new Anycast one
  • If IP exists and isn't assigned, take it
  • If server is decomissioned, the free IP will be taken

Update MAC address on pre-existing interfaces

Describe the bug
If a device already has interfaces with no MAC set (e.g. device type template interfaces) netbox-agent bails out. It seems it can't find the interface because it's searching by mac&name. I think it should be searching by device&name since this is enough to uniquely identify interfaces.

Expected behavior
MAC address should be updated for existing interfaces.

Configuration file
N/A

Environment:
N/A

Additional context
This is the patch I applied to get it to work for me. This probably breaks other stuff but at least shows what I mean.

# diff /usr/local/lib/python3.6/site-packages/netbox_agent/network.old.py /usr/local/lib/python3.6/site-packages/netbox_agent/network.py
145,155c145,149
<         if nic['mac'] is None:
<             interface = self.nb_net.interfaces.get(
<                 name=nic['name'],
<                 **self.custom_arg_id,
<             )
<         else:
<             interface = self.nb_net.interfaces.get(
<                 mac_address=nic['mac'],
<                 name=nic['name'],
<                 **self.custom_arg_id,
<             )
---
> 	# Should/can this query by device&name?
>         interface = self.nb_net.interfaces.get(
>             name=nic['name'],
>             **self.custom_arg_id,
>         )
409a404,409
>                 nic_update += 1
>
>             if nic['mac'] != getattr(interface, 'mac_address', ''):
>                 logging.info('Updating interface {interface} mac to: {mac}'.format(
>                     interface=interface, mac=nic['mac']))
>                 interface.mac_address = nic['mac']

Handle HPE BL460c Gen10

We used to have a clear dmidecode output with previous generation:

Handle 0x0030, DMI type 204, 20 bytes
HP ProLiant System/Rack Locator
    Rack Name: Bay-D22
    Enclosure Name: blade-D22-DC3
    Enclosure Model: BladeSystem c7000 Enclosure G2
    Enclosure Serial: CZ3332KXX
    Enclosure Bays: 16
    Server Bay: 2
    Bays Filled: 1

But now we need to parse this shittier output..

Handle 0x0066, DMI type 204, 20 bytes
OEM-specific Type
    Header and Data:
        CC 14 66 00 01 02 03 04 10 01 05 06 00 00 00 00
        00 00 00 00
    Strings:
        Z04b
        blade-z04b
        BladeSystem c7000 Enclosure G3
        7
        CZ29040XXX
        10.0.0.1

Handle IPv6 addresses

The project uses the Python netifaces library
Right now we use netifaces like this:

netifaces.ifaddresses(interface).get(netifaces.AF_INET)

We also need to query netifaces.AF_INET6 to get IPv6 addresses
It should work out of the box for Netbox as it handles IPv4 and IPv6 the same way in the API

config.py does not support netbox url with self-signed certificates

Hi
could you update config.py, so it can work with self-signed certificates?
Proposal:

  • add an option to netbox-agent: --ignore-ssl-verify

Modifications (in addition of adding the option):

  • in config.py line line 1
import requests
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

then line 80:

if (no_ssl_verify):
    session = requests.Session()
    session.verify = False

nb = pynetbox.api(
     url=get_config().netbox.url,
     token=get_config().netbox.token
)
if (no_ssl_verify):
    nb.http_session = session
return nb

Thanks!

Change network discovery behavior

We shouldn't hardcode the behavior of network interface discovery and instead use parameters in the configuration file to let the user choose what they want to ignore

That way we will manage corner cases of users having loopback IP (ipvs, bgp etc)
And also create vlan interfaces, which is not working right now

I propose something like:

network:
  ignore_interfaces: "(dummy.*|docker.*)"
  ignore_ips: "(127\.0\.0\..*)"

Handle Power Supply

An example of dmidecode output for multiple power supply

Handle 0x0066, DMI type 39, 22 bytes
System Power Supply
        Power Unit Group: 1
        Location: Not Specified
        Name: Power Supply 1
        Manufacturer: HPE
        Serial Number: 5FJCB0ALLAR795
        Asset Tag: Not Specified
        Model Part Number: 830272-B21
        Revision: Not Specified
        Max Power Capacity: 1600 W
        Status: Present, OK
        Type: Switching
        Input Voltage Range Switching: Auto-switch
        Plugged: Yes
        Hot Replaceable: Yes

Handle 0x0067, DMI type 39, 22 bytes
System Power Supply
        Power Unit Group: 1
        Location: Not Specified
        Name: Power Supply 2
        Manufacturer: HPE
        Serial Number: 5FJCB0ALLAR3CM
        Asset Tag: Not Specified
        Model Part Number: 830272-B21
        Revision: Not Specified
        Max Power Capacity: 1600 W
        Status: Present, OK
        Type: Switching
        Input Voltage Range Switching: Auto-switch
        Plugged: Yes
        Hot Replaceable: Yes

Handle 0x0068, DMI type 39, 22 bytes
System Power Supply
        Power Unit Group: 1
        Location: Not Specified
        Name: Power Supply 3
        Manufacturer: HPE
        Serial Number: 5FJCB0ALLAR2CL
        Asset Tag: Not Specified
        Model Part Number: 830272-B21
        Revision: Not Specified
        Max Power Capacity: 1600 W
        Status: Present, OK
        Type: Switching
        Input Voltage Range Switching: Auto-switch
        Plugged: Yes
        Hot Replaceable: Yes

Blade management

I've got several inputs saying that I should match the Slot name from dmidecode and not harcode the slot with Blade {{ slot_number }}

So we need to modify it the following way:

Dell

Handle 0x0202, DMI type 2, 14 bytes
Base Board Information
        Manufacturer: Dell Inc.
        Product Name: Not Specified
        Version: Not Specified
        Serial Number: .8CKL35J..05.
        Asset Tag: Not Specified
        Features:
                Board is removable
                Board is replaceable
        Location In Chassis: Slot 05
        Chassis Handle: 0x0300
        Type: Interconnect Board

And take Slot 05

Handle 0x0200, DMI type 2, 14 bytes
Base Board Information
        Manufacturer: Dell Inc.
        Product Name: 0MN3VC
        Version: A01
        Serial Number: .4866F5J.CN7016327F004Z.01a.
        Asset Tag:                               
        Features:
                Board is a hosting board
                Board is removable
                Board is replaceable
                Board is hot swappable
        Location In Chassis: Slot 01a    
        Chassis Handle: 0x0300
        Type: Server Blade

And take Slot 01a

HP

Handle 0x0030, DMI type 204, 20 bytes
HP ProLiant System/Rack Locator
        Rack Name: Bay-D22
        Enclosure Name: blade-D22-xx
        Enclosure Model: BladeSystem c7000 Enclosure G2
        Enclosure Serial: CZ3332KYYD
        Enclosure Bays: 16
        Server Bay: 2
        Bays Filled: 1

And take Bay 2

Server local inventory

It would be a nice feature to have the following component as Inventory Item:

  • CPU
  • Memory
  • RAID cards
  • Disks

One of the challenge will be to synchronize these infos

RaspeberryPi support - dmidecode crash

running netbox_agent on a rapsberry PI makes the program crash:

# netbox_agent -c nbagent.yaml --register
Traceback (most recent call last):
  File "/usr/local/bin/netbox_agent", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/netbox_agent/cli.py", line 44, in main
    return run(config)
  File "/usr/local/lib/python3.7/dist-packages/netbox_agent/cli.py", line 22, in run
    dmi = dmidecode.parse()
  File "/usr/local/lib/python3.7/dist-packages/netbox_agent/dmidecode.py", line 74, in parse
    _data = _parse(buffer)
  File "/usr/local/lib/python3.7/dist-packages/netbox_agent/dmidecode.py", line 216, in _parse
    raise ParseError("Unable to parse 'dmidecode' output")
netbox_agent.dmidecode.ParseError: Unable to parse 'dmidecode' output

root@node05:/home/pi# dmidecode
# dmidecode 3.2
Scanning /dev/mem for entry point.
# No SMBIOS nor DMI entry point found, sorry.

where I guess we could use other info like:

$cat /proc/cpuinfo
...
$cat /proc/device-tree/model
Raspberry Pi 4 Model B Rev 1.1
$ inix -F
inxi -F
System:    Host: node05.home.lab Kernel: 4.19.97-v7l+ armv7l bits: 32 Console: tty 1
           Distro: Raspbian GNU/Linux 10 (buster)
Machine:   Type: ARM Device System: Raspberry Pi 4 Model B Rev 1.1 details: BCM2835 rev: c03111
           serial: 10000000256decd8
Argument "Raspberry Pi 4 Model B Rev 1.1" isn't numeric in sprintf at /usr/bin/inxi line 6969.
CPU:       Topology: Quad Core model: ARMv7 v7l variant: cortex-a72 bits: 32 type: MCP
           Speed: 1500 MHz min/max: 600/1500 MHz Core speeds (MHz): 1: 1500 2: 1500 3: 1500 4: 1500
Graphics:  Device-1: bcm2835-vc4 driver: vc4_drm v: N/A
           Device-2: bcm2835-hdmi driver: N/A
           Display: server: No display server data found. Headless machine? tty: 120x48
           Message: Advanced graphics data unavailable in console for root.
Audio:     Device-1: bcm2835-audio driver: bcm2835_audio
           Device-2: bcm2835-hdmi driver: N/A
           Sound Server: ALSA v: k4.19.97-v7l+
Network:   Message: No ARM data found for this feature.
           IF-ID-1: eth0 state: down mac: xxxx
          IF-ID-2: wlan0 state: up mac: xxx

or even all usefull commands in there: https://raw.githubusercontent.com/raspberrypi/utils/master/raspinfo/raspinfo

Any way it could be supported in some way?
Thanks

_choices deprecated

Hiya there,

Trying to run this, it causes an error due to the _choices being removed in Netbox v2.8.0, as listed here
netbox-community/netbox#3416

So currently when updating I see the error *api/dcim/_choices/ could not be found.

Can we update to use the new OPTIONS method?

Add logging

We need to see what the scripts does (create, update, delete), logging should be mandatory

Blade moving

If you move a blade from a chassis to another chassis, we need to decomission the old one and update the new location

Supermicro big-twin and fat twins..

Got several different Supermicro's, here's some dmi information:

This is a Fat twin (4u, 8 nodes)

it's missing the slot information... not filled in; so we've been setting part of the hostname to chassis slot ie, the hostname is row-rack-chassis-node aka 'd8-r13-c1-n1' - row d8, rack 13, chassis 1, node 1

I also prefer to use the base board for the blade type, not the system information product name. allows me to keep the chassis separate from the blades.

System Information
	Manufacturer: Supermicro
	Product Name: SYS-F618R2-RTPT+
	Version: 0123456789
	Serial Number: S16604645724353
	UUID: 00000000-0000-0000-0000-0cc47a59ec4a
	Wake-up Type: Power Switch
	SKU Number: To be filled by O.E.M.
	Family: To be filled by O.E.M.

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
	Manufacturer: Supermicro
	Product Name: X10DRFR-NT
	Version: 1.10
	Serial Number: VM155S027997
	Asset Tag: To be filled by O.E.M.
	Features:
		Board is a hosting board
		Board is replaceable
	Location In Chassis: To be filled by O.E.M.
	Chassis Handle: 0x0003
	Type: Motherboard
	Contained Object Handles: 0

Handle 0x0003, DMI type 3, 25 bytes
Chassis Information
	Manufacturer: Supermicro
	Type: Other
	Lock: Not Present
	Version: 0123456789
	Serial Number: CF418AE16N30163
	Asset Tag: To Be Filled By O.E.M.
	Boot-up State: Safe
	Power Supply State: Safe
	Thermal State: Safe
	Security Status: None
	OEM Information: 0x00000000
	Height: Unspecified
	Number Of Power Cords: 1
	Contained Elements: 1
		<OUT OF SPEC> (0)
	SKU Number: To be filled by O.E.M.

Power DMI; there is actually 4 power supplies in this chassis, but
two are for 4 nodes, and the other 2 are for a different node.

System Power Supply
        Power Unit Group: 1
        Location: PSU1
        Name: PWS-1K62P-1R
        Manufacturer: SUPERMICRO
        Serial Number: P1K65CF11CT1837
        Asset Tag: N/A
        Model Part Number: PWS-1K62P-1R
        Revision: 1.3
        Max Power Capacity: 1620 W
        Status: Present, OK
        Type: Switching
        Input Voltage Range Switching: Auto-switch
        Plugged: Yes
        Hot Replaceable: No
        Input Voltage Probe Handle: 0x004A
        Cooling Device Handle: 0x004C
        Input Current Probe Handle: 0x004D

Handle 0x004F, DMI type 39, 22 bytes
System Power Supply
        Power Unit Group: 2
        Location: PSU2
        Name: PWS-1K62P-1R
        Manufacturer: SUPERMICRO
        Serial Number: P1K65CF11CT1838
        Asset Tag: N/A
        Model Part Number: PWS-1K62P-1R
        Revision: 1.3
        Max Power Capacity: 1620 W
        Status: Present, OK
        Type: Switching
        Input Voltage Range Switching: Auto-switch
        Plugged: Yes
        Hot Replaceable: No

A big twin (2u, 4 node chassis/server)

System Information
	Manufacturer: Supermicro
	Product Name: SYS-2028BT-HNC0R+
	Version: 0123456789
	Serial Number: S262328X8404031
	UUID: 00000000-0000-0000-0000-AC1F6B568514
	Wake-up Type: Power Switch
	SKU Number: Default string
	Family: Default string

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
	Manufacturer: Supermicro
	Product Name: X10DRT-B+
	Version: 1.01
	Serial Number: HM182S007228
	Asset Tag: Default string
	Features:
		Board is a hosting board
		Board is replaceable
	Location In Chassis: Default string
	Chassis Handle: 0x0003
	Type: Motherboard
	Contained Object Handles: 0

Handle 0x0003, DMI type 3, 22 bytes
Chassis Information
	Manufacturer: Supermicro
	Type: Other
	Lock: Not Present
	Version: 0123456789
	Serial Number: C217BAG43A10386
	Asset Tag: Default string
	Boot-up State: Safe
	Power Supply State: Safe
	Thermal State: Safe
	Security Status: None
	OEM Information: 0x00000000
	Height: Unspecified
	Number Of Power Cords: 1
	Contained Elements: 0
	SKU Number: Default string

and the power from them:

Handle 0x0016, DMI type 39, 22 bytes
System Power Supply
        Power Unit Group: 1
        Location: PSU1
        Name: PWS-2K22A-1R
        Manufacturer: SUPERMICRO
        Serial Number: P2K22CH52JT1112
        Asset Tag: N/A
        Model Part Number: PWS-2K22A-1R
        Revision: 1.3
        Max Power Capacity: 2200 W
        Status: Present, OK
        Type: Switching
        Input Voltage Range Switching: Auto-switch
        Plugged: Yes
        Hot Replaceable: Yes

Handle 0x0017, DMI type 39, 22 bytes
System Power Supply
        Power Unit Group: 2
        Location: PSU2
        Name: PWS-2K22A-1R
        Manufacturer: SUPERMICRO
        Serial Number: P2K22CH52JT1111
        Asset Tag: N/A
        Model Part Number: PWS-2K22A-1R
        Revision: 1.3
        Max Power Capacity: 2200 W
        Status: Present, OK
        Type: Switching
        Input Voltage Range Switching: Auto-switch
        Plugged: Yes
        Hot Replaceable: Yes

Supermicro Supermicro X9DRT-HF+

After getting the fixed code from #117 i still have issues but on other location. We definitely need more "catches" of errors.

This time the error is:
INFO:root:Creating RAID Card Supermicro SMC2108 (SN: FW-AD116BVAARBWA)
DEBUG:urllib3.connectionpool:http://XXXX:80 "GET /api/dcim/inventory-items/?device_id=2&tag=hw-disk HTTP/1.1" 200 52
DEBUG:urllib3.connectionpool:http://XXXX:80 "GET /api/dcim/manufacturers/?name=SMC2108 HTTP/1.1" 200 168
DEBUG:urllib3.connectionpool:http://XXXX:80 "POST /api/dcim/inventory-items/ HTTP/1.1" 201 453
INFO:root:Creating Disk SMC2108 003051452737b65c1a9072ab05b00506
DEBUG:urllib3.connectionpool:http://XXXX:80 "GET /api/dcim/manufacturers/?name= HTTP/1.1" 200 2269
Traceback (most recent call last):
File "/usr/local/bin/netbox_agent", line 11, in
load_entry_point('netbox-agent==0.5.1.dev5+gdc4f90d.d20200221', 'console_scripts', 'netbox_agent')()
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev5+gdc4f90d.d20200221-py3.7.egg/netbox_agent/cli.py", line 40, in main
return run(config)
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev5+gdc4f90d.d20200221-py3.7.egg/netbox_agent/cli.py", line 35, in run
server.netbox_update(config)
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev5+gdc4f90d.d20200221-py3.7.egg/netbox_agent/server.py", line 337, in netbox_update
self.inventory.update()
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev5+gdc4f90d.d20200221-py3.7.egg/netbox_agent/inventory.py", line 461, in update
self.do_netbox_disks()
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev5+gdc4f90d.d20200221-py3.7.egg/netbox_agent/inventory.py", line 402, in do_netbox_disks
self.create_netbox_disk(disk)
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev5+gdc4f90d.d20200221-py3.7.egg/netbox_agent/inventory.py", line 351, in create_netbox_disk
manufacturer = self.find_or_create_manufacturer(disk["Vendor"])
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev5+gdc4f90d.d20200221-py3.7.egg/netbox_agent/inventory.py", line 75, in find_or_create_manufacturer
name=name,
File "/usr/local/lib/python3.7/dist-packages/pynetbox-4.2.4-py3.7.egg/pynetbox/core/endpoint.py", line 142, in get
"get() returned more than one result. "
ValueError: get() returned more than one result. Check that the kwarg(s) passed are valid for this endpoint or use filter() or all() instead.

Will check this script later on my X10 chassis. We also have X11 chassis, but havent tested there.

Automatic decommission

For those that always want their Netbox 100% up-to-date, a cron-mode could be provided so that every agent fills in a last update date while another mechanism on the Netbox-side would be in charge of removing expired resources.

Add ability to manually specify if a server is a blade or not

Some manufacturers doesn't not let us find if a server is a blade programatically

In order to let the user manage this his way, we need to let it handle this case with CLI/config parameters like:

--is-blade
--chassis <S/N of the chassis>
--blade-slot "Slot 01"

handles #149 case

Datacenter awareness

I think it's mandatory to create the Device (chassis / blade / server) within a datacenter and if possible within a rack.

The problem is that there's no generic way to get these infos because it's not a standard like dmidecode. Each company has its own nomenclature and from what I can see, the info can be in:

  • a hostname (ie: web-42.nyc)
  • a file (ie: /etc/server_infos)
  • a saltstack grain (ie: salt-call grains.get datacenter)
  • another webservice (ie: curl https://other-api.internal.company.com/server?service_tag=XXX | jq -r .datacenter)

Since netbox_agent needs to be as generic as possible, we cannot put this kind of code in the project.

We may need a configuration file for the agent (http proxy, netbox URL, api_key and so on), so we could do something like:

DATACENTER_LOCATION = '<driver>:<location>'

Some examples could be:

DATACENTER_LOCATION = 'file:/etc/server_infos'
DATACENTER_LOCATION = 'url:https://other-api.internal.company.com/server?service_tag={{ service_tag }}'
DATACENTER_LOCATION = 'cmd:hostname'
DATACENTER_LOCATION = 'cmd:lldpctl'

and a DATACENTER_LOCATION_REGEX that would extract the slug/name of the datacenter within the output.

People could add a new "driver" if needed.

Ideas?

Better Supermicro Support (currently bails out)

Have a bailout now with the right lshw version.:

I attached the last lsow output as file

Traceback (most recent call last):
File "/usr/local/bin/netbox_agent", line 11, in
load_entry_point('netbox-agent==0.5.1.dev3+gbb7c8c8.d20200216', 'console_scripts', 'netbox_agent')()
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev3+gbb7c8c8.d20200216-py3.7.egg/netbox_agent/cli.py", line 40, in main
return run(config)
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev3+gbb7c8c8.d20200216-py3.7.egg/netbox_agent/cli.py", line 35, in run
server.netbox_update(config)
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev3+gbb7c8c8.d20200216-py3.7.egg/netbox_agent/server.py", line 336, in netbox_update
self.inventory = Inventory(server=self)
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev3+gbb7c8c8.d20200216-py3.7.egg/netbox_agent/inventory.py", line 56, in init
self.lshw = LSHW()
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev3+gbb7c8c8.d20200216-py3.7.egg/netbox_agent/lshw.py", line 48, in init
self.find_cpus(j)
File "/usr/local/lib/python3.7/dist-packages/netbox_agent-0.5.1.dev3+gbb7c8c8.d20200216-py3.7.egg/netbox_agent/lshw.py", line 107, in find_cpus
c["product"] = obj["product"]
KeyError: 'product'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.