Giter Club home page Giter Club logo

katprep's Introduction

Build Status codecov

katprep

katprep is a Python toolkit for automating system maintenance and generating patch reports for systems managed with Foreman/Katello or Red Hat Satellite 6.x.

This can be very useful if you need to document software changes due to IT certifications like ISO/IEC 27001:2005 or many other.

katprep can automate the following infrastructure tasks:

  • create/remove virtual machine snapshots hypervisor independently (e.g. VMware vSphere, KVM, XEN, Hyper-V,...) by utilizing libvirt and the VMware vSphere Python API bindings (pyVmomi)
  • schedule/remove downtimes within your monitoring system (Nagios/Icinga, Icinga2)
  • patch and reboot affected systems
  • document system changes in a customizable report by utilizing Pandoc (HTML, Markdown,...)

This software is a complete rewrite of my other toolkit satprep.

Documentation and contribution

The project documentation is created automatically using Sphinx - it can be found in the doc folder of this repository. Check-out this website for an online mirror.

You want to contribute? That's great! Please check-out the Issues tab of this project and share your thoughts/ideas in a new issue - also, pull requests are welcome!

How does this work?

katprep uses Puppet host parameters to assign additional meta information to systems managed with Foreman/Katello or Red Hat Satellite such as:

  • monitoring/virtualization system managing the host
  • differing object names within those systems
  • snapshots required before system maintenance

katprep workflow

If you plan to execute maintenance tasks, katprep triggers (katprep_maintenance utility) monitoring and virtualization hosts to schedule downtimes and create VM snapshots. Once these tasks have been completed, katprep can automatically trigger the patch installation and system reboot. After verifying your systems, katprep can remove downtimes and snapshots automatically. Before and after patching systems, it is necessary to create an inventory report of your system landscape. These reports contain information such as outstanding patches - after patching your systems, the katprep_report utility automatically calculares differences and creates patch reports for all updated hosts.

As a result, patching big system landscapes becomes less time-consuming with katprep: it's only executing three commands - independent whether you are patching 1 host or 1000 hosts.

To make the installation even easier, an auto-discover functionality can scan your monitoring systems and hypervisors and link gathered information with Foreman/Katello and Red Hat Satellite automatically (katprep_populate).

katprep's People

Contributors

dependabot-preview[bot] avatar okin avatar stdevel avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

katprep's Issues

Invalid hosts crash katprep_snapshot

Having invalid hosts (e.g. virt-who hosts or invalid content hosts) in the inventory when running katprep_snapshot will make the tool crash:

$ katprep_snapshot -C pinkepank.auth -s giertz.shittyrobots.loc
...
ERROR:katprep_snapshot:Unable to get system information, check filter options!
INFO:katprep_snapshot:Report './errata-snapshot-report-giertz-20170617-1409.json' created.

Unregistered content hosts crashes katprep_snapshot

When checking unregistered content hosts, katprep_snapshot crashes:

INFO:katprep_snapshot:Checking system 'pinkepank.test.loc' (#123)...
...
TypeError: 'NoneType' object has no attribute '__getitem__'

Environment:

  • Red Hat Enterprise Linux 7.4 (x86_64)
  • Red Hat Satellite 6.2.14

Also install package updates

Beginning with Foreman 1.14.3, non-errata package updates can be triggered/installed using the UI - would be handy if katprep_maintenance could do the same.

Choose shorter class names

It would be handy to have shorter class names - to shorten lines like this:

from katprep.client.BasicIcinga2APIClient import BasicIcinga2APIClient

to this:

from katprep.client.icinga2 import Icinga2Client

Pandoc PDF template

Currently, there is only a HTML and Markdown template - but most companies might want to have fancy PDF reports.

Some report flags are incorrect

Some report flags are currently incorrect:

  • Icinga2 monitoring state flags are flooding the report
  • Physical/VM flags
  • reboot required flags

Implement encryption for authentication containers

Currently, authentication containers contain login credentials in plain text - it would be great to have an encryption functionality. When loading an authentication container, a passphrase could be prompted in order to decrypt/load the data.

Configuration file functionality

For some scripts of the katprep framework plenty of parameters can be specified - it would be great to have a hidden configuration file that automatically pre-fills these parameters.

Serve manpages

It would be very handy to have manpages for users without browsers

Retrieving monitoring status for Icinga currently not working

INFO:katprep_maintenance:No downtime for host 'pinkepank.giertz.loc' found, probably cleaned-up.
Traceback (most recent call last):
  File "/home/cstan/.local/bin/katprep_maintenance", line 9, in <module>
    load_entry_point('katprep==0.1', 'console_scripts', 'katprep_maintenance')()
  File "/home/cstan/katprep/katprep/maintenance.py", line 726, in cli
    main(options, args)
  File "/home/cstan/katprep/katprep/maintenance.py", line 705, in main
    options.func(options, options.func)
  File "/home/cstan/katprep/katprep/maintenance.py", line 370, in verify
    crit_services = MON_CLIENTS[get_host_param_from_report(REPORT, host, "katprep_mon")].get_services(mon_name)
  File "/home/cstan/katprep/katprep/clients/NagiosCGIClient.py", line 427, in get_services
    result = self.__api_get(url)
  File "/home/cstan/katprep/katprep/clients/NagiosCGIClient.py", line 175, in __api_get
    return self.__api_request("get", sub_url)
  File "/home/cstan/katprep/katprep/clients/NagiosCGIClient.py", line 146, in __api_request
    raise SessionException("Unable to authenticate")
katprep.clients.SessionException: Unable to authenticate

Backport Uyuni support?

Maybe it would be a good idea to backport the Uyuni and SUSE Manager support?

For this, the following things would need to be done:

  • replace API calls in maintenance.py, parameters.py, populate.py and snapshot.py with more generic functions (e.g. get hosts, get packages,...) and add optional server type (Uyuni) in order to make katprep more modular
  • replace API calls in ForemanAPIClient.py with nailgun (see also issue #134)
  • implement UyuniAPIClient.py
  • implement unit tests and Uyuni vagrant boxes

It's a moderate code change but would also enable using two popular utilities. satprep (which was designed for these tools) is really outdated and updating it might take more effort..

Open questions

  • How to deal with unsupported filtering options? (Spacewalk does not support locations and environments)
    • throw UnsupportedFilterException
  • SpacewalkAPIClient.py would need a function which combines multiple calls in a JSON result (combining errata information as well as custom variables, etc.) in order to enable reports
  • depends on issue #110

python-requests changed SSL default behavior

On some newer Python requests versions, the script is failing:

 File "/usr/lib64/nagios/plugins/ForemanAPIClient.py", line 339, in get_id_by_name
self.api_get("/{}s".format(api_object)) File "/usr/lib64/nagios/plugins/ForemanAPIClient.py", line 182, in
api_get return self.__api_request("get", sub_url, "", hits, page) File "/usr/lib64/nagios/plugin
/ForemanAPIClient.py", line 150, in __api_request headers=self.HEADERS File "/usr/lib/python2.7/site
packages/requests/sessions.py", line 476, in get return self.request('GET', url, **kwargs) File "/usr/li
/python2.7/site-packages/requests/sessions.py", line 464, in request resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send r = adapter.send(request,
**kwargs) File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 431, in send raise SSLError(e,
request=request) requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
(_ssl.c:579)

It seems like newer module versions enforce SSL checking which has not been the default, yet. The affected library is ForemanAPIClient.

Supply essential documentation

Currently, Sphinx automatically documents classes and other code. It is also necessary to have additional documentation about the framework including:

  • Installation
  • Authentication
  • Tested setups
  • Customizing
  • Parameter overview
  • Example implementations

Travis CI failing

Currently, Travis CI is failing because of missing Python packages. A fix might be changing the Python 3.x version.

Revert system maintenance

It would be great if katprep_maintenance.py could revert system maintenance - e.g. revert VM snapshot or uninstalling errata.

Rebooting virtualized systems after maintenance doesn't work

Rebooting after maintenance doesn't work for all systems. The reason is that katprep_maintenance.py triggers the soft power_action call using the Foreman API. This function only triggers ACPI if appropriate fencing drivers are installed. So, we need to find another solution here - e.g. triggering the reboot command..

Filter IPs when populating variables with katprep_populate

Currently, the first IP found is used when populating host information. If a virtual machine has multiple network interfaces or protocols (IPv4/6), this could be a problem:

$ katprep_populate -C giertz.auth -s pinkepank.test.loc --virt-uri vc.test.loc --virt-type pyvmomi --mon-url http://mon.test.loc/test/icinga --mon-type nagios -n -d
...
DEBUG:katprep_populate:HYPERVISOR: Found VM 'pinkepank.giertz.loc' with IP 'fe6a:251e:328:1000::2'

It would be great to have IPv4/6 filters or the possibility to match multiple IPs.

Auto-ignore Satellite / Foreman system

Checking the Satellite/Foreman system might fail if the system is not managed by itself:

$ katprep_snapshot -C pinkepank.auth -s shittyrobot.giertz.loc
INFO:katprep_snapshot:Checking system 'shittyrobot.giertz.loc' (#1)...
ERROR:katprep_snapshot:Unable to get system information, check filter options!

It would be great to auto-ignore the Satellite/Foreman system. For integrating the system an additional parameter such as --include-satellite could be implemented.

Show maintenance progress

katprep_maintenance triggers API calls to queue Foreman tasks. I would be nice to also see the progress (e.g. using the verify command).

Empty inputs not possible for katprep_parameters

Currently, katprep_paremeters does not support supplying empty inputs for major variables. It would be good if empty parameters would just be ignored (e.g. to not change pre-defined values)

Add hostgroup filter

It would be handy, if generic filters would include hostgroups. Currently, they only support Puppet environments, locations and organizations.
This requires adding a sub-call which checks hostgroup memberships per host.

Update online documentation

Currently, the online documentation is outdated. It would be great to hand-over this to Travis CI to always have the latest version online.

Implement unit tests and CI

Currently, there are no unit tests resulting in manual tests. Unit tests are essential for continuous integration.
On the other hand, especially katprep_maintenance is kinda fragile sometimes. Having unit tests might help improving it.

  • Foreman tests
  • Nagios test s
  • Icinga tests
  • Icinga2 tests
  • Pyvmomi tests
  • Libvirt tests
  • CI template

Master password needs to be entered multiple times

Currently, the master password needs to be entered multiple times if an authentication container is used:

$ katprep_maintenance -C pinkepank.auth -s ewa.test.loc --insecure -n errata-snapshot*.json prepare
INFO:katprep_maintenance:This is just a SIMULATION - no changes will be made.
File password ('Foreman'@'ewa.test.loc'): 
File password ('Virtualization vc.test.loc'@'vc.test.loc'): 
...

Would be great to have caching.

Migrate unit tests to pytest

To be more "state of the art", it would be great to migrate the tests to pytest. This would also enable test fixtures and markers.

Check Foreman task for successful installation of errata

After sending the API call to Foreman to install errata for a host, the host, if it's a VM, gets directly rebooted if the needs-reboot flag or the reboot option is set and than the installation fails.

It should be checked if the installation task was successful prior to rebooting the host.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.