Automatic completion should be more relevant:

• if you type fotoobo fgt get version it should present you with a list of the hosts from the inventory

Same applies for all other commands

Add a new command fotoobo fgt config info

The subcommand info should display the FortiGate information from the parsed configuration file.

If you set a syslog server in fotoobo.yaml for logging which is not resolvable by dns (name not known or timeout) it gives a treaceback:

socket.gaierror: [Errno -2] Name or service not known

or

socket.gaierror: [Errno -3] Temporary failure in name resolution

Instead fotoob should give a propper error message (and maybe continue anyway?)

When the syslog server is not resolvable (see #72) there is an exception originated in the python standard-Library:

Exception ignored in atexit callback: <function shutdown at 0x7f39db663eb0>
Traceback (most recent call last):
  File "/usr/lib64/python3.10/logging/__init__.py", line 2183, in shutdown
    h.close()
  File "/usr/lib64/python3.10/logging/handlers.py", line 945, in close
    self.socket.close()
AttributeError: 'SysLogHandler' object has no attribute 'socket'

How to reproduce:

• Configure a SysLog server that has a non-resolvable hostname/fqdn
• Run some fotoobo command (for example fotoobo fgt get version)

This seems to be an upstream issue:

• First reported here, but closed now: python/cpython#82961
• But it seems it raised again with this: python/cpython#82535

Probably the latter will solve the imminent effect, but not the base problem. Perhaps we need to fix this upstream.

There should be an option that returns the data structure as json returned by the monitor commands, and can be used in the jinja2 templates.

If the global policy in FortiManager cannot be assiged there should be an alert message (e-mail, webhook, ...).

At the moment we use @patch from unittest.mock for monkeypatching in tests.
We should switch to monkeypatch from pytest to be consistent in all tests.

In the utils the functions are named the same as the files. For very function there is an own file. This give naming clashes.
We strip one level and put the functions in a file one level higher
So there is no more file with the same name as the function within => Naming clash resolved

Candidates for paralle processing are:

• fotoobo fgt backup
• fotoobo fgt get version
• fotoobo fgt monitor hamaster

How should we do parallel processing at all?

on macos, poetry 1.4.1 was release a few days ago.

Problem
i was not able to install fotoobo with 'poetry install'

• Installing black (22.10.0): Failed

  _WheelFileValidationError

  ["In /Users/MYUSERNAME/Library/Caches/pypoetry/artifacts/9a/87/53/8a817f0bf6562226d9de0d3ad0eb2eb28c8191e978d21740991796e053/black-22.10.0-1fixedarch-cp311-cp311-macosx_11_0_x86_64.whl, hash / size of black-22.10.0.dist-info/WHEEL didn't match RECORD"]

  at /usr/local/Cellar/poetry/1.4.1/libexec/lib/python3.11/site-packages/installer/sources.py:289 in validate_record
      285│                         f"In {self._zipfile.filename}, hash / size of {item.filename} didn't match RECORD"
      286│                     )
      287│ 
      288│         if issues:
    → 289│             raise _WheelFileValidationError(issues)
      290│ 
      291│     def get_contents(self) -> Iterator[WheelContentElement]:
      292│         """Sequential access to all contents of the wheel (including dist-info files).
      293│ 

Solution
downgrad poetry locall to the previous version:

poetry self update 1.4.0

and all seems to work fine ...

Changed in Python version 3.10: Optional can now be written as X | None. See union type expressions.
Also Union can use | instead.

But this would mean we will end support for Python 3.8 and 3.9

As the whole project is named Fortinet TOOL box it makes sense to rename the utils to tools.

https://arc42.org/

Due to security reasons a backup transferred to an FTP server should support SFTP (encrypted)

Example: https://sftptogo.com/blog/python-sftp/

ToDo:

• Make SFTP the default, but allow to use FTP in the inventory
• Log a warning if FTP is used, because config (and especially the hashed passwords) are transferred unencrypted to the backup server

The command fgt monior hamaster should be able to create a json output file with a jinja2 template for a monitoring system
At the moment it is only possible to create e-mail notification
It's like with the ems monitor commands

check should be a subcommand for fotoobo fgt config.
config is the main command for manipulating .conf files. (offline execution)
There will be other subcommands soon

For settings which are always the same there should be a possibility to set global options by device type.
Every option can be overwritten on asset level by its own setting

If we execute pytest cli tests in an environment where the hostname 'dummy' is not resolvable we get a name not known error.

• Analyze why there is a dns lookup at all and fix it.
• Rename the hostname dummy in the test inventory to dummy_hostname

Python does not automatically expand user HOME shortcuts in the fotoobo.yaml (for example for the inventory file).
We have to do this manually, for example with os.path.expanduser() (see https://stackoverflow.com/questions/2668909/how-to-find-the-real-user-home-directory-using-python for details)

Maybe we should change all path handling to pathlib as it seems to be more intuitive. For examples see: https://medium.com/@ageitgey/python-3-quick-tip-the-easy-way-to-deal-with-file-paths-on-windows-mac-and-linux-11a072b58d5f

Official docs: https://docs.python.org/3/library/pathlib.html

>       assert syslog_formatter.format(log_record) == expected_string
E       AssertionError: assert '<10>1 1970-0... test_message' == '<10>1 1970-0... test_message'
E         Skipping 55 identical trailing characters in diff, use -v to show
E         - <10>1 1970-01-01T01:00:00+02:00 dummy
E         ?                            ^
E         + <10>1 1970-01-01T01:00:00+01:00 dummy
E         ?                            ^

Wouldn't it be better to check the time format with a regex instead of a fully qualified string?

Add a utility to get the parsed configuration
By adding a path it should be possible to get just parts of the configuration
(pretty)print or save it in json format

Command structure seems more accidental than intentional at the moment: Sometimes the command structure is very specific (fotoobo fgt check hamaster), sometimes there seems to lack a subcommand (fotoobo convert checkpoint).

I think there should be a general structure like:

• fotoobo <asset-type> <do-word> <object> (e.g. fotoobo fgt check hamaster)
• or fotoobo <do-word> <asset-type> <object> (e.g. fotoobo convert checkpoint assets)

When there is an error fotoobo will just print a line and exit. We should use the facility of rich here, and print a panel with the error message.

Every time I use the toolbox for a task, it logs in, then when done out. Next task, back in. It would be more efficient to store the login session and reuse it.

It seems the tests use the fotoobo.log if one is found. And if there is a syslog server configured but name resolution fails some tets fail because it tries to resolve the hostname given in the fotoobo logging configuration:

___ test_cli_convert_no_args ___

    def test_cli_convert_no_args() -> None:
        """Test convert cli without issuing any arguments"""
        result = runner.invoke(app, ["convert"])
>       assert result.exit_code == 2
E       AssertionError: assert 1 == 2
E        +  where 1 = <Result gaierror(-3, 'Temporary failure in name resolution')>.exit_code

tests/cli/test_cli_convert.py:19: AssertionError

or

___ test_cli_convert_no_args ___

    def test_cli_convert_no_args() -> None:
        """Test convert cli without issuing any arguments"""
        result = runner.invoke(app, ["convert"])
>       assert result.exit_code == 2
E       AssertionError: assert 1 == 2
E        +  where 1 = <Result gaierror(-2, 'Name or service not known')>.exit_code

tests/cli/test_cli_convert.py:19: AssertionError

The Command fmg post has wrong options order. FortiManger should be optional (default: fmg) and at the end of the options list. As with other fmg commands.
This is a breaking change!

Add option to send e-mail notification whenever fgt config check finds a non matching configuration

The test function test_cli_app_fmg_set_help() must be renamed as we changed the fmg command from set to post.

There are differences in the output/error handling for different commands:

• fotoobo fgt get version presents the data in a nice table, even in the case of failure
• fotoobo fmg get version will just print the error (in case of error), even no "Error"-panel

If an absolute path is given to the ems monitor utilities they all fail as using a relative path is hardcoded in save_with_template()

template_env = jinja2.Environment(loader=jinja2.FileSystemLoader("./"), trim_blocks=True)

Change the logic to be able to use relative AND absulte paths

Check if the FortiManager contains objects which are not present in the Checkpoint (or other) export file.
The check should generate a report.
What happens if we try to delete a FMG global object which is used in an ADOM?

It seems SFTP connection works sometimes different from normal FTP, at least when using ftplib from python.

we need to fix this. Hint: https://duckduckgo.com/?t=ffab&q=ftplib+error_perm+530&ia=web

During the review of the CLI structure & help texts we saw, that the whole template mechanism used by the monitoring command is basically undocumented. Especially document the following:

• How to create templates
• What variables exist during template execution (for which command & generally)

The syslog message format is defined in rfc5424 section 6

https://www.rfc-editor.org/rfc/rfc5424#section-6

• change the syslog message format to comply with rfc5424
• document it

From rfc5424 I understood the following syntax:
VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID STRUCTURED-DATA MSG

PRIVAL = Facility * 8 + Severity
VERSION = 1
APP-NAME = fotoobo
PROCID = os.getpid()
MSGID = "AUDIT" for audit messages
STRUCTURED-DATA = "-" (we leave it empty)

At the moment an autit message looks like this:
AUDIT:fotoobo:username=vader:hostname=deathstar:command="/home/vader/.local/bin/fotoobo greet"
But it should be this way:
<110>1 TIMESTAMP deathstar fotoobo 666 AUDIT - username=vader hostname=deathstar command="/home/vader/.local/bin/fotoobo greet"

It is not always the case that internal devices have certificates signed by a global CA accepted by the host operating system that runs fotoobo.

So we should add options to the configuration / inventory to add custom CA certificates / known good certificate information:

• Add a custom CA to the inventory (or globally to fotoobo.yaml?)
• Perhaps give the possibility to provide the known good hash of the respective certificate per device in the inventory. -> Need to check, whether Python SSL/TLS support supports this...

At the moment fortimanager.post() only return the number of objects that could not be set.
It would be better to return a list of the objects with the corresponding error codes (as returned by the FortiManager instance).

According to https://www.rfc-editor.org/rfc/rfc5424#section-6 the hostname comes before the app-name.

The help texts should be reviewed (for example fotoobo fgt backup -h):

• Better describe what happens if an optional argument is omitted (backup all FortiGates in inventory)
• Better describe the meaning of the arguments: What is the intention of the argument (e.g. source/target to read/write to)?
• Probably also other improvements

The connection port for Fortinet devices is appended to the hostname option. e.g.: fortgate.local:10443
In case the hostname is used for anything else than https connections it's not optimal if the port number is appended to the hostname directly.
Add an option https_port to the Fortinet devices and specify 443 as default.

We found that fotoobo using easysnmp is not running on every linux system. It depends on system tools. See installation notes: https://easysnmp.readthedocs.io/en/latest/#installation
We change the SNMP library to pysnmp which is a native Python implementation.

Load the fotoobo configuration in the following order:

• Command line options
• fotoobo.yaml in local folder
• fotoobo.yaml in user config folder (~/.config/fotoobo.yaml)

If none of the above is presend use good defaults
Make the config search method extendable to add more directories in the future

The command fmg assignis hardcoded to always assign the global policy default. There should be additional option to specify the policy to assign. If we make that option optional and set its default to `default' it would not break existing behaviour.

The business logic in the tools module should always return a defined datastructure or throw an exception.
They should never print or save data directly.
This prepares for potential FastAPI bindings.

in /home/pspiess/projects/mnet/fotoobo/tests/cli/fmg/test_cli_fmg_get.py we do not test for exception with with pytest.rises(...).
Should test for exceptions like in other tests

Backup_dir is only used for fgt backup and schould be removed from global setting in fotoobo.yaml
Instead add backup_dir as argument for the CLI command