migros / fotoobo Goto Github PK
View Code? Open in Web Editor NEWThe mighty Fortinet toolbox
License: GNU Lesser General Public License v3.0
The mighty Fortinet toolbox
License: GNU Lesser General Public License v3.0
Command structure seems more accidental than intentional at the moment: Sometimes the command structure is very specific (fotoobo fgt check hamaster
), sometimes there seems to lack a subcommand (fotoobo convert checkpoint
).
I think there should be a general structure like:
fotoobo <asset-type> <do-word> <object>
(e.g. fotoobo fgt check hamaster
)fotoobo <do-word> <asset-type> <object>
(e.g. fotoobo convert checkpoint assets
)It seems the tests use the fotoobo.log if one is found. And if there is a syslog server configured but name resolution fails some tets fail because it tries to resolve the hostname given in the fotoobo logging configuration:
___ test_cli_convert_no_args ___
def test_cli_convert_no_args() -> None:
"""Test convert cli without issuing any arguments"""
result = runner.invoke(app, ["convert"])
> assert result.exit_code == 2
E AssertionError: assert 1 == 2
E + where 1 = <Result gaierror(-3, 'Temporary failure in name resolution')>.exit_code
tests/cli/test_cli_convert.py:19: AssertionError
or
___ test_cli_convert_no_args ___
def test_cli_convert_no_args() -> None:
"""Test convert cli without issuing any arguments"""
result = runner.invoke(app, ["convert"])
> assert result.exit_code == 2
E AssertionError: assert 1 == 2
E + where 1 = <Result gaierror(-2, 'Name or service not known')>.exit_code
tests/cli/test_cli_convert.py:19: AssertionError
Check if the FortiManager contains objects which are not present in the Checkpoint (or other) export file.
The check should generate a report.
What happens if we try to delete a FMG global object which is used in an ADOM?
Every time I use the toolbox for a task, it logs in, then when done out. Next task, back in. It would be more efficient to store the login session and reuse it.
Automatic completion should be more relevant:
Same applies for all other commands
There are differences in the output/error handling for different commands:
fotoobo fgt get version
presents the data in a nice table, even in the case of failurefotoobo fmg get version
will just print the error (in case of error), even no "Error"-panelMaybe we should change all path handling to pathlib as it seems to be more intuitive. For examples see: https://medium.com/@ageitgey/python-3-quick-tip-the-easy-way-to-deal-with-file-paths-on-windows-mac-and-linux-11a072b58d5f
Official docs: https://docs.python.org/3/library/pathlib.html
During the review of the CLI structure & help texts we saw, that the whole template mechanism used by the monitoring command is basically undocumented. Especially document the following:
It seems SFTP connection works sometimes different from normal FTP, at least when using ftplib
from python.
we need to fix this. Hint: https://duckduckgo.com/?t=ffab&q=ftplib+error_perm+530&ia=web
If the global policy in FortiManager cannot be assiged there should be an alert message (e-mail, webhook, ...).
Changed in Python version 3.10: Optional
can now be written as X | None
. See union type expressions.
Also Union
can use |
instead.
But this would mean we will end support for Python 3.8 and 3.9
Candidates for paralle processing are:
fotoobo
fgt
backup
fotoobo
fgt
get
version
fotoobo
fgt
monitor
hamaster
How should we do parallel processing at all?
Due to security reasons a backup transferred to an FTP server should support SFTP (encrypted)
Example: https://sftptogo.com/blog/python-sftp/
ToDo:
The command fmg assign
is hardcoded to always assign the global policy default
. There should be additional option to specify the policy to assign. If we make that option optional and set its default to `default' it would not break existing behaviour.
If you set a syslog server in fotoobo.yaml for logging which is not resolvable by dns (name not known or timeout) it gives a treaceback:
socket.gaierror: [Errno -2] Name or service not known
or
socket.gaierror: [Errno -3] Temporary failure in name resolution
Instead fotoob should give a propper error message (and maybe continue anyway?)
When there is an error fotoobo will just print a line and exit. We should use the facility of rich here, and print a panel with the error message.
Add a utility to get the parsed configuration
By adding a path it should be possible to get just parts of the configuration
(pretty)print or save it in json format
For settings which are always the same there should be a possibility to set global options by device type.
Every option can be overwritten on asset level by its own setting
The syslog message format is defined in rfc5424 section 6
https://www.rfc-editor.org/rfc/rfc5424#section-6
From rfc5424 I understood the following syntax:
VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID STRUCTURED-DATA MSG
PRIVAL = Facility * 8 + Severity
VERSION = 1
APP-NAME = fotoobo
PROCID = os.getpid()
MSGID = "AUDIT" for audit messages
STRUCTURED-DATA = "-" (we leave it empty)
At the moment an autit message looks like this:
AUDIT:fotoobo:username=vader:hostname=deathstar:command="/home/vader/.local/bin/fotoobo greet"
But it should be this way:
<110>1 TIMESTAMP deathstar fotoobo 666 AUDIT - username=vader hostname=deathstar command="/home/vader/.local/bin/fotoobo greet"
Backup_dir is only used for fgt backup and schould be removed from global setting in fotoobo.yaml
Instead add backup_dir as argument for the CLI command
The business logic in the tools module should always return a defined datastructure or throw an exception.
They should never print or save data directly.
This prepares for potential FastAPI bindings.
Python does not automatically expand user HOME shortcuts in the fotoobo.yaml (for example for the inventory file).
We have to do this manually, for example with os.path.expanduser() (see https://stackoverflow.com/questions/2668909/how-to-find-the-real-user-home-directory-using-python for details)
Add option to send e-mail notification whenever fgt config check finds a non matching configuration
on macos, poetry 1.4.1 was release a few days ago.
Problem
i was not able to install fotoobo with 'poetry install'
• Installing black (22.10.0): Failed
_WheelFileValidationError
["In /Users/MYUSERNAME/Library/Caches/pypoetry/artifacts/9a/87/53/8a817f0bf6562226d9de0d3ad0eb2eb28c8191e978d21740991796e053/black-22.10.0-1fixedarch-cp311-cp311-macosx_11_0_x86_64.whl, hash / size of black-22.10.0.dist-info/WHEEL didn't match RECORD"]
at /usr/local/Cellar/poetry/1.4.1/libexec/lib/python3.11/site-packages/installer/sources.py:289 in validate_record
285│ f"In {self._zipfile.filename}, hash / size of {item.filename} didn't match RECORD"
286│ )
287│
288│ if issues:
→ 289│ raise _WheelFileValidationError(issues)
290│
291│ def get_contents(self) -> Iterator[WheelContentElement]:
292│ """Sequential access to all contents of the wheel (including dist-info files).
293│
Solution
downgrad poetry locall to the previous version:
poetry self update 1.4.0
and all seems to work fine ...
At the moment we use @patch
from unittest.mock for monkeypatching in tests.
We should switch to monkeypatch from pytest to be consistent in all tests.
At the moment fortimanager.post() only return the number of objects that could not be set.
It would be better to return a list of the objects with the corresponding error codes (as returned by the FortiManager instance).
If an absolute path is given to the ems monitor utilities they all fail as using a relative path is hardcoded in save_with_template()
template_env = jinja2.Environment(loader=jinja2.FileSystemLoader("./"), trim_blocks=True)
Change the logic to be able to use relative AND absulte paths
If we execute pytest cli tests in an environment where the hostname 'dummy' is not resolvable we get a name not known error.
in /home/pspiess/projects/mnet/fotoobo/tests/cli/fmg/test_cli_fmg_get.py we do not test for exception with with pytest.rises(...)
.
Should test for exceptions like in other tests
Add a new command fotoobo
fgt
config
info
The subcommand info
should display the FortiGate information from the parsed configuration file.
In the utils the functions are named the same as the files. For very function there is an own file. This give naming clashes.
We strip one level and put the functions in a file one level higher
So there is no more file with the same name as the function within => Naming clash resolved
Load the fotoobo configuration in the following order:
If none of the above is presend use good defaults
Make the config search method extendable to add more directories in the future
The test function test_cli_app_fmg_set_help() must be renamed as we changed the fmg command from set to post.
It is not always the case that internal devices have certificates signed by a global CA accepted by the host operating system that runs fotoobo.
So we should add options to the configuration / inventory to add custom CA certificates / known good certificate information:
fotoobo.yaml
?)The connection port for Fortinet devices is appended to the hostname option. e.g.: fortgate.local:10443
In case the hostname is used for anything else than https connections it's not optimal if the port number is appended to the hostname directly.
Add an option https_port to the Fortinet devices and specify 443 as default.
There should be an option that returns the data structure as json returned by the monitor
commands, and can be used in the jinja2 templates.
The command fgt monior hamaster
should be able to create a json output file with a jinja2 template for a monitoring system
At the moment it is only possible to create e-mail notification
It's like with the ems monitor
commands
The Command fmg post
has wrong options order. FortiManger should be optional (default: fmg) and at the end of the options list. As with other fmg commands.
This is a breaking change!
> assert syslog_formatter.format(log_record) == expected_string
E AssertionError: assert '<10>1 1970-0... test_message' == '<10>1 1970-0... test_message'
E Skipping 55 identical trailing characters in diff, use -v to show
E - <10>1 1970-01-01T01:00:00+02:00 dummy
E ? ^
E + <10>1 1970-01-01T01:00:00+01:00 dummy
E ? ^
Wouldn't it be better to check the time format with a regex instead of a fully qualified string?
The help texts should be reviewed (for example fotoobo fgt backup -h
):
check
should be a subcommand for fotoobo
fgt
config
.
config
is the main command for manipulating .conf files. (offline execution)
There will be other subcommands soon
As the whole project is named Fortinet TOOL box it makes sense to rename the utils to tools.
When the syslog server is not resolvable (see #72) there is an exception originated in the python standard-Library:
Exception ignored in atexit callback: <function shutdown at 0x7f39db663eb0>
Traceback (most recent call last):
File "/usr/lib64/python3.10/logging/__init__.py", line 2183, in shutdown
h.close()
File "/usr/lib64/python3.10/logging/handlers.py", line 945, in close
self.socket.close()
AttributeError: 'SysLogHandler' object has no attribute 'socket'
How to reproduce:
fotoobo fgt get version
)This seems to be an upstream issue:
Probably the latter will solve the imminent effect, but not the base problem. Perhaps we need to fix this upstream.
We found that fotoobo using easysnmp is not running on every linux system. It depends on system tools. See installation notes: https://easysnmp.readthedocs.io/en/latest/#installation
We change the SNMP library to pysnmp which is a native Python implementation.
According to https://www.rfc-editor.org/rfc/rfc5424#section-6 the hostname comes before the app-name.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.