n4soc / fortilogcsv Goto Github PK

Hi, I tried running the converter however the resulting .csv file was blank.

Hi, I have an even longer log file, however, a few columns are being left out.

ForwardTrafficLog-disk.log

I get below error during the conversion of log using the script convert.py

Traceback (most recent call last):
File "C:\Temp\FW Logs\convert.py", line 20, in
for line in log_data:
File "C:\Users\akamburvenk2\AppData\Local\Programs\Python\Python38-32\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 191: character maps to

Hi, I got this issue when running the file:

File "D:\08.Library\Python\Convert log Forti\fortilogcsv.py", line 5
if len(sys.argv) 1
^
SyntaxError: invalid syntax

Would you please kindly help to see how to fix it?

Hello,
I have a problem when one entry contains the "=" sign. It happens when Fortimail has some emails containing non utf characters. These are saved inside the Fortimail csv log as "=?utf-8?q?" and then not converted correctly.
I am no expert on python, but the conversion should be:
search for the first "=" character, then everything that comes afterwards, until next comma is the text, so do not convert it.
Thank you in advance for support!

Hello, getting the following error on both macos and linux (ubuntu):

Traceback (most recent call last):
File "/Users/tako/Tools/fortilogcsv/convert.py", line 21, in
for line in log_data:
File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/codecs.py", line 714, in next
return next(self.reader)
File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/codecs.py", line 645, in next
line = self.readline()
File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/codecs.py", line 558, in readline
data = self.read(readsize, firstline=True)
File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/codecs.py", line 504, in read
newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd8 in position 0: invalid continuation byte

Best regards
mrtako

Hello,
Also client_name portion is gone after the conversion...
Thank you in advance!

Hey @mrrothe!

This is a very useful script! I do not use FortiAnalyzer and the Fortigates do not have the issue to download logs in CSV format, so I was pleased to find someone had already done the legwork of parsing through the field=value log format the Foritgates provide and converting it to CSV.

However, in running the script the first time I found that key fields were being omitted from the CSV the script generated. After comparing the raw log with the CSV, I found it was fields that did not contain quotation marks (dstip for example).

I have changed the regex to account for this and have tested it on quite a few lines from a raw log file. I wanted to create this issue in order to create a pull request with my changes.

Not an issue more a feature request.

Currently I have multiple .log files from Fortinet.
If I process each file, the .csv column set and order are different between .csv files, which makes joining them difficult.
If this script could input multiple .log files, it seems to handle column sets fairly dynamically, and could output to one .csv file with it all in order.

Unless anyone knows of a better way to do this.

The URL field value is not present after converting. Attaching original and converted log example, only addresses and URLS are changed from original.
webfilter_example.csv
webfilter_example.log

Conversion log:
$ Python convert.py webfilter_example.log
[+] Reading logs from webfilter_example.log
[+] Processing log fields
[+] Writing CSV
[+] Finished - 12 rows written to webfilter_example.csv