jasonish / py-idstools Goto Github PK
View Code? Open in Web Editor NEWidstools: Snort and Suricata Rule and Event Utilities in Python (Including a Rule Update Tool)
License: Other
idstools: Snort and Suricata Rule and Event Utilities in Python (Including a Rule Update Tool)
License: Other
UnicodeDecodeError is caused by raw fields like:
tcp_options_raw
ip_options_raw
ip6_source_raw
ip6_destination_raw
... the error message contains:
UnicodeDecodeError('utf8', "\x01\x01\x08\n's\x04j\x14\xb2{", 10, 11, 'invalid start byte')
This can be re-created, before indexing into elasticsearch, using:
output["packet_details"] = packet.decode_ethernet(data)
from elasticsearch.serializer import JSONSerializer, Deserializer, DEFAULT_SERIALIZERS, TextSerializer
test_serializer = JSONSerializer().dumps(output["packet_details"])
For testing I'm using unified2-current.log and these files:
https://github.com/mephux/unified2/tree/master/example/seeds
... these log files are from 2010/11, so old but not too old, just trying to test as
many new/old unified2 files as I can find.
The current solution is to ignore these fields and not store
them in elasticsearch, but it might be useful to have them
in some format/encoding.
Suggestions ?
Is there some encoding/decoding/formatting of the "*_raw" fields I should do before
trying to index them ?
I am merging Suricata ET rules on a server where Suricata daemon is not present, but I know which version of it runs on the boxes. However, rulecat doesn't use '-enhanced' ruleset in such a case, it goes with '-1.3' only.
Would it be possible to add an option with Suricata version override? Or just an option instructing rulecat to download and use the enhanced ruleset?
When I look at the barnyard2 data in the mysql table "sensor" the value for
"sid" is 1, but the value of "sensor_id" from u2json.py is always 0.
u2json
-C=/my/classification.config
-S=/my/sid-msg.map
-G=/my/gen-msg.map
--directory=/my
--prefix=snort.unified2
--follow
--bookmark
--packets
--extra-data
I think it should be 1, or am I missing something
... or is this a "hard coded" setting based on the directory of snort logs ?
Suggestions ?
Just wondering if I would be missing any important info if I ignore these raw options:
for event_packet in event["packets"]:
data = event_packet["data"]
decode_ethernet_data = packet.decode_ethernet(data)
decode_ethernet_data.pop("tcp_options_raw", None)
decode_ethernet_data.pop("ip_options_raw", None)
decode_ethernet_data.pop("ip6_source_raw", None)
decode_ethernet_data.pop("ip6_destination_raw", None)
. . .
... aren't these already present in other fields?
I came up with a way to handle these payload fields:
"tcp_payload"
"udp_payload"
"icmp_payload" ... the only one for IPv6?
... so if I can ignore the "_raw" fields above then I will be finished with an open source program to read
unified2 files and index them into elasticsearch
... no barnyard2, no mysql, no ancient snort schema :)
idstools-rulecat crashes with an exception when using Python 3 and Suricata is installed:
[root@9d3d1f76ead0 ~]# idstools-rulecat
Traceback (most recent call last):
File "/usr/bin/idstools-rulecat", line 12, in <module>
sys.exit(main())
File "/usr/lib/python3.4/site-packages/idstools/scripts/rulecat.py", line 725, in main
args.url.append(resolve_etopen_url(args.suricata))
File "/usr/lib/python3.4/site-packages/idstools/scripts/rulecat.py", line 642, in resolve_etopen_url
suricata_version = idstools.suricata.get_version(suricata_path)
File "/usr/lib/python3.4/site-packages/idstools/suricata.py", line 61, in get_version
m = re.search("version ((\d+)\.(\d+)\.?(\d+|\w+)?)", output.strip())
File "/usr/lib64/python3.4/re.py", line 170, in search
return _compile(pattern, flags).search(string)
TypeError: can't use a string pattern on a bytes-like object
The error is in parsing the output of "suricata -V".
HI there,
first of all thanks for this library, it is very good and helpful!
However either I'm doing something wrong or there is a problem with bookmarking.
I'm working with version 0.3.1 and have following (simplified) code. For debugging purposes this I was not using a for statement:
bookmark_filename, bookmark_offset = json.load(open(config['ids']['bookmark_file']))
self.ids_reader = unified2.SpoolEventReader(
config['ids']['ids_log_dir'],
config['ids']['ids_log_prefix'],
init_filename=bookmark_filename,
init_offset=bookmark_offset)
event = self.ids_reader.next()
logging.log(logging.DEBUG, "Processing new event %d from sensor %d" % (event['event-id'], event['sensor-id']))
json.dump(self.ids_reader.tell(), open(config['ids']['bookmark_file'], "w"))
So what I've expected is that if I run this piece of code multiple times, it accesses each event in the the specified files. However if I look in the logging output I see that there is a jump over one record:
2014-04-16 17:30:30,957 [INFO ] Initializing unified2.SpoolEventReader
2014-04-16 17:30:30,969 [INFO ] Start processing new incoming and unprocessed old events
2014-04-16 17:30:30,973 [WARNING ] Discarding non-event type while not in event context.
2014-04-16 17:30:30,977 [DEBUG ] Processing new event 106 from sensor 0
2014-04-16 17:31:42,452 [INFO ] Initializing unified2.SpoolEventReader
2014-04-16 17:31:42,453 [INFO ] Start processing new incoming and unprocessed old events
2014-04-16 17:31:42,454 [WARNING ] Discarding non-event type while not in event context.
2014-04-16 17:31:42,455 [DEBUG ] Processing new event 108 from sensor 0
2014-04-16 17:31:44,677 [INFO ] Initializing unified2.SpoolEventReader
2014-04-16 17:31:44,677 [INFO ] Start processing new incoming and unprocessed old events
2014-04-16 17:31:44,678 [WARNING ] Discarding non-event type while not in event context.
2014-04-16 17:31:44,678 [DEBUG ] Processing new event 110 from sensor 0
It seems that this warning indicates that something went wrong while parsing the file. Did I do anything wrong or is there an issue with the bookmarking?
Note: If I parse the file without the bookmark I can access all records.
Thanks
root@snort1:/etc/snort# idstools-u2json -C /etc/snort/classification.config -S /etc/snort/sid-msg.map -G /etc/snort/gen-msg.map /var/log/snort/snort.log.1454584880
INFO: Loaded 514 rule message map entries.
INFO: Loaded 38 classifications.
Traceback (most recent call last):
File "/usr/local/bin/idstools-u2json", line 12, in
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/idstools/scripts/u2json.py", line 308, in main
for record in reader:
File "/usr/local/lib/python2.7/dist-packages/idstools/unified2.py", line 586, in next
record = self.reader.next()
File "/usr/local/lib/python2.7/dist-packages/idstools/unified2.py", line 549, in next
return read_record(self.fileobj)
File "/usr/local/lib/python2.7/dist-packages/idstools/unified2.py", line 522, in read_record
raise err
EOFError
idstools-u2json --output test.json --packets --extra-data snort2.log.1393411837
WARNING: No alert message map entries loaded.
WARNING: No classifications loaded.
Traceback (most recent call last):
File "/usr/local/bin/idstools-u2json", line 5, in
pkg_resources.run_script('idstools==0.5.0dev', 'idstools-u2json')
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 528, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1394, in run_script
execfile(script_filename, namespace, namespace)
File "/usr/local/lib/python2.7/dist-packages/idstools-0.5.0dev-py2.7.egg/EGG-INFO/scripts/idstools-u2json", line 12, in
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/idstools-0.5.0dev-py2.7.egg/idstools/scripts/u2json.py", line 374, in main
encoded = json.dumps(output_filter.filter(event))
File "/usr/lib/python2.7/json/init.py", line 243, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xd8 in position 272: invalid continuation byte
Possible solution to base64 encode data field:
Change from line 186:
output_data = {
"data-type": data["data-type"],
"event-type": data["event-type"],
"type": data["type"],
"data": data["data"]
}
to
output_data = {
"data-type": data["data-type"],
"event-type": data["event-type"],
"type": data["type"],
"data": base64.b64encode(data["data"])
}
cd /tmp
git clone https://github.com/jasonish/py-idstools
cd py-idstools
git rev-parse HEAD
#6821df33bfaaf39331cf27843144d48cc3f9b8dc
PYTHONPATH=. python3 bin/idstools-rulecat -o /etc/suricata/rules
2017-05-18 11:20:43,213 - <INFO> -- Found Suricata version 3.2.1 at /usr/sbin/suricata.
Traceback (most recent call last):
File "bin/idstools-rulecat", line 12, in <module>
sys.exit(main())
File "/tmp/py-idstools/idstools/scripts/rulecat.py", line 766, in main
files = Fetch(args).run()
File "/tmp/py-idstools/idstools/scripts/rulecat.py", line 284, in run
files.update(self.fetch(url))
File "/tmp/py-idstools/idstools/scripts/rulecat.py", line 262, in fetch
tmp_filename = self.get_tmp_filename(url)
File "/tmp/py-idstools/idstools/scripts/rulecat.py", line 251, in get_tmp_filename
url_hash = hashlib.md5(url).hexdigest()
TypeError: Unicode-objects must be encoded before hashing
I'm trying to replicate, kind of, the info that Barnyard2 provides in database tables, but
certain event fields seem to be missing, or I'm not sure how to get them, in idstools.
I'm using:
u2json_reader = unified2.SpoolEventReader(
directory=args.directory,
prefix=args.prefix,
follow=args.follow,
bookmark=args.bookmark)
In the generator, I print out the event dict as received by:
for event in u2json_reader:
. . .
... here's a list of event fields I can't find in the output, and
I've referenced the mysql table/column names as used by barnyard2:
signature:
sig_priority ... is: priority
sig_class_id ... is: signature-id ?
sig_class_name
sig_rev ... is: signature-revision
sig_sid
sig_gid
iphdr:
ip_ver
ip_hlen
ip_tos
ip_len
ip_id
ip_flags
ip_off
ip_ttl
ip_proto ... is: proto ?
ip_csum
tcphdr:
tcp_seq
tcp_ack
tcp_off
tcp_res
tcp_flags
tcp_win
tcp_csum
tcp_urp
udphdr: (not sure about these)
udp_len
udp_csum
icmphdr:
icmp_type ... is seen in output
icmp_code ... is seen in output
icmp_csum
icmp_id
icmp_seq
Suggestions?
I think:
"unified2.log", tail=True)
in the README.rst should be:
"unified2.log", follow=True)
... not really as issue :)
Some rule in snort 2.9.7.2-1 is creating a breaking case in the python module:
INFO: Loaded 25034 rule message map entries.
INFO: Loaded 38 classifications.
Traceback (most recent call last):
File "py-idstools/bin/idstools-u2json", line 12, in
sys.exit(main())
File "/root/py-idstools/idstools/scripts/u2json.py", line 308, in main
as_json = json.dumps(formatter.format(record))
File "/usr/lib64/python2.7/json/init.py", line 243, in dumps
return _default_encoder.encode(obj)
File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xec in position 633: invalid continuation byte
Using the latest bits. If I clear out the unified log and restart it it will work until this sequence is hit again.
If a semicolon (;
) is present (escaped, obviously) in the rule msg
option the parser gets confused.
Example:
$ my_ryle = '''alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"example of semicolon\; usage here"; classtype:trojan-activity; sid:9999; rev:1;)'''
$ parsed_rule = rule.parse(my_rule)
$ print parsed_rule.msg
'"example of semicolon\\'
$ print parsed_rule.options
[{'name': 'msg', 'value': '"example of semicolon\\'}, {'name': 'usage here"', 'value': None}, {'name': 'classtype', 'value': 'trojan-activity'}, {'name': 'sid', 'value': '9999'}, {'name': 'rev', 'value': '1'}]
Some rules (ETPRO) really use semicolon in the msg option.
Should zero-sized files like:
-rw------- 1 snort snort 0 Oct 13 17:31 snort.u2.1413221470
cause SpoolEventReader to not all/other unified2 files in the same folder ?
I'm probably missing something simple, but this was working before ... but now:
>>> import idstools
>>> from idstools import packet
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name packet
with:
python 2.7.6
pip freeze idstools==0.4.4
... this worked using 0.5.0dev
Please ignore the above, I forgot that you have to do:
pip install git+https://github.com/jasonish/py-idstools.git
and not:
pip install idstools
... but maybe this will help others that make the same mistake.
I have the following in my disable.conf file:
re:.*
re:^#.*
re:ET.* TOR
re:ET.* SCADA
re:ET.* SCADA_SPECIAL
re:ET.* DELETED
re:ET.* SURICATA STREAM
and some individual rule IDs. Then, in my enable.conf file I have a few entries like this:
re:^[^#].* EXPLOIT .*
re:^[^#].* CINS .*
re:^[^#].* CURRENT_EVENTS .*
and some other individual rule IDs.
I am still getting tons of other rules enabled, such as many of the ET DELETED rules, etc. Can you please explain the order of precedence, etc.? Does disable override enable, modify, etc.? Is there a way for me to tell rulecat to favor disable over enable?
An odd one. I don't get this with the one in tests dir:
(Event)
u2spewfoo.py(116): for row in rows:
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(118): print("\t" + "\t".join(parts))
sensor id: 0 event id: 1 event second: 1490338878 event microsecond: 955846
u2spewfoo.py(116): for row in rows:
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(118): print("\t" + "\t".join(parts))
sig id: 2402000 gen id: 1 revision: 4391 classification: 30
u2spewfoo.py(116): for row in rows:
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
--- modulename: u2spewfoo, funcname: print_address
u2spewfoo.py(83): if len(addr) <= 15:
u2spewfoo.py(84): return addr
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
--- modulename: u2spewfoo, funcname: print_address
u2spewfoo.py(83): if len(addr) <= 15:
u2spewfoo.py(84): return addr
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(118): print("\t" + "\t".join(parts))
priority: 2 ip source: 91.223.133.13 ip destination: x.x.x.x
u2spewfoo.py(116): for row in rows:
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(118): print("\t" + "\t".join(parts))
src port: 58978 dest port: 8899 protocol: 6 impact_flag: 0 blocked: 0
u2spewfoo.py(116): for row in rows:
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117): parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
--- modulename: trace, funcname: _unsettrace
trace.py(80): sys.settrace(None)
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/lib/python2.7/trace.py", line 819, in <module>
main()
File "/usr/lib/python2.7/trace.py", line 807, in main
t.runctx(code, globs, globs)
File "/usr/lib/python2.7/trace.py", line 513, in runctx
exec cmd in globals, locals
File "bin/idstools-u2spewfoo", line 12, in <module>
sys.exit(main())
File "/home//nobackup/build/py-idstools/idstools/scripts/u2spewfoo.py", line 191, in main
print_record(record)
File "/home/nobackup/build/py-idstools/idstools/scripts/u2spewfoo.py", line 173, in print_record
print_event(record)
File "/home/nobackup/build/py-idstools/idstools/scripts/u2spewfoo.py", line 117, in print_event
parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
KeyError: 'pad2'
Running latest snort-2.9.9.0. Anything I can do to troubleshoot on this end? Running the u2spewfoo that comes with snort runs fine. Thank you.
in EVENT_FIELDS and EVENT_IP6_FIELDS in unified2.py, you have the "sport-itype" and "dport-icode" fields. These should be four fields instead of two:
sport -> source Port
dport -> destination port
itype -> ICMP Type
icode -> ICMP code
But not in its raw format, make it usable.
Which is the format of the 'data' field that provides SpoolEventReader? Is the same for every reader. Ii is no the hexadecimal and it seems that the library add some content.
Here is an extract example:
\x00!\xd7j\xe4\x00\xdcJ>\x88*R\x08\x00E\xc0\x00q\xf4
CentOS7, python 3.4.5 - have a problem with
py-idstools/idstools/scripts/u2eve.py
Line 302 in 5862a93
Failed to encode record as JSON: __init__() got an unexpected keyword argument 'encoding':
but looking at https://docs.python.org/3/library/json.html#basic-usage json.dumps does not have the encoding argument, and encoding in json.loads is deprecated.
This request is divided in two step:
$ idstools-rulecat --url "https://www.snort.org/rules/community" --merged community.rules
2017-05-16 17:51:56,704 - <INFO> -- Found Suricata version 3.1.2 at /usr/bin/suricata.
2017-05-16 17:51:56,705 - <INFO> -- Checking https://www.snort.org/rules/community.md5.
2017-05-16 17:51:57,674 - <ERROR> -- Failed to check remote checksum: HTTP Error 406: Not Acceptable
2017-05-16 17:51:57,675 - <INFO> -- Fetching https://www.snort.org/rules/community.
100% - 295175/295175
2017-05-16 17:51:59,439 - <INFO> -- Done.
2017-05-16 17:51:59,439 - <INFO> -- Loaded 0 rules.
2017-05-16 17:51:59,440 - <INFO> -- Disabled 0 rules.
2017-05-16 17:51:59,440 - <INFO> -- Enabled 0 rules.
2017-05-16 17:51:59,440 - <INFO> -- Modified 0 rules.
2017-05-16 17:51:59,440 - <INFO> -- Dropped 0 rules.
2017-05-16 17:51:59,440 - <INFO> -- Enabled 0 rules for flowbit dependencies.
2017-05-16 17:51:59,440 - <INFO> -- Writing community.rules: added: 0; removed 0; modified: 0
But if I manually download the file from my browser what I get is a .tar
archive which contains a community.rules
with about 3500 rules.
With the latest version (master) also snort registered rules download is supperted via the --url
parameter:
idstools-rulecat --url "https://www.snort.org/rules/snortrules-snapshot-2976.tar.gz?oinkcode=<onikcode>"
What I propose is to add a couple of arguments to handle snort like ET:
--snort-version Snort version (choose from 2976, 2983, 2990)
--snort-community Download snort community rules
--snort-registered <oink-code> Download snort registered rules providing your oink-code
The URLs for snort rules are:
# snort community
https://www.snort.org/rules/community
# snort registered
https://www.snort.org/rules/snortrules-snapshot-<snort-version>.tar.gz?oinkcode=<oinkcode>
Rule:
alert tcp 93.174.88.0/21 any -> $HOME_NET any (msg:"SN: Inbound TCP traffic from suspect network (AS29073 - NL)"; flags:S; reference:url,https://suspect-networks.io/networks/cidr/13/; threshold: type limit, track by_dst, seconds 30, count 1; classtype:misc-attack; sid:71918985; rev:1;)
fast hit:
05/02-16:18:54.125546 [**] [1:71918985:1] SN: Inbound TCP traffic from suspect network (AS29073 - NL) [**] [Classification: Misc Attack] [Priority: 2] {TCP} 93.174.95.106:40082 -> x.x.x.x:2222
json hit from the unified file:
{"timestamp": "2017-05-02T16:18:54.125546+0000", "sensor_id": 0, "event_id": 1738, "event_second": 1493741934, "event_type": "alert", "src_ip": "93.174.95.106", "src_port": 40082, "dest_ip": "x.x.x.x", "dest_port": 2222, "proto": "TCP", "flow_id": 5007401709, "alert": {"action": "allowed", "gid": 1, "signature_id": 71918985, "rev": 1, "signature": null, "category": "Misc Attack", "severity": 2}, "packet": "fA7<snip>AAAAAAAAAAA==", "packet_printable": "|...H.0.\r\n........E(.(.v..q.2.]._j...C....;.......P.............", "packet_info": {"linktype": 1}}
It appears that the ids_alert field is missing when these hit.
When reading a unified2 log from Snort, the decode_record
function fails with the following error:
struct.error: unpack requires a bytes object of length 84
The EVENT_IP6_FIELDS
tuple tells the decoder to look for only 64 bytes. I am not sure if this is because of the sensor I used or because Snort has changed the signature. I fixed the bug by replacing
Field("source-ip.raw", 4, "16s"), Field("destination-ip.raw", 4, "16s")
with,
Field("source-ip.raw", 16, "16s"), Field("destination-ip.raw", 16, "16s")
Is that possible? Say I have u2eve reading two separate u2 files from two different snort instances....can I write to just one json file? Or do I have to write to 2 separate ones? Thanks Jason!
I'm trying to get at what analysts call "payload" data using idstools-u2json. I tried using the master branch with the additional params of --packets --extra-data, and I never see anything in extra-data field, but I do see what looks like binary data in the 'packet'(s) field.
When using Barnyard2 it places the payload data, if any, in the mysql/db table called 'data', which is represented in the column 'data_payload' as a hex string ... it would be nice to store this as both
hex and ascii in elasticsearch.
Also, I can't find the sensor(sid), hostname, or interface fields ... as sometimes our snort events are gathered from multiple sensors.
Suggestions ?
... apologies if this is the wrong place to ask a question as this isn't a bug/fix
From the docs:
http://idstools.readthedocs.org/en/latest/maps.html
and
http://blog.jasonish.org/2013/07/easy-unified2-file-reading-in-python.html
... figured out that MsgMap should be SignatureMap, but running this:
from idstools import maps
sigmap = maps.SignatureMap()
sigmap.load_generator_map(open("/etc/snort/gen-msg.map"))
sigmap.load_signature_map(open("/etc/snort/community-sid-msg.map"))
print(sigmap.get(1, 2495))
print(sigmap.get(1, 1))
gives the following error:
Traceback (most recent call last):
File "resolve_event_msg_and_class_name.py", line 6, in <module>
sigmap.load_signature_map(open("/etc/snort/community-sid-msg.map"))
File "/usr/local/lib/python2.7/dist-packages/idstools/maps.py", line 136, in load_signature_map
"sid": int(parts[0]),
ValueError: invalid literal for int() with base 10: ''
I tried lots of gids/sids but no luck ... suggestions ?
Otherwise, I like these tools as it's hard to find snort related stuff ... thanks for this :)
I'm trying to use idstools.rule.parse
to validate rules, and this one has an invalid direction:
idstools.rule.parse("""#alert tcp $HOME_NET any >>> $EXTERNAL_NET $HTTP_PORTS ...)
but the rule is returned as valid
... I think the only valid directions are '<>', '<-', '->'
... everything else seems to work for validation
This app works great with snort.u2 files...I'm wondering if you can integrate appid u2 files as well?
[15:52:26 tester:/opt/snort/var/log$] sudo idstools-u2json --snort-conf /opt/snort/etc/snort.conf -C /opt/snort/etc/classification.config -S /opt/snort/etc/sid-msg.map -G /opt/snort/etc/gen-msg.map --output /tmp/test.json --stdout appid-events-u2.log.1474407059
INFO: Loaded 48171 rule message map entries.
INFO: Loaded 70 classifications.
Traceback (most recent call last):
File "/usr/local/bin/idstools-u2json", line 12, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/idstools/scripts/u2json.py", line 318, in main
for record in reader:
File "/usr/local/lib/python2.7/dist-packages/idstools/unified2.py", line 606, in next
record = self.reader.next()
File "/usr/local/lib/python2.7/dist-packages/idstools/unified2.py", line 569, in next
return read_record(self.fileobj)
File "/usr/local/lib/python2.7/dist-packages/idstools/unified2.py", line 539, in read_record
return decode_record(rtype, buf)
File "/usr/local/lib/python2.7/dist-packages/idstools/unified2.py", line 500, in decode_record
return DECODERS[record_type].decode(buf)
File "/usr/local/lib/python2.7/dist-packages/idstools/unified2.py", line 347, in decode
values = struct.unpack(self.format, buf)
struct.error: unpack requires a string argument of length 76
Thank you.
When parsing a merged rule file, using in example rule.parse_file
, is not possible to assign the group parameter and the rule group is not parsed into the object.
However the group is somehow standardized in the rule msg
using this convention:
msg second word | rule file |
---|---|
CNC | botcc.rules |
CINS | ciarmy.rules |
COMPROMISED | compromised.rules |
DROP | drop.rules (also dshield.rules ) |
ACTIVEX | emerging-activex.rules |
ATTACK_RESPONSE | emerging-attack_response.rules |
CHAT | emerging-chat.rules |
CURRENT_EVENTS | emerging-current_events.rules |
DELETED | emerging-deleted.rules |
DNS | emerging-dns.rules |
DOS | emerging-dos.rules |
EXPLOIT | emerging-exploit.rules |
FTP | emerging-ftp.rules |
GAMES | emerging-games.rules |
ICMP_INFO | emerging-icmp_info.rules |
ICMP | emerging-icmp.rules |
IMAP | emerging-imap.rules |
INAPPROPRIATE | emerging-inappropriate.rules |
INFO | emerging-info.rules |
MALWARE | emerging-malware.rules |
MISC | emerging-misc.rules |
MOBILE_MALWARE | emerging-mobile_malware.rules |
NETBIOS | emerging-netbios.rules |
P2P | emerging-p2p.rules |
POLICY | emerging-policy.rules |
POP3 | emerging-pop3.rules |
RPC | emerging-rpc.rules |
SCADA | emerging-scada.rules |
SCAN | emerging-scan.rules |
SHELLCODE | emerging-shellcode.rules |
SMTP | emerging-smtp.rules |
SNMP | emerging-snmp.rules |
SQL | emerging-sql.rules |
TELNET | emerging-telnet.rules |
TFTP | emerging-tftp.rules |
TROJAN | emerging-trojan.rules |
USER_AGENTS | emerging-user_agents.rules |
VOIP | emerging-voip.rules |
WEB_CLIENT | emerging-web_client.rules |
WEB_SERVER | emerging-web_server.rules |
WEB_SPECIFIC_APPS | emerging-web_specific_apps.rules |
WORM | emerging-worm.rules |
TOR | tor.rules |
So, if rules are well writtent and no group is provided the parser should extract it from the msg
second word.
If you find this interesting I can make a PR for this feature.
This is a request encouraged by Fedora packaging policy
https://bugzilla.redhat.com/show_bug.cgi?id=1398369
https://fedoraproject.org/wiki/Packaging:LicensingGuidelines?rd=Packaging/LicensingGuidelines#License_Clarification
This fails:
python2 /usr/bin/idstools-eve2pcap merged.log.eve -o merged.log.eve2pcap --dlt RAW
python3 /usr/bin/idstools-eve2pcap merged.log.eve -o merged.log.eve2pcap --dlt RAW
ValueError: time data '2017-01-02T09:51:30.644790+0100' does not match format '%Y-%m-%dT%H:%M:%S.%f-0600'
with the following merged.log.eve
:
{"timestamp": "2017-01-02T09:51:30.644790+0100", "sensor_id": 0, "event_type": "alert", "src_ip": "10.255.2.100", "src_port": 50621, "dest_ip": "10.255.2.160", "dest_port": 80, "proto": "TCP", "flow_id": 100663492, "alert": {"action": "allowed", "gid": 1, "signature_id": 10000002, "rev": 1, "signature": null, "category": null, "severity": 0}, "packet": "RQAAPBCRQABABg8qCv8CZAr/AqDFvQBQfykbqgAAAACgAnIQTs0AAAIEBbQEAggKAAULOwAAAAABAwMH"}
The problem is in
py-idstools/idstools/scripts/eve2pcap.py
Line 133 in 5862a93
On python >= 3.2 one could do:
dt = datetime.strptime(timestamp, "%Y-%m-%dT%H:%M:%S.%f%z")
but for both python2 and python3 maybe this?
from dateutil import parser
dt = parser.parse(timestamp)
According to the docs https://www.snort.org/faq/readme-unified2/ and http://manual-snort-org.s3-website-us-east-1.amazonaws.com/node44.html the Extra Data record only supports 24 bytes where your EXTRA_DATA_FIELDS enum contains 32, the first two fields (event-type, event-length) do not appear to be documented. As these two sites differ in portions referencing the same events it could be a documentation issue and not your code.
ip-source => source-ip
ip-destination => destination-ip
apparently it changed a while ago.
Instead mask it as or something.
Topic :) Thank you.
Here is the end of the run. Rules download properly, but the merged rule file is never updated. If I run with no arguments it grabs the open source rules and does not appear to error out.
Enabled 480 rules for flowbit dependencies.
Recording file /etc/suricata/rules/merged.rules with hash '94bbee8c808b0d5052f8917e51b0749a'.
Traceback (most recent call last):
File "/usr/local/bin/idstools-rulecat", line 12, in
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/idstools/scripts/rulecat.py", line 752, in main
write_merged(args.merged, rulemap)
File "/usr/local/lib/python2.7/dist-packages/idstools/scripts/rulecat.py", line 415, in write_merged
idstools.rule.parse_fileobj(open(filename)))
File "/usr/local/lib/python2.7/dist-packages/idstools/scripts/rulecat.py", line 454, in build_rule_map
if rule.id not in rulemap:
File "/usr/local/lib/python2.7/dist-packages/idstools/rule.py", line 158, in id
return (int(self.gid), int(self.sid))
TypeError: int() argument must be a string or a number, not 'NoneType'
Here is my config:
--etpro=
--suricata=/usr/local/bin/suricata
--merged=/etc/suricata/rules/merged.rules
--sid-msg-map-2=/etc/suricata/rules/gen-msg2.map
--sid-msg-map=/etc/suricata/rules/gen-msg.map
--enable=/etc/suricata/rules_enable.conf
--disable=/etc/suricata/rules_disable.conf
--modify=/etc/suricata/rules_modify.conf
--threshold-in=/etc/suricata/threshold.in
--threshold-out=/etc/suricata/threshold.config
--post-hook=sudo kill -USR2 $(cat /var/run/suricata.pid)
--temp=/etc/suricata/rules/raw/
-v
I've verified that all the paths listed above exist. Help is appreciated. Not sure if this is a problem with my version or a problem with the way I've configured it. Used pip to install, idstools (0.5.2)
The Rule
object has no ruleset
attribute.
Examples:
alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET MALWARE Windows executable sent when remote host claims to send an image 2"; flow: established,from_server; content:"|0d 0a|Content-Type|3a| image/jpeg|0d 0a 0d 0a|MZ"; fast_pattern:12,20; classtype:trojan-activity; sid:2020757; rev:2;)
is from ET ruleset
alert ip any any -> any any (msg:"GPL ATTACK_RESPONSE id check returned root"; content:"uid=0|28|root|29|"; classtype:bad-unknown; sid:2100498; rev:7; metadata:created_at 2010_09_23, updated_at 2010_09_23;)
is from SNORT GPL
Usually the ruleset is the first word in the Rule.msg
attribute, so it should not be difficult to extract. If someone does not use this convention it should be possible to opt-out this parsing function, maybe letting the parser get this value as an optional argument.
Right now, the default log level is info, adding -v is debug. A better option for -v is somewhere between debug and info, then perhaps -vv for debug.
One idea:
rpmlint (a tool for checking RPM packaging style) uses shebang to decide whether a python file is a script.
https://github.com/jasonish/py-idstools/tree/1dccb619555097cbbddb60095fd8f7da385c722a/idstools/scripts packages the following files as non-executable scripts under the python modules directory:
idstools/scripts/u2fast.py
idstools/scripts/u2spewfoo.py
idstools/scripts/u2eve.py
idstools/scripts/u2json.py
idstools/scripts/eve2pcap.py
idstools/scripts/rulecat.py
idstools/scripts/gensidmsgmap.py
The recommended handling, as reported by rpmlint is to remove the shebang "#! /usr/bin/env python" from those files, like in the case of idstools/scripts/dumpdynamicrules.py
Not an issue really, but I noticed there's something called ruleman ... are there any docs or demos of how to use this feature ? We are considering writing something as most of the available tools are too costly or unfinished open source projects.
The way unified2 logging works, the packets (or any non-event record) following an event record may not be associated with that event, but an event that happened before it.
Currently the record aggregator will bundle all non-event records with the most recent event until a new event record is received. Instead it will need to verify that the record does belong to the event that is currently being aggregated.
This also means that an Event object may not contain all the packet records associated it with it, as there may be an event record in the unified2 log file before a packet record for the previous event is seen.
I think the only solution is that the EventReader's make a best effort to gather all the records together in an event. Perhaps the aggregator could have a timeout where it holds onto multiple events for an amount of time waiting for more records, but there could still be a record that appears outside that window.
Still needs some thought.
based on my checking on unified2.py the EVENT_IP6_FIELDS should be fixed:
Field("source-ip.raw", 4, "16s"), ---> Field("source-ip.raw", 16, "16s"),
Field("destination-ip.raw", 4, "16s"), ---> Field("destination-ip.raw", 16, "16s"),
otherwise these errors will be return:
"struct.error: unpack requires a bytes object of length 76" on line 343
format is '>LLLLLLLLL16s16sHHBBBB' and len(buf) should be 76 but self.fixed_len is 52 now
u2eve did not recognise the last event created by snort. u2spewfoo show the last event, so it's successfully updated to the unified2 log file. u2eve only reports the prior event.
I would like to request a drop.conf for changing all/some/specified "alert" rules to "drop" rules.
When try to process a unified log file with u2json, all I get is the following:
Discarding non-event type while not in event context.
Discarding non-event type while not in event context.
Discarding non-event type while not in event context.
Discarding non-event type while not in event context.
Discarding non-event type while not in event context.
...
The command I am running is: sudo idstools-u2json --snort-conf /etc/snort/snort.conf unified2.log
and I know for a fact that data is in that file
Most likely an OE on my part but any help would be greatly appreciated
Unifiedbeats, which is just what I was looking for, is no longer being maintained. My hope and desire is to use idstools to continually read u2 files and send all the data to a listening elasticsearch instance. The bonus with u2 is the extradata and actual packet fields, otherwise just using rsyslog to read the fast file is more all I'd need. Is there a way I can achieve this with idstools as they are now? Thank you.
Rulecat script when using the option --merged
always generate a report as you can see in the code, WHICH is logged with level INFO
.
If the option -q
or --quiet
is given this results in an unnecessary execution, because the report is generated but never logged, because the loggin level raises to WARNING
.
This code shoul be preceeeded by an if
statement that checks if the logging level will permit report display before generating it.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.