The csirtg-smrt-v1 from csirtgadgets

problem: does not support taxii fetcher

https://github.com/TAXIIProject/libtaxii
http://libtaxii.readthedocs.org/en/latest/getting_started.html
https://pypi.python.org/pypi/taxii-services
http://libtaxii.readthedocs.org/en/stable/

syslog support

tail syslog file
parse syslog
listen on syslog socket

problem: --fields flag for output unsupported

Uncaught KeyboardInterrupt during sleep

Hi,

following will cause an unclean exit while using systemd.

Nov 28 12:28:26 localhost csirtg-smrt[188175]: Traceback (most recent call last):
Nov 28 12:28:26 localhost csirtg-smrt[188175]:   File "/usr/local/bin/csirtg-smrt", line 9, in <module>
Nov 28 12:28:26 localhost csirtg-smrt[188175]:     load_entry_point('csirtg-smrt==0.0.0a12', 'console_scripts', 'csirtg-smrt')()
Nov 28 12:28:26 localhost csirtg-smrt[188175]:   File "/usr/local/lib/python2.7/dist-packages/csirtg_smrt/smrt.py", line 232, in main
Nov 28 12:28:26 localhost csirtg-smrt[188175]:     sleep((r * 60))
Nov 28 12:28:26 localhost csirtg-smrt[188175]: KeyboardInterrupt

Nov 28 12:28:26 localhost systemd[1]: csirtg-smrt.service: Main process exited, code=exited, status=1/FAILURE

Commit after IssueID

incorporate with dshield

write output plugin

https://github.com/xme/ossec2dshield/blob/master/ossec2dshield.pl

csirtg-smrt needs to keep track of what it submits

remote feed caching

in cifv2 we cache a feed appended with the feed name in the feed config. This causes us to download the same feed multiple times if we were to do something like this:

dga-feed-cryptolocker:
  remote: http://osint.bambenekconsulting.com/feeds/dga-feed.txt
  pattern: ^([^,]+)\,Domain used by (Cryptolocker[^,]+)\,([^,]+)\,([^\r\n]+)
dga-feed-p2p-gameover-zeus:
  remote: http://osint.bambenekconsulting.com/feeds/dga-feed.txt
  pattern: ^([^,]+)\,Domain used by (P2P Gameover Zeus[^,]+)\,([^,]+)\,([^\r\n]+)

This results in dga-feed.txt downloaded multiple times:

ls -lh /var/smrt/cache/osint.bambenekconsulting.com-dga*
-rw-rw-r-- 1 cif cif 83M Sep 20 20:15 /var/smrt/cache/osint.bambenekconsulting.com-dga-feed-cryptolocker
-rw-rw-r-- 1 cif cif 83M Sep 20 20:15 /var/smrt/cache/osint.bambenekconsulting.com-dga-feed-p2p-gameover-zeus

In v3 let's attempt to only download dga-feed.txt one time, cache it and then parsed the single cached feed.

csirtgadgets/bearded-avenger#16

slackbot listener

http://hirelofty.com/blog/how-build-slack-bot-mimics-your-colleague/

problem: does not parse stix messages

probably for good reason, but...

native grok integration

https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
korg
libgrok
https://www.elastic.co/blog/little-logstash-lessons-part-using-grok-mutate-type-data

pastebin example

PSN ComboLIST 5K+

[email protected]:freedom1
[email protected]:18903417
[email protected]:vale123
[email protected]:pikolo
[email protected]:toro1985
[email protected]:trublu21
[email protected]:9w3cpxt
[email protected]:Thomas
[email protected]:clayton5

problem: when no defaults/values are specified in the top, client errors out

windows build

https://github.com/ahupp/python-magic
https://mail.python.org/pipermail/python-list/2008-May/491014.html
http://www.darwinsys.com/file/
https://ci.appveyor.com/project/wesyoung/csirtg-smrt-py/build/1.0.2/job/j53y27raqiu55n17

problem: if feed site is down, wget aborts

Traceback (most recent call last):
  File "/usr/local/bin/csirtg-smrt", line 11, in <module>
    load_entry_point('csirtg-smrt==0+unknown', 'console_scripts', 'csirtg-smrt')()
  File "/usr/local/lib/python2.7/dist-packages/csirtg_smrt/smrt.py", line 328, in main
    raise e
subprocess.CalledProcessError: Command '['wget', '--header', 'User-Agent: csirtg-smrt/0+unknown (csirtgadgets.org)', '--timeout=120', '-q', 'http://data.phishtank.com/data/online-valid.json.gz', '-N', '-P', '/tmp/smrt/phishtank.com']' returned non-zero exit status 8

problem: there is no native apwg fetcher

solution: integrate as fetcher (or sep utility?) https://github.com/csirtgadgets/cif-apwg-py

problem: only load .yml files, not .yml~ or .yml.off files

spamcop parsing example

https://www.spamcop.net/fom-serve/cache/338.html

add --filter-indicator debug option

handle nested / complex json data structures

cif-smrt in cifv2 does not handle nested json structures well.

ability to post-process a parsed data structure

Let's use AlienVault ipv4 reputation data as an example data structure.

CSV file

60.173.9.26#3#2#Scanning Host;Malicious Host#CN#Hefei#31.863899231,117.280799866#11;3

it has multiple delimiters:

primary: hash
secondary: semi-colon.

If parsed into a key pair data structure it would look something like this:

DataStruct1

{ 
  ipv4: '60.173.9.26',
  key1: '3', 
  key2: '2',
  classification: 'Scanning Host;Malicious Host',
  CountryCode: 'CN',
  key3: 'Hefei',
  lat: '31.863899231',
  long: '117.280799866',
  key4: '11;3',
}

cif-smrt in cifv2, we could write a regex for Scanning Host;Malicious Host and replace it with a single string of our choosing. What we can't do is parse that value into multiple key-pair values and then decide to overwrite them if desired.

What would be nice in to have is, the ability to post-parse a parsed data structure a second time with cif-smrt.

CSV -> initial Data structure -> second data structure. Ideally we could go from CSV to something like this:

DataStruct2

{ 
  ipv4: '60.173.9.26',
  key1: '3', 
  key2: '2',
  classification1: 'Scanning Host',
  classification2: 'Malicious Host',
  CountryCode: 'CN',
  key3: 'Hefei',
  lat: '31.863899231',
  long: '117.280799866',
  key4: '11;3',
}

or

DataStruct2

{ 
  ipv4: '60.173.9.26',
  key1: '3', 
  key2: '2',
  classification1: 'scanners',
  classification2: 'malware',
  CountryCode: 'CN',
  key3: 'Hefei',
  lat: '31.863899231',
  long: '117.280799866',
  key4: '11;3',
}

test garwarn blog parsing

https://github.com/csirtgadgets/massive-octo-spice/blob/develop/src/rules/example/garwarn.yml

ability to see parsed data (debug out) without submission

In CIFv2, when writing a parser for a new feed, we have debug output but that debug output is seen on submission to cifv2 as cif-smrt is tightly coupled to cifv2.

In this version, cifv2 should be a destination option. So what I guess i'm really asking for is stdout and file based output in addition to cif and csirtg

stdout (csv, json, table)
file (csv,json,table)
cifv2, cifv3
csirtg

output: json

write out json string instead of to remote

problem: no tests for skip regex

problem: smrt doesn't parse cif feeds natively

solution: add native cif parser

problem: skip regex flag only exists in top level

needs to exist within each "feed" context.

--fireball mode not remembering things properly

https://github.com/csirtgadgets/csirtg-smrt-py/blob/master/csirtg_smrt/parser/pattern.py#L87

problem: in service mode, make wait interval is not configurable

solution: make it configurable

features need to be broken up into separate issues

omni-tool (data normalization)

Can be run as a service/damon
Input: http, stdin, file, tail (file), REST API (JSON), tcp socket, syslog socket
Parser: csv, pipe, regex, json, delimited, rss, xml, html, text, email, syslog, stix
Output: csv, html, json, table, xml, stix
Application support (in and out):
- CIF, csirtg, Security Onion, Suricata, Snort, Bro, HoneyDrive, Kippo, SpamAssassin, DenyHosts, Cuckoo, Logstash, CRITS, Viper, etc

problem: missing ability to only goback a few days on new feed run

fireball mode

enable bulk fireball mode

problem: altid not being variablized correctly

{'tlp': 'white', 'application': '', 'asn_desc': 'TELEF\\195\\148NICA BRASIL S.A, BR', 'firsttime': '2016-12-14T17:37:41.000000Z', 'group': 'everyone', 'timezone': 'america/sao_paulo', 'version': '0.00a0', 'longitude': -43.2333, 'altid': 'https://csirtg.io/search?q=<indicator>', 'cc': 'BR', 'protocol': '6', 'city': 'Rio de Janeiro', 'portlist': '23', 'latitude': -22.9, 'asn': '18881', 'description': 'sourced from firewall logs (incomming', 'altid_tlp': 'white', 'tags': ['scanner'], 'lasttime': '2016-12-14T17:37:41.000000Z', 'peers': [{'rir': 'lacnic', 'asn': '12956', 'cc': 'BR', 'prefix': '179.178.0.0/20'}], 'provider': 'csirtg.io', 'itype': 'ipv4', 'indicator': '179.178.0.0', 'confidence': 9}

sqlite: Unicode type received for non-unicode bind

hmm... something strange for the archiver?

csirtg-smrt[95809]: /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/sqltypes.py:185: SAWarning: Unicode type received non-unicode bind param value '170.78.158.65'. (this warning may be suppressed after 10 occurrences)

mailman listener

ignore if there are no obvious indicators
procmail recipe

problem: in service mode- should fork the loop

help with memory management

problem: skip regex does not like quotes "timetamp","ip"

not sure if it's the regex or the match...

except NotImplementedError raised on pattern parser

incorporate with deny hosts

write plugin

https://github.com/denyhosts/denyhosts/tree/master/plugins

problem: feed remote doesn't support regex

for example, if a feed is dropped in a directory with a different timetamp name, there's no way to automatically figure that out.

solution: enable remote to contain regex

break by default

should submit in non-bulk mode by default, make sure things break from the get go. then do an --bulk flag to submit as bulk (max 500/batch?). mimic this in the v3 api.

csirtgadgets / csirtg-smrt-v1 Goto Github PK

csirtg-smrt-v1's Introduction

Getting Started

Getting Involved

COPYRIGHT AND LICENCE

csirtg-smrt-v1's People

Contributors

Stargazers

Watchers

Forkers

csirtg-smrt-v1's Issues

Recommend Projects

Recommend Topics

Recommend Org