This project forked from certtools/intelmq
IntelMQ is a solution for IT security teams for collecting and processing security feeds using a message queuing protocol.
Home Page: http://www.enisa.europa.eu/activities/cert/support/incident-handling-automation
License: GNU Affero General Public License v3.0
This should not be the case. Please show an error message is default boilerplate does not exist.
Not all texts will be suitable for all cases
EDIT by @sebix: Typo
For every event which passes through the squelcher, the whole ruleset is tranversed and a postgresql lookup is made. This seems like a good candidate for tuning (later on).
Just writing this down so that we don't forget.
See ISO8601. Always always add the TZ.
Also important that we keep the lexicographic ordering
(sorry for the confusing subject title, I'll try to explain):
When you start intelmqcli and an ASN has many events and you go to the detailed view of that ASN, the text is hardly shown (just the first line) and then all the hundreds of event lines are shown. However, if we have many event lines, all the test will scroll by and you won't be able to see the whole picture anymore.
Here is my proposal: also have the events text displayed via less(1).
TBD... there might be a better solution.
Right now, if no taxonomy is given, it gets set to '%' in RTIR. It's better to user 'other' or 'unknown' for that. (need to compare with eCSIRT II)
... just a short description what it does, what it is and a list of all potential parameters (see other tickets)
This is quite important.
The feed.code field has a code name for each feeder. Some feeds don't want to be named when sending notifications to clients. Hence the feed.code. It allows to anonymize but still keep a 1-1 mapping of feed.name and feed.code.
I propose the following solution for intelmqcli:
For the general process:
2015-11-03 14:38:58,240 - squelcher-expert - DEBUG - Receive message u'{"classification.taxonomy": "Malicious Code", "feed.url": "https://example.com/today", "time.observation": "2015-10-08T01:38:08+00:00", "raw": "XXXXXX=", "classification.identifier": "dga", "__type": "Event", "classification.type": "c&c", "source.fqdn": "tsvridclsswsow.me", "feed.name": "Fraunhofer DGA"}'...
2015-11-03 14:38:58,240 - squelcher-expert - ERROR - Bot has found a problem.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/intelmq-1.0.0-py2.7.egg/intelmq/lib/bot.py", line 97, in start
self.process()
File "/usr/local/lib/python2.7/dist-packages/intelmq-1.0.0-py2.7.egg/intelmq/bots/experts/squelcher/expert.py", line 65, in process
in_net = (ip_address(event['source.ip']) in
KeyError: u'source.ip'
I think the squelcher assumes that source.ip is always defined. But that really actually depends on the processing chain. In the case of domains arriving, we do not necessarily have an IP address.
What to do? Can we define an "ignore" behaviour for the squelcher?
Like something like report in the log file, but continue?
Hi all,
It took me a while to realize that https://github.com/certtools/intelmq and https://github.com/certat/intelmq have diverged a bit.
Would it make sense to merge some of the patches upstream? Especially intelmqcli.
The ISPs should get a quick info what this mail is about.
CONFIG[key].update(foo) does not work.
Instead use CONFIG.update(user_config).
(line 38)
So far, at cert.at we always attach the data (CSV file) to the investigation. So, it really is a MIME attachment and not inline text. We should stay consistent here IMHO.
from the squelcher code:
SELECT_QUERY = '''
SELECT COUNT(*) FROM {table}
WHERE
"time.source" + INTERVAL '%s HOURS' > LOCALTIMESTAMP AND
"classification.type" = %s AND
"classification.identifier" = %s AND
"source.ip" = %s AND
notify IS TRUE
'''
Let's keep that in seconds please just for the sake of consistency.
It will help the recipient of the investigation report to know which URL (feed.name as well as feed.url) to look up in order to verify our claim.
It makes sense to add this info IMHO.
At work a feature request appeared: can you please implement a command line parameter for intelmqcli so that it will only fetch data from a specific feed. For example:
$ intelmqcli --feed=abuse.ch ....
Thank you!
The way it works at our org, it would be good to send the events which can not be assigned to a specific ASN/abuse contact email address to some special address @ cert.at
When doing a test and setting the recipient to a test email address, you are afterwards asked if you want to save this email.
Save recipient '[email protected]' for ASNs 760? [Y/n]n
However, if a contact for that ASN already exists, what then? Add the contact ("," separated list)? Or overwrite it?
In any case, I believe it is important to have a safety-check there.
Otherwise it's too easy to press Y by default.
And that can screw up our contactdb which we developed over many years.
The ("boilerplate") texts are stored in the database as (key, text_body) pairs, where key is a identifier name.
Please add a command line parameter to list all keys.
This will be used for selecting the proper text (body).
add a -n / --dry-run option which does what it's name implies.
We should have a possibility to skip individual events ( in the detail view) or skip an entire group of events for an ASN.
Please use the [k] key for this (since it's already in muscle memory with old scripts) :)
Stack trace when displaying data which seems to have unicode characters in it (for example source.geolocation.city)
Traceback (most recent call last):
File "./intelmqcli", line 728, in <module>
query_by_as(asn_count[answer]['contacts'], feed=args.feed)
File "./intelmqcli", line 490, in query_by_as
query_by_as(contact, requestor=requestor, feed=feed)
File "./intelmqcli", line 450, in query_by_as
handle.write(showed_text + '\n')
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 460: ordinal not in range(128)
Example data:
classification.identifier,classification.taxonomy,classification.type,destination.port,extra,id,malware.name,notify,protocol.transport,source.asn,source.geolocation.cc,source.ip,source.network,source.port,time.observation,time.source
virut,Malicious Code,botnet drone,65520,"{""adip"": ""x.x.111.121"", ""feed_id"": ""5a0ae481c377f3584bf1504ea4e45d87""}",1221124,virut,True,tcp,8339,AT,1.2.3.4,1.2.0.0/16,491XX,2015-11-26 21:15:23+01:00,2015-11-26T20:14:53
Obviously, the "extra" field countains double double quotes (" ). Looks wrong. But might be right. Probably a result of the CSV printing ?
Weird error message in the squelcher:
"2015-12-14T13:12:45.174962": {
"bot_id": "squelcher-expert",
"message": "{\"source.geolocation.latitude\": 22.25, \"time.source\": \"2015-12-14T12:37:19+00:00\", \"feed.accuracy\": 0.0, \"feed.url\": \"XXXX\", \"source.geolocation.longitude\": 114.1667, \"raw\": \"XXXX=\", \"__type\": \"Event\", \"source.network\": \"43.X.X.0/24\", \"destination.port\": 22, \"source.registry\": \"apnic\", \"protocol.transport\": \"tcp\", \"source.ip\": \"43.X.X.56\", \"source.abuse_contact\": \"[email protected]\", \"classification.taxonomy\": \"Intrusion Attempts\", \"protocol.application\": \"ssh\", \"time.observation\": \"2015-12-14T13:11:28+00:00\", \"source.geolocation.cc\": \"HK\", \"classification.identifier\": \"sshbruteforce\", \"classification.type\": \"brute-force\", \"source.asn\": 6XX57, \"feed.name\": \"XXX\"}",
"source_queue": "squelcher-expert-queue",
"traceback": "Traceback (most recent call last):\n File \"/usr/local/lib/python2.7/dist-packages/intelmq-1.0.0-py2.7.egg/intelmq/lib/bot.py\", line 97, in start\n self.process()\n File \"/usr/local/lib/python2.7/dist-packages/intelmq-1.0.0-py2.7.egg/intelmq/bots/experts/squelcher/expert.py\", line 87, in process\n event['source.ip']))\nInterfaceError: cursor already closed\n"
In the code , there is nothing which closes the cursor or opens it again except for the start() and stop() methods.
Can it be the case that it was closed because the PG DB restarted in parallel?
What's the best option in this case?
(low prio)
OTRS is another ticket system used by many teams. We should also support OTRS in the long run.
CSV files should include a version info of the format description. Add a version field as the very last field in the CSV file. The fields there will be empty .
How to repeat:
export LANG=C--> does not work. I believe, it should also work with LANG=C. Or at least have the proper error message. Or set LANG=en_US.UTF-8 in its own ENV.
Need to reorder the fields in the CSV attachment:
in the intelmqcli tool, the menu options should be visible even when the terminal does not support bold letters.
One way to do this is to show the menu options like this:
====================================================================================================
To: abuse@myisp
Subject: 2015-10-08: 1 incidents for your AS 1234
classification.identifier,classification.type,feed.name,id,source.asn,source.ip
smtp,ids alert,BlockList.de,1234,1234,1.2.3.4
----------------------------------------------------------------------------------------------------
[b]ack, [s]end, show [t]able, change [r]equestor or [q]uit?
So, the '[', ']' brackets show things clearly.
It would be nice to have graphs within check_mk which show for example the queue depth of queues in intelmq or other similar stats.
Think about how to include this, make a proposal and / or demo.
(low prio)
We need to attach a new investigation/incident (created by intelmqcli) to an existing incident_report (passed along from the rt-fetcher as a field in the event).
Currently the squelcher relies on source.ip, source.asn and source.network. However some sources do not provide these infos. Thus, the squelcher crashes.
Additionally, if these fields are amended later, they are not relevant for squelching.
monitoring: subject says it all.
Maybe we should move this issue or solve this issue directly in certtools/intelmq.
The code is already there, but not used:
if True: # TODO: implement zip config
attachment = csvfile
attachment.seek(0)
filename = 'events.csv'
else:
attachment = io.BytesIO()
ziphandle = zipfile.ZipFile(attachment, mode='w')
ziphandle.writestr('events.csv', csvfile.getvalue())
ziphandle.close()
attachment.seek(0)
filename = 'events.zip'
Can we somehow have the zipped attachments? How about making it a cmd line option?
Make sure that RT understands the right mime foo for displaying CSV attachemtns in the receivers mailbox correctly.
if all tickets are created but intelmqcli crashes in RT.reply() (for example when text is None) then the eventdb entry is also marked as sent and does not show up in intelmqcli. This should not be the case. Either everything went through smoothly and the event is marked as sent or it should still remain open.
(--> DB transactions?)
See also #38:
--------------------------------------------------
{u'bot_id': u'postgresql-output',
u'message': {u'__type': u'Event',
u'classification.identifier': u'c&c server',
u'classification.taxonomy': u'Malicious Code',
u'classification.type': u'c&c',
u'destination.port': 6667,
u'extra': u'{"feed_id": "4e4cdd50289b071f114f0fb1a06e2eb1"}',
u'feed.name': u'n6stomp',
u'feed.url': u'stomp://n6stream.cert.pl:61614//exchange/cert.at/#',
u'notify': True,
u'raw': XXX=',
u'source.fqdn': u'XXX.example.comXXX',
u'time.observation': u'2015-12-11T12:16:21+00:00',
u'time.source': u'2015-12-11T01:36:03+00:00'},
u'source_queue': u'postgresql-output-queue',
u'traceback': [u'Traceback (most recent call last):',
u' File "/usr/local/lib/python2.7/dist-packages/intelmq-1.0.0-py2.7.egg/intelmq/lib/bot.py", line 97, in start',
u' self.process()',
u' File "/usr/local/lib/python2.7/dist-packages/intelmq-1.0.0-py2.7.egg/intelmq/bots/outputs/postgresql/output.py", line 57, in process',
u' self.cur.execute(query, values)',
u'OperationalError: SSL connection has been closed unexpectedly',
u'']}
need to discuss this how to set it from within intelmqcli
The sent_at field in the events table should be generated by the initdb script.
In order to have a consistent CSV output format, do not automatically compress/shrink the CSV file.
The shrink_csv() function should be called optionally when told so by a command line switch "--compress-csv"
make the table mode sticky (i.e. keep the state of table mode vs. non-table mode (csv)) in a single session and/or make it configurable via the .intelmqcli/intelmqcli.conf file
a completely empty report ticket is useless
New algorithm:
For all classification.taxonomy, do:
evlist = get all open events
evlist_new = map (! exists (report_id in ev) for ev in evnlist)
report_id = create incident_report(evlist_new als attachment, subject="intelmq events of " + datum + classification.taxonomy)
set report_id for all ev in evlist_new
create_incident(e, ritir_classification = classification.taxonomy)
link incident to report_id and to all existing ones (see evlist)
for all mail recipients in envlist, do:
inv_id = create investigation (mail addr, body, cvs attachment, rtir_classifcation = classification.taxonomy)
link inv_id to incident_id
Less = more
Sometimes, some events make it through all the way from the collector to the pg DB but there is no contact info nor an ASN.
Of course, not much can be done with these kinds of events.
However, when [a](sending all), you will get a stack trace for those:
Traceback (most recent call last):
File "./intelmqcli", line 681, in <module>
query_by_as(int(item['asn']), automatic=True,
TypeError: int() argument must be a string or a number, not 'NoneType'
Let's fail gracefully in this case. and just ignore the data and/or print a warning.
Traceback (most recent call last):
File "intelmq/lib/bot.py", line 97, in start
self.process()
File "/home/aaron/intelmq/intelmq/bots/experts/modify/expert.py", line 70, in process
apply_action(event, default_action)
File "/home/aaron/intelmq/intelmq/bots/experts/modify/expert.py", line 35, in apply_action
event.add(name, value.format(msg=event), sanitize=True, force=True)
KeyError: u'malware.name'
2015-11-17 18:27:50,069 - modify-expert - INFO - Last Correct Message(event): {u'source.ip': u'27.27.241.236', u'feed.accuracy': 0.0, u'feed.url': u'https://lists.blocklist.de/lists/mail.txt', u'protocol.application': u'smtp', u'event_description.text': u'IP reported as having run attacks on the service Mail, Postfix', u'time.observation': u'2015-11-17T17:03:32+00:00', u'raw': u'MjcuMjcuMjQxLjIzNg==', u'classification.identifier': u'smtp', u'classification.type': u'ids alert', u'classification.taxonomy': u'Intrusion Attempts', u'feed.name': u'
2015-11-17 18:27:50,069 - modify-expert - INFO - Current Message(event): {u'source.ip': u'223.4.233.212', u'time.source': u'2015-11-11T06:47:04+00:00', u'feed.accuracy': 0.0, u'feed.url': u'https://dragonresearchgroup.org/insight/sshpwauth.txt', u'protocol.application': u'ssh', u'time.observation': u'2015-11-17T17:03:36+00:00', u'raw': u'TkEgICAgICAgICAgIHwgIE5BICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAyMjMuNC4yMzMuMjEyICB8ICAyMDE1LTExLTExIDA2OjQ3OjA0ICB8ICBzc2hwd2F1dGg=', u'classification.type': u'brute-force', u'destination.port': 22, u
2015-11-17 18:27:50,069 - modify-expert - INFO - Bot will restart in 15 seconds.
Each bot should have the possibility to report a "heartbeat" success status message.
One simple way to do this is to call a specific URL with some parameter(s).
Example:
HTTP GET http://example.com/cgi-bin/add_service.sh?foobar
In this case, foobar is the string which identifies that the "foobar" bot was successful.
Each bot needs to know what success means and afterwards call this URL.
If a text in the boilerplate SQL table is in UTF-8, intelmqcli will die with an error:
Traceback (most recent call last):
File "intelmq/bin/intelmqcli", line 605, in <module>
query_by_as(asn_count[answer]['contacts'], feed=args.feed)
File "intelmq/bin/intelmqcli", line 344, in query_by_as
'''.format(to=requestor, subj=subject, text=text)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 271: ordinal not in range(128)
Please, let's be able to have UTF-8 there as well.
Pressing "t" a second time in intelmqcli does not switch you back to non-table mode.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
š Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ššš
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ā¤ļø Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.