abusesa / abusehelper Goto Github PK

View Code? Open in Web Editor NEW

114.0 20.0 18.0 4.66 MB

A framework for receiving and redistributing abuse feeds

License: MIT License

Python 100.00%

abusehelper's Introduction

AbuseHelper

AbuseHelper is an open-source framework for receiving and redistributing abuse feeds and threat intel.

Running tests & linter

We run automated tests (for Python 2.6, 2.7 and PyPy) and flake8 linter for each repository push. View the logs at https://circleci.com/gh/abusesa/abusehelper.

To run the tests locally you need to have tox installed (for example via pip install tox). Then, while in the project directory, run:

$ tox

Changelog

See CHANGELOG.md.

Security Announcements

AbuseHelper Security Announcement 2016-01.
- Summary: Multiple places failed to check X.509 certificates leading to possibility to MITM connections.

Community extensions

This project provides the core AbuseHelper functionality, including choice bots and tools.

The AbuseHelper Community repository builds upon the core, for example by offering a fine selection of community-maintained bots.

License

Files are (c) respective copyright holders if named in the specific file, everything else is current (c) by Synopsys, Inc. Everything is licensed under MIT license, see LICENSE.

abusehelper's People

Contributors

Stargazers

Watchers

Forkers

ics-forks ibrown aaronkaplan razuz bkaathi cloudxtreme poeblu olivierh59500 rajivraj adunbrook memongo zhangjunqiang mkulasi hartl3y94 darkmoonzika acealchemycyberblaze termuxinqr35 cyberbuck

abusehelper's Issues

Yielding roomreader improperly leads to the crash of startupbot and losing track of other bot PIDs

It seems that when yielding bot in a following manner:

from abusehelper.core.startup import Bot
from abusehelper.tools.roomreader import main as roomreader

...

bot = Bot([bot configurations])
yield bot("roomreader", roomreader)

Leads to startupbot to crash with following traceback:

2016-04-18 10:18:24Z startup[2824] CRITICAL Traceback (most recent call last):
2016-04-18 10:18:24Z startup[2824] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/bot.py", line 308, in execute
2016-04-18 10:18:24Z startup[2824] CRITICAL     return self.run()
2016-04-18 10:18:24Z startup[2824] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/startup.py", line 295, in run
2016-04-18 10:18:24Z startup[2824] CRITICAL     return idiokit.main_loop(self.configs() | self.read() | self.main())
2016-04-18 10:18:24Z startup[2824] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 298, in _next
2016-04-18 10:18:24Z startup[2824] CRITICAL     next = require_stream(self._gen.send(peel_args(args)))
2016-04-18 10:18:24Z startup[2824] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/startup.py", line 285, in main
2016-04-18 10:18:24Z startup[2824] CRITICAL     self._processes[conf] = self._launch(conf), strategy
2016-04-18 10:18:24Z startup[2824] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/startup.py", line 140, in _launch
2016-04-18 10:18:24Z startup[2824] CRITICAL     path, _ = os.path.split(conf.module)
2016-04-18 10:18:24Z startup[2824] CRITICAL   File "/usr/lib64/python2.6/posixpath.py", line 82, in split
2016-04-18 10:18:24Z startup[2824] CRITICAL     i = p.rfind('/') + 1
2016-04-18 10:18:24Z startup[2824] CRITICAL AttributeError: 'function' object has no attribute 'rfind'

Correct way of launching the roomreader bot was found. Nevertheless, as the startupbot crashed in this manner, the rest of the botnet processes were left running even if botnet command showed that no bots were running for the botnet. After starting the botnet again, there were two identical botnets running on top of each other based on the PIDs along with their file handles.

Add Possibility to Filter Events Older than X

It was proposed that we add an option for source bots, which end up parsing the same source file over and over, where:

you could define a threshold value (in seconds for example).
it would imply that events with a "source time" older than X would not get sent into the pipeline
it would prevent the old events being resent in case there is a change in the feed data format for example and the state file does not "catch" the old events.

The following bots at least would benefit from this option:

abusehelper.bots.openbl.openblbot
abusehelper.bots.phishtank.phishtankbot

Remove abusehelper.core.serialize backwards compatibility code

Between March and April 2013 abusehelper.core.serialize switched to a more space-efficient way of serializing dictionary object. This led to incompatibilities between bots deployed to different machines, as all installations couldn't always be updated simultaneously. So those changes were reverted, but with a supposedly temporary addition to also accept the Mar-Apr 2013 format as well.

As no version since has serialized dictionary objects in the Mar-Apr 2013 format we can ~safely remove the temporary code. Just remember to tell about this in the changelog.

Shadowserver Open-mDNS report type

Shadowsserver has added a new report type that we should support: https://www.shadowserver.org/wiki/pmwiki.php/Services/Open-mDNS

OpenBL Bot Crash

2016-06-15 21:17:46Z openbl[17092] CRITICAL Traceback (most recent call last):
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/bot.py", line 308, in execute
2016-06-15 21:17:46Z openbl[17092] CRITICAL     return self.run()
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/bot.py", line 417, in run
2016-06-15 21:17:46Z openbl[17092] CRITICAL     return idiokit.main_loop(throw_stop_on_signal() | self._run())
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-06-15 21:17:46Z openbl[17092] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/bot.py", line 406, in _run
2016-06-15 21:17:46Z openbl[17092] CRITICAL     yield self.lobby.offer(self.bot_name, service)
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-06-15 21:17:46Z openbl[17092] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/services.py", line 229, in offer
2016-06-15 21:17:46Z openbl[17092] CRITICAL     yield self.fork() | service.run()
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-06-15 21:17:46Z openbl[17092] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/services.py", line 307, in run
2016-06-15 21:17:46Z openbl[17092] CRITICAL     state = yield self.errors | self.kill_sessions() | self.main(state)
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-06-15 21:17:46Z openbl[17092] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/bot.py", line 628, in main
2016-06-15 21:17:46Z openbl[17092] CRITICAL     cleanup, arg = yield self._poll_queue.wait()
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-06-15 21:17:46Z openbl[17092] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/utils.py", line 448, in wait
2016-06-15 21:17:46Z openbl[17092] CRITICAL     yield waiter | idiokit.sleep(timeout)
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-06-15 21:17:46Z openbl[17092] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/services.py", line 325, in kill_sessions
2016-06-15 21:17:46Z openbl[17092] CRITICAL     yield idiokit.consume()
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-06-15 21:17:46Z openbl[17092] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 735, in map
2016-06-15 21:17:46Z openbl[17092] CRITICAL     value = yield next()
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-06-15 21:17:46Z openbl[17092] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 735, in map
2016-06-15 21:17:46Z openbl[17092] CRITICAL     value = yield next()
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/services.py", line 340, in _guarded
2016-06-15 21:17:46Z openbl[17092] CRITICAL     state = yield session
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 354, in _next
2016-06-15 21:17:46Z openbl[17092] CRITICAL     next = require_stream(self._gen.send(peel_args(args)))
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/bot.py", line 600, in feed
2016-06-15 21:17:46Z openbl[17092] CRITICAL     yield self.poll(*key) | self.dedup(key)
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-06-15 21:17:46Z openbl[17092] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/bot.py", line 570, in dedup
2016-06-15 21:17:46Z openbl[17092] CRITICAL     event = yield idiokit.next()
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 354, in _next
2016-06-15 21:17:46Z openbl[17092] CRITICAL     next = require_stream(self._gen.send(peel_args(args)))
2016-06-15 21:17:46Z openbl[17092] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/bots/openbl/openblbot.py", line 43, in _poll
2016-06-15 21:17:46Z openbl[17092] CRITICAL     ip, time = line.split()
2016-06-15 21:17:46Z openbl[17092] CRITICAL ValueError: need more than 1 value to unpack

CSVReader Error Handling

It seems Shadowserver has sent out a batch of broken CSV files with new-line chracters in unquoted fields.

2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL Traceback (most recent call last):
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/bot.py", line 313, in execute
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     return self.run()
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/bot.py", line 421, in run
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     return idiokit.main_loop(throw_stop_on_signal() | self._run())
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/bot.py", line 410, in _run
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     yield self.lobby.offer(self.bot_name, service)
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/services.py", line 229, in offer
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     yield self.fork() | service.run()
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/services.py", line 307, in run
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     state = yield self.errors | self.kill_sessions() | self.main(state)
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 735, in map
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     value = yield next()
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/services.py", line 325, in kill_sessions
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     yield idiokit.consume()
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 735, in map
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     value = yield next()
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 735, in map
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     value = yield next()
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/services.py", line 340, in _guarded
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     state = yield session
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/imapbot.py", line 229, in poll
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     yield self.fetch_mails(self.filter)
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/imapbot.py", line 308, in fetch_mails
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     yield self.handle(parts)
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/imapbot.py", line 328, in handle
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     skip_rest = yield handler(headers, fileobj)
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/shadowservermail.py", line 124, in handle_text_plain
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     result = yield self.parse_csv(headers, filename, fileobj)
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/shadowservermail.py", line 72, in parse_csv
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     self.normalize(subject, match.groupdict()))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/shadowservermail.py", line 60, in normalize
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     yield idiokit.send(event)
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 354, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.send(peel_args(args)))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/utils.py", line 352, in csv_to_events
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     for row in _CSVReader(fileobj, charset=charset, delimiter=delimiter):
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/utils.py", line 339, in __iter__
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     for row in reader:
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?

Tracking issue: Convert default URLS from HTTP -> HTTPS where possible

Many bots fetch data from sources via HTTP. Convert these into HTTPS where possible.

This is tracking issue to collect all pull requests modifying bots. One pull request per bot.

Remove abusehelper.core.startup and .runtime backwards compatibility code

In June 2011 abusehelper.core.startup and abusehelper.core.runtime were overhauled, and to this day they both contain small blocks of code to facilitating backwards compatibility. For reference here are links to the blocks in the current master branch tip:

In the following 4+ years there have been several compatibility-breaking changes anyway. Most likely we can remove this compatibility code, with a prominent note in the changelog.

OpenBL Bot Undefined Name

abusehelper/bots/openbl/openblbot.py:62:20: F821 undefined name 'normalize_time'

abusehelper.core.utils.fetch_url: SNI support

The custom HTTPS handler introduced at b7e5f72 doesn't support SNI.
Fetching a resource from a host enforcing SNI yield the following error:

2017-03-28 00:40:31Z alienvault[32602] INFO Poll skipped: Download failed: 'https://reputation.alienvault.com/reputation.data' (<urlopen error [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:661)>)

Using Python's (2.7.9+) urllib2.urlopen to fetch the same resource works as expected.

New Shadowserver report type: Open-DB2

https://www.shadowserver.org/wiki/pmwiki.php/Services/Open-DB2

Remove abusehelper.core.log.EventLogger#warn and #fatal

For logging each AbuseHelper bot has the self.log object, which is an abusehelper.core.log.EventLogger instance. EventLogger basically is an overlay on the logging.Logger class, but provides some extra methods, among them warn and fatal. They are just aliases for warning and critical, respectively.

There is no real reason for those two aliases to exist. Furthermore they don't appear to be used - at all - in the abusehelper and ahcommunity repositories. Therefore the methods can probably just be removed with a brief mention in the changelog.

incremental relaunching strategy

The StartupBot implements a strategy() method which delays bot relaunching. It would be nice if the delay would increment with each relaunch (when return code is not 0) to avoid hammering servers and filing up logs.
Example scenario:

bot exists with status other than 0;
first relaunch delay is 15 seconds;
bot exists again with a non-zero status;
relaunch delay is incremented 30 seconds;
non-zero exist;
relaunch delay 60 seconds;
...

@jviide-work suggests to have a maximum delay so we don't grow impatient while waiting for bots to start.

Move abusehelper.core.dshield out of abusehelper.core

abusehelper.core should probably have as little provider-specific code as possible. Therefore we should move the DShield feed bot out. Some options what to do with the bot:

Move it to a new abusehelper.bots.dshield subpackage.
Offer the bot to AbuseHelper Community if someone is interested in maintaining the bot.
Deprecate the bot.

Quick polling on the project's chat channel seemed to indicate that no-one there was actually using the bot in production. However it was generally considered to be good for getting something running quickly, as well as providing a good code example on how to write new feeds. This can be helped by highlighting some other bot that is relatively simple and does not too much setup effort. Maybe abusehelper.bots.malwaredomainlist or abusehelper.bots.danger.bruteforceblocker?

abusehelper.bots.archivebot.csvarchivebot depends on deprecated abusehelper.core.archivebot

This issue blocks pull request #24.

abusehelper.bots.archivebot.csvarchivebot should be either deprecated or changed to use for example abusehelper.bots.archivebot instead.

bots version

Premise: INFO Starting service 'historyexpert' version unknown

It would be nice if version.py would be updated to use the git commit hash. Ideally this should allow custom bots (I'm thinking ahcommunity package) to use this somehow. In ahcommunity we sometimes use a module level dunder, __version__ to identify bots. An idea will be to have the parent (abusehelper.core.bot.Bot) read it.
This will help identifying bot's version with a quick look in logs.

setup.py install doesn't work on PyPy

When installing AbuseHelper using PyPy (pypy setup.py install) the script doesn't find the needed packages. The problem lies in the function is_package() which is called when collecting packages in function collect_package().

The problematic function below:

def is_package(path):
    try:
        imp.find_module(".", [path])
    except ImportError:
        return False
    return True

This will always raise ImportError on PyPy with a message ImportError: No module named . The result is that the actual code is not copied to site-packages folder, only the abusehelper-*.egg-info file appears. On Python the same function seems to work, returning something like (None, 'abusehelper/.', ('', '', 5)).

abusehelper.core.shadowservermail: Crash on incomplete data & Python 2.6

@svimes-work reported that abusehelper.core.shadowservermail crashes on a some mails it's supposed to be able to handle:

2015-12-08 09:45:31Z shadowservermail[74486] INFO Handling mail u'...' from u'...'
2015-12-08 09:45:31Z shadowservermail[74486] INFO Opening a ZIP attachment
2015-12-08 09:45:31Z shadowservermail[74486] INFO Parsing CSV data from the ZIP attachment
2015-12-08 09:46:03Z shadowservermail[74486] CRITICAL Traceback (most recent call last):
2015-12-08 09:46:03Z shadowservermail[74486] CRITICAL   File "abusehelper/core/bot.py", line 302, in execute
2015-12-08 09:46:03Z shadowservermail[74486] CRITICAL     return self.run()
...
2015-12-08 09:46:03Z shadowservermail[74486] CRITICAL   File "abusehelper/core/utils.py", line 129, in csv_to_events
2015-12-08 09:46:03Z shadowservermail[74486] CRITICAL     for row in _csv_reader(fileobj, charset=charset, delimiter=delimiter):
2015-12-08 09:46:03Z shadowservermail[74486] CRITICAL   File "abusehelper/core/utils.py", line 123, in _csv_reader
2015-12-08 09:46:03Z shadowservermail[74486] CRITICAL     for row in csv.reader(lines, **keys):
2015-12-08 09:46:03Z shadowservermail[74486] CRITICAL Error: newline inside string

Turns out the crashing mail contains a compressed .csv.zip attachment. The uncompressed data is incomplete, ending in the middle of a quoted string. The URL embedded in the mail returned the same incomplete data. Here's a similar but artificial example - note that the data just ends mid-line, there's no trailing newline or anything:

"timestamp","ip","hostname"
"2015-12-08 09:51:14","192.0.2.15","www.example.com"
"2015-12-08 09:52:30","198.51.100.3","mail.examp

It seems that the crash is triggered only when running the bot against such data using Python 2.6. Python 2.7 survives just fine.

Improve path sanitization in archivebot

abusehelper.bots.archivebot uses XMPP room name when choosing location where to write archives. Room name is not sanitized, so this might allow path traversal attack from lobby. Bot might also crash or at least fail to write archive if room name contains characters that are not allowed by filesystem.

Deprecate abusehelper.core.archivebot

As part of effort to clean up abusehelper.core, we should deprecate old abusehelper.core.archivebot. If it still used widely we could move it to the community repository or if not, then just delete it.

Preferred way to save archive of channel is to use new JSON archivebot at abusehelper.bots.archivebot.archivebot. New bot produces compressed JSON output and rotates archives daily.

Add docs/Ontology.md for collaborating on the harmonization ontology

Add docs/Ontology.md for collaborating on the data harmonization ontology. Or docs/Harmonization.md, or whatever is deemed best.

Fix flake8 errors

Assigned to Teemu for now, he can delegate this forward.

$ flake8
./abusehelper/bots/archivebot/archivebot.py:91:5: E722 do not use bare except'
./abusehelper/core/bot.py:319:13: E722 do not use bare except'
./abusehelper/core/imapbot.py:192:9: E722 do not use bare except'
./abusehelper/core/log.py:37:9: E722 do not use bare except'
./abusehelper/core/roomgraph.py:222:13: E722 do not use bare except'
./abusehelper/core/runtime.py:152:9: E722 do not use bare except'
./abusehelper/core/services.py:288:13: E722 do not use bare except'
./abusehelper/core/services.py:350:13: E722 do not use bare except'
./abusehelper/core/startup.py:259:9: E722 do not use bare except'
./abusehelper/core/mailer.py:127:17: E722 do not use bare except'
./abusehelper/core/mail/imapbot.py:179:9: E722 do not use bare except'
./abusehelper/core/tests/test_utils.py:109:5: E722 do not use bare except'

These are soon ignored in tox.ini, so please remove E722 after fixing:

[flake8]
ignore = E501,E722

abusehelper.core.startup: Log the name of the signal that killed a subprocess

We could improve abusehelper.core.startup logging by making it clearer when its subprocess was terminated by a signal. There seems to be a somewhat common case of confusion when doing after-the-fact debugging: People spend time trying to figure out why a bot "crashed" when it was actually deliberately kill -9'd.

Currently when one of its subprocesses terminates abusehelper.core.startup logs something like:

2016-01-20 22:15:15Z startup[6734] INFO Bot 'greatbot' exited with return value 0

2016-01-20 22:16:07Z startup[6734] INFO Bot 'myfeed' exited with return value -9

In the latter example the bot was sent SIGKILL externally using kill -9, which we may divine from the return value. To quote the documentation for subprocess.Popen.returncode:

A negative value -N indicates that the child was terminated by signal N (Unix only).

In such cases the log message could mention that a signal was the culprit - maybe even help figure out the signal's name:

2016-01-20 22:16:07Z startup[6734] INFO Bot 'myfeed' was terminated by signal 9 (SIGKILL)

Move abusehelper.core.shadowservermail out of abusehelper.core

abusehelper.core should probably have as little provider-specific code as possible. Therefore we should move the ShadowServer mail feed bot out.

The bot seems to be so widely used that it's best to keep it in the abusehelper package by creating a subpackage abusehelper.bots.shadowserver. Two things are up in the air:

Should wait for pull request #6 ("Generalized mail parsing") to land?
If so, should we provide a limited-time transition wrapper for the bot or just urge people to move straight to the new style of launching mail feed bots (which should not be a huge change)?

New Shadowserver report type: Open-XDMCP

https://www.shadowserver.org/wiki/pmwiki.php/Services/Open-XDMCP