csirtgadgets / massive-octo-spice Goto Github PK

DEPRECATED - USE v3 (bearded-avenger)

Home Page: https://github.com/csirtgadgets/bearded-avenger-deploymentkit/wiki

License: GNU Lesser General Public License v3.0

Shell 12.49% Perl 63.08% HTML 22.46% Python 0.33% Makefile 0.84% M4 0.79%

massive-octo-spice's Introduction

CIFv2 is Entering End of Life

CIFv2 was developed around Ubuntu14, CIFv3 has been developed against Ubuntu16. If you're just starting out, you should be test driving CIFv3:

https://github.com/csirtgadgets/bearded-avenger-deploymentkit

Getting Started

Getting Involved

There are many ways to get involved with the project. If you have a new and exciting feature, or even a simple bugfix, simply fork the repo, create some simple test cases, generate a pull-request and give yourself credit!

If you've never worked on a GitHub project, this is a good piece for getting started.

COPYRIGHT AND LICENCE

Free use of this software is granted under the terms of the GNU Lesser General Public License (LGPLv3). For details see the files COPYING included with the distribution.

massive-octo-spice's People

Contributors

Stargazers

Watchers

massive-octo-spice's Issues

v2 docker app

https://github.com/Mazon/Jorgmundr

user management

cuckoo integration

token integration (was: apikeys)

Apache spikes to 100% and hangs

Try switching to apache2-mpm-prefork

Ubuntu

This will remove apache2-mpm-worker and install -prefork with handles connections a little better but at the cost of performance (negligible). If you begin to see "UNAUTHORIZED" messages, try tuning your apache2.conf MaxClient to 256 and restarting apache (under the 'mpm_prefork_module' section).

$ sudo aptitude install apache2-mpm-prefork
$ sudo service apache2 restart

cif router doesn't respect --nolog from API

cif -q asn/1234

PSGI interface

python SDK

cif-client zmq timeout

ping timeout different than send timeout...

ping timeout should be ~5secs, send timeout should be 30-60 secs

rpm

https://github.com/ansible/ansible/tree/devel/packaging/debian

gamelinux passive sensor integration

xml feeds integration

deb

example

https://github.com/ansible/ansible/tree/devel/packaging/debian

encode observables option

as sha1, sha256, etc..

something like:

...
tlp = green
observable_encoding = 'sha1hex',
...

blog parser

perl SDK

fqdn subdomain searches

https://github.com/csirtgadgets/massive-octo-spice/blob/develop/src/lib/CIF/Storage/ElasticSearch.pm#L260

cif -q subdomain.example.com will return 'example.com' results.

analytics integration

hanging off cif-router publisher

makefile issue in master

cp bin/cif-smrt blib/script/cif-smrt
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/cif-smrt
cp bin/cif-router blib/script/cif-router
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/cif-router
make[1]: *** No rule to make target `bin/cif', needed by `blib/script/cif'. Stop.
make[1]: Leaving directory `/home/andy/massive-octo-spice/src'
make: *** [default] Error 2
( cd src && CRITIC= make test )
make[1]: Entering directory `/home/andy/massive-octo-spice/src'
make[1]: *** No rule to make target `bin/cif', needed by `blib/script/cif'. Stop.
make[1]: Leaving directory `/home/andy/massive-octo-spice/src'
make: *** [test] Error 2
root@cif2:~/massive-octo-spice#

alexa top1m has urls in it?

[Tue Sep 16 18:55:06 2014] [debug] Routing to a callback.
[Tue Sep 16 18:55:06 2014] [debug] Routing to controller "CIF::REST::Observables" and action "create".
[Tue Sep 16 18:55:18 2014] [error] Attribute (observable) does not pass the type constraint because: Invalid URL 'xhamster.com/user/video' at /usr/local/lib/perl/5.18.2/Mouse/Util.pm line 383.
    Mouse::Util::throw_error('Mouse::Meta::Attribute=HASH(0x4d0e7c0)', 'Attribute (observable) does not pass the type constraint beca...', 'data', 'xhamster.com/user/video', 'depth', -1) called at /usr/local/share/perl/5.18.2/Module/PluginFinder.pm line 232
    Module::PluginFinder::construct('Module::PluginFinder=HASH(0x4c0b680)', 'HASH(0x7905c98)', 'end', 10000, 'provider', 'alexa.com', 'tags', 'whitelist', 'application', 'ARRAY(0x7d4bc18)', 'confidence', 25, 'protocol', 'tcp', 'description', 'alexa #2324', 'altid_tlp', 'green', 'altid', 'http://www.alexa.com/siteinfo/xhamster.com/user/video', 'start', ...) called at /vagrant/src/bin/../lib/CIF/ObservableFactory.pm line 27
    CIF::ObservableFactory::try {...} () called at /usr/share/perl5/Try/Tiny.pm line 81
    eval {...} called at /usr/share/perl5/Try/Tiny.pm line 72
    Try::Tiny::try('CODE(0x8e17bb0)', 'Try::Tiny::Catch=REF(0x7d48f90)') called at /vagrant/src/bin/../lib/CIF/ObservableFactory.pm line 30
    CIF::ObservableFactory::new_plugin('CIF::ObservableFactory', 'HASH(0x7905c98)') called at /vagrant/src/bin/../lib/CIF/Client.pm line 129
    CIF::Client::submit('CIF::Client=HASH(0x8e178c8)', 'HASH(0x8e1db20)') called at /vagrant/src/bin/../lib/CIF/REST/Observables.pm line 52
    CIF::REST::Observables::create('CIF::REST::Observables=HASH(0x786fa88)') called at /usr/local/share/perl/5.18.2/Mojolicious.pm line 126
    Mojolicious::__ANON__(undef, 'CIF::REST::Observables=HASH(0x786fa88)', 'CODE(0x51b87d8)', 1) called at /usr/local/share/perl/5.18.2/Mojolicious/Plugins.pm line 20
    Mojolicious::Plugins::__ANON__() called at /usr/local/share/perl/5.18.2/Mojolicious/Plugins.pm line 23
    Mojolicious::Plugins::emit_chain('Mojolicious::Plugins=HASH(0x50ca6c0)', 'around_action', 'CIF::REST::Observables=HASH(0x786fa88)', 'CODE(0x51b87d8)', 1) called at /usr/local/share/perl/5.18.2/Mojolicious/Routes.pm line 106
    Mojolicious::Routes::_action('CIF::REST=HASH(0x3e67810)', 'CIF::REST::Observables=HASH(0x786fa88)', 'CODE(0x51b87d8)', 1) called at /usr/local/share/perl/5.18.2/Mojolicious/Routes.pm line 191
    Mojolicious::Routes::_controller('Mojolicious::Routes=HASH(0x4bf59a0)', 'Mojolicious::Controller=HASH(0x519a2b8)', 'HASH(0x51b0100)', 1) called at /usr/local/share/perl/5.18.2/Mojolicious/Routes.pm line 43
    Mojolicious::Routes::continue('Mojolicious::Routes=HASH(0x4bf59a0)', 'Mojolicious::Controller=HASH(0x519a2b8)') called at /usr/local/share/perl/5.18.2/Mojolicious/Routes.pm line 45
    Mojolicious::Routes::continue('Mojolicious::Routes=HASH(0x4bf59a0)', 'Mojolicious::Controller=HASH(0x519a2b8)') called at /usr/local/share/perl/5.18.2/Mojolicious/Routes.p

cif -q peer/1.2.34

PSGI doesn't yet support the es REST queries

routeviews integration

bgp prefix searches

cif -q 1.2.3.0/24

cif-router and elastic search plugin needs to be fixed.

https://github.com/csirtgadgets/massive-octo-spice/blob/develop/src/lib/CIF/Storage/ElasticSearch.pm#L260

JSON feeds integration

regex bug in domain code

https://groups.google.com/forum/#!topic/ci-framework/VDUxNd5rPf8

I was getting this same error and found my source data had an empty domain field - www..com for instance, or 'www.fred..com'.

api doc

improve doc based on:

https://github.com/interagent/http-api-design

NLP integration

bgpmon integration

dns proxy options

https://groups.google.com/forum/?hl=en&fromgroups=#!topic/ci-framework/dvwGdgCfFT8

postprocessing plugins need to accept dns_proxy flag
Pull needs to accept dns_proxy flag for non_proxy requests
some code needs to change, will adapt for RC4.

LWP::UserAgent would need a Socket override to make this happen as it depends in the internal getbyhostname() function (the system) to do dns lookups.

ruby SDK

dns tap integration

investigate and document bind9 behaviors when using dns forwarders

https://groups.google.com/forum/#!topic/ci-framework/HWey6K5733k

cif-smrt config examples

create example pages in the wiki for smrt configs

rails api

geoip failures on ubuntu 14 LTS

CIF::Meta::GeoIPUndefined subroutine &B::perlstring called at /usr/local/share/perl/5.18.2/GeoIP2/Role/Model.pm line 63.
Compilation failed in require at /usr/local/share/perl/5.18.2/GeoIP2/Database/Reader.pm line 9.
BEGIN failed--compilation aborted at /usr/local/share/perl/5.18.2/GeoIP2/Database/Reader.pm line 9.
Compilation failed in require at lib/CIF/Meta/GeoIP.pm line 8.
BEGIN failed--compilation aborted at lib/CIF/Meta/GeoIP.pm line 8.
Compilation failed in require at (eval 158) line 2.
 at /usr/share/perl5/Module/Pluggable.pm line 32.
# Tests were run but no plan was declared and done_testing() was not seen.

this is a known issue and being resolved upstream:

maxmind/GeoIP2-perl#3

geoip meta info instead of dns lookups

http://dev.maxmind.com/geoip/geoip2/geolite2/
http://dev.maxmind.com/geoip/legacy/geolite/

move ns resolution upstream?

nginx integration

Regular Console Error - mkdir /var/smrt/cache: Permission denied

After installation of cif v2 alpha,2
root@cif2:/usr/share/elasticsearch/bin# EV: error in callback (ignoring): mkdir /var/smrt/cache: Permission denied at /opt/cif/bin/../lib/perl5/CIF/Smrt/Fetcher/Uri.pm line 89.

This seems to prevent new records being created. Running cif-smrt from the command line as root seems to prevent the error.

Python ping and query works remotely but not on server

When the py-cif-sdk cif client is run on a remote server with a ping command, the command works:

On client:
$ cif --no-verify-ssl -T 1234 -R 'https://cif.vm.local/v2' -p
roundtrip: 0.480535984039 ms

From server logs:
[Tue Jul 29 20:14:36 2014] [debug] GET "/v2/ping".
[Tue Jul 29 20:14:36 2014] [debug] Routing to a callback.
[Tue Jul 29 20:14:36 2014] [debug] Routing to a callback.
[Tue Jul 29 20:14:36 2014] [debug] 200 OK (0.001494s, 669.344/s).

However when the client is run on the same host as the CIF server, this fails:

On client:
$ cif --no-verify-ssl -T 1234 -R 'https://cif.vm.local/v2' -p
2014-07-29 20:53:39,019 - ERROR - cif.sdk.client::MainThread - request failed: 404
roundtrip: request failed: 404 ms

From server logs:
[Tue Jul 29 20:38:16 2014] [debug] GET "/v2/_ping".
[Tue Jul 29 20:38:16 2014] [debug] Template "not_found.development.html.ep" not found.
[Tue Jul 29 20:38:16 2014] [debug] Template "not_found.html.ep" not found.
[Tue Jul 29 20:38:16 2014] [debug] Rendering cached inline template "569839bd27413e47e26c8476bacb2540".
[Tue Jul 29 20:38:16 2014] [debug] Rendering cached inline template "4fcf2af99f1803a7a26c2e9b04430f8c".
[Tue Jul 29 20:38:16 2014] [debug] 404 Not Found (0.009301s, 107.515/s).

It's unclear to me why when on the local machine the "ping" turns into "_ping". I can find only one reference to "_ping":
./massive-octo-spice/src/lib/CIF/Router/RESTApp.pm:
Line 84: return response(200, { timestamp => [gettimeofday()] }) if($req_id eq '_ping');

So it's unclear to me why the different behavior when on the same server. Additionally, this 404 behavior is similar for queries.

Working with git develop branch of clients and alpha-2 of massive-octo-spice.

email parser

kibana countrycode filter bug

when a country is clicked on the map, it presents the query as uppercase country code, but elastic-search only recognizes the search in lower case.

something the way the schema is created.

Easybutton install completes with an error 'chown: cannot access ‘/var/smrt’: No such file or directory'

Installing /opt/cif/bin/cif.psgi
Installing /opt/cif/bin/cif-router
Appending installation info to /opt/cif/lib/perl5/x86_64-linux-gnu-thread-multi/perllocal.pod
make[1]: Leaving directory /home/andy/massive-octo-spice/src' [ -d /etc/cif/rules ] || /usr/bin/install -c -m 0770 -d /etc/cif/rules ( cd src/rules && find . -type d -print ) | while read dir ; do \ /usr/bin/install -c -m 0770 -d "/etc/cif/rules/$dir" ; \ done ( cd src/rules && find . -type f -print ) | while read file ; do \ /usr/bin/install -c -m 0660 "src/rules/$file" "/etc/cif/rules/$file" ; \ done [ -d /var ] || /usr/bin/install -c -m 0770 -d /var /usr/bin/install -c -m 666 "contrib/GeoLite2-City.mmdb" "/var/cache/GeoLite2-City.mmdb" chmod 0770 /etc/cif/rules chown -R cif /etc/cif/rules chgrp -R cif /etc/cif/rules ( cd elasticsearch && make init ) make[1]: Entering directory/home/andy/massive-octo-spice/elasticsearch'
/usr/bin/curl -w "\n" -XPUT 'http://localhost:9200/_template/template_cif/' -d @cif-template.json
{"acknowledged":true}
make[1]: Leaving directory `/home/andy/massive-octo-spice/elasticsearch'
copying init.d scripts...
setting /etc/default/cif
chown: cannot access ‘/var/smrt’: No such file or directory

CIF Router is not running after the easy button install.

andy@cif2:/massive-octo-spice$ curl -k -w "\n" -X GET 'https://localhost:443/v2/ping?token=1234'
{"timestamp":[1409688499,300464]}
andy@cif2:/massive-octo-spice$ sudo /opt/cif/bin/cif-smrt --testmode -d -M
[sudo] password for andy:
[2014-09-02T21:08:38,148Z][INFO][main:332]: cleaning up tmp: /var/smrt/cache
[2014-09-02T21:08:38,148Z][INFO][main:266]: checking for router...
[2014-09-02T21:08:38,149Z][INFO][CIF::Client:124]: generating ping request...
[2014-09-02T21:08:38,166Z][INFO][CIF::Client:130]: sending ping...
[2014-09-02T21:08:38,166Z][DEBUG][CIF::Client:181]: encoding...
[2014-09-02T21:08:38,166Z][DEBUG][CIF::Client:185]: sending upstream...

[2014-09-02T21:10:38,302Z][DEBUG][CIF::Client::Broker::ZeroMQ:103]: cif-router timeout...
[2014-09-02T21:10:38,302Z][WARN][CIF::Client:137]: timeout...
[2014-09-02T21:10:38,302Z][ERROR][main:269]: router unavailable... ping timeout
[2014-09-02T21:10:38,305Z][INFO][main:226]: shutting down...
andy@cif2:/massive-octo-spice$
andy@cif2:/massive-octo-spice$ sudo service cif-router start

Starting cif-router cif-router [Started]

The test mode ingest works after cif-router is started but no further ingests take place after that initial load.

easybutton fails to configure apache

[Sat Sep 13 17:02:03.616519 2014] [perl:error] [pid 21952:tid 140563636483840] [client 10.0.2.2:59893] Error while loading /opt/cif/bin/cif.psgi: Can't locate CIF.pm in @INC (you may need to install the CIF module) (@INC contains: /etc/perl /usr/local/lib/perl/5.18.2 /usr/local/share/perl/5.18.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.18 /usr/share/perl/5.18 /usr/local/lib/site_perl . /etc/apache2) at /opt/cif/bin/cif.psgi line 18.\nBEGIN failed--compilation aborted at /opt/cif/bin/cif.psgi line 18.\n

in the develop tree

DNS Masq

Using local DNS resolvers on a CIF server

Ubuntu Edition

Q: Why would you want to do this?
A: You want to be able to use LDAP or AD authentication on your CIF server for local accounts and you need (say) SRV records for this to work.

Q: Why not just use local DNS for everything then?
A: Our local DNS has blacklists; we don't want to afflict our CIF instance with things required for local system administration.

I started with an Ubuntu 12.04 installation running a vanilla CIF install. You're going to be angry with two things: resolvconf, and dnscache. Both are default in Ubuntu.

You will need: a functional krb5.conf file, and any changes you might need to make to your PAM stack. Consult your local Unix admin.

You will be doing: making it so that resolvconf doesn't screw you over, and replacing the BIND install CIF docs call for with dnsmasq. You need to use not-BIND, because BIND cannot be told "for this domain, use this dns server; for that domain, use that dns server." One thing to watch for is dnsmasq wants to also be a DHCP server.

I have not validated the below steps to ensure they are in the correct order; I distilled this document from my 500-line text file describing all the things I tried that did not work, along with miscellaneous cursing. I'd attach that for your entertainment, but it has some site-specific stuff I'd rather not share.

What I did (the short version):

Ignore the warnings and edit /etc/resolv.conf by hand to point to local DNS.
sudo dpkg --purge bind9 (and clean up any directories it leaves behind)
sudo apt-get autoremove && sudo apt-get install dnsmasq

When you do this, your resolv.conf will get touched hard.

Edit /etc/dnsmasq.conf, set:
listen-address=127.0.0.1
server=/(your AD domain)/(your site DNS server)
server=/(your values).in-addr.arpa/(your site DNS server)
sudo aptitude purge resolvconf
sudo dnsmasq restart

Some handy debugging things I learned:

sudo service dnsmasq stop && sudo dnsmasq -d
Combined with the liberal use of dig commands, helpful in debugging what queries are hitting your dnsmasq server and what happened to them afterwards.

References:

soft nofile 20000
hard nofile 30000

reference: https://groups.google.com/d/topic/ci-framework/LDNSJnf6BZg/discussion

geoip omni errors

2014-09-17T19:52:19,403Z][DEBUG][CIF::Meta::BGP:28]: checking: 65.211.1.0
[2014-09-17T19:52:19,557Z][DEBUG][CIF::Meta::GeoIP:51]: checking: 65.211.1.151
[Wed Sep 17 19:52:19 2014] [error] Can't locate object method "omni" via package "GeoIP2::Database::Reader" at /opt/cif/bin/../lib/perl5/CIF/Meta/GeoIP.pm line 54.

[Wed Sep 17 19:52:19 2014] [debug] Template "exception.development.json.ep" not found.
[Wed Sep 17 19:52:19 2014] [debug] Template "exception.json.ep" not found.
[Wed Sep 17 19:52:19 2014] [debug] Template "exception.development.html.ep" not found.
[Wed Sep 17 19:52:19 2014] [debug] Template "exception.html.ep" not found.
[Wed Sep 17 19:52:19 2014] [debug] Rendering inline template "8122878e3bc1034543706fb58b4de464".
[Wed Sep 17 19:52:19 2014] [debug] Rendering inline template "4fcf2af99f1803a7a26c2e9b04430f8c".
[Wed Sep 17 19:52:19 2014] [debug] 500 Internal Server Error (0.674932s, 1.482/s).
^C