flyballlabs / threatdetectionservice Goto Github PK

In preparation for the upcoming release we need to style the homepage and login page

Add the ability to search and update agents. We are going to add the MAC Address

This includes but not limited to adding, updating and deleting Agent configuration data. Once the data is added we should be able to call the agent API and return that data.

This script is responsible for configuring a Metron installation to work with the Threat Management Platform (TMP). The first release is focused on making it work with the Metron Quickstart VM.

I've created a standard approach based on the flask documentation that all of our entities within our API should implement. Please apply this approach to the User and Agent entities.

The following is in the "Adding an API" section of this document:

https://github.com/flyballlabs/threatdetectionservice/blob/master/CONTRIBUTING.md

Catch Login errors that happen because the API has a failure other then the API Server not being available.

Needs to have cmds expanded to accept 'provision' as a cmd.
API needs to have this functionality added as well.

A user should be able to specify how they want notification data to be sent. Per the following ways"

Email:

-provide an email field
-provide a score level. For example, only send messages when the score is 5 or less

SMS:

• provide a phone number
• provide a score level in which to be notified. For example, only send SMS messages when the score is 5 or more

This will allow a user to copy and paste and email into the system and then perform facial recognition on a pre-defined set of mugshots

The Notifications API will be used to notify a SOC User that a Threat was received with a particular Threat Level. The following components will need to be changed:

Metron Threat Intel Configs:

• parser/enrichment/triage_configs

Please use the issue # to commit any changes. This will allow us to track the commits related to this Issue

We need to have an approach that will automatically update the database schema when changes occur at the DB level

Users are authenticated by the api but agents should have verification as well.
At a minimum, when an agent calls to the api, it should be checked against the database, if their is a match then we know an authenticated user created that agent.
This will alleviate any script injection or zombie agent usage.

The configuration scripts should be Python based with a name of config.py. There should be a script for api and gui components such as api/config.py and gui/config.py. The scripts should provide a place to put central configuration info.

Can not get metron to enrich threat intel via threat intel bulk loader.
When I try the threat intel bulk load cmd zookeeper hangs at connection, see output below:

2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=2.6.32-642.6.2.el6.x86_64
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=root
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/root
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/assets-hosts
2017-01-16 02:38:57,726 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=node1:2181 sessionTimeout=90000 watcher=hconnection-0x4671115f0x0, quorum=node1:2181, baseZNode=/hbase-unsecure
2017-01-16 02:38:57,747 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Opening socket connection to server node1/192.168.66.121:2181. Will not attempt to authenticate using SASL (unknown error)
2017-01-16 02:38:57,754 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Socket connection established to node1/192.168.66.121:2181, initiating session
2017-01-16 02:38:57,756 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-16 02:38:57,861 WARN [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=node1:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
2017-01-16 02:38:59,233 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Opening socket connection to server node1/192.168.66.121:2181. Will not attempt to authenticate using SASL (unknown error)
2017-01-16 02:38:59,234 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Socket connection established to node1/192.168.66.121:2181, initiating session
2017-01-16 02:38:59,234 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-16 02:38:59,334 WARN [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=node1:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
2017-01-16 02:39:01,153 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Opening socket connection to server node1/192.168.66.121:2181. Will not attempt to authenticate using SASL (unknown error)
2017-01-16 02:39:01,153 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Socket connection established to node1/192.168.66.121:2181, initiating session
2017-01-16 02:39:01,155 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect

After replacing the ojdbc.jar file mentioned in the error zookeeper client is unable to connect. See below:

2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=2.6.32-642.6.2.el6.x86_64
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=root
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/root
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/assets-hosts
2017-01-16 02:38:57,726 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=node1:2181 sessionTimeout=90000 watcher=hconnection-0x4671115f0x0, quorum=node1:2181, baseZNode=/hbase-unsecure
2017-01-16 02:38:57,747 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Opening socket connection to server node1/192.168.66.121:2181. Will not attempt to authenticate using SASL (unknown error)
2017-01-16 02:38:57,754 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Socket connection established to node1/192.168.66.121:2181, initiating session
2017-01-16 02:38:57,756 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-16 02:38:57,861 WARN [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=node1:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid

Agents need to be searchable through gui.
Need to be able to add agents using gui.
Should be searchable by company and per single agent as well.
Should be able to add agent to database.