Giter Club home page Giter Club logo

darwin's Introduction

D.A.R.W.I.N

Darwin is an open source Artificial Intelligence Framework for CyberSecurity. It can be compiled and run on both FreeBSD and Linux.

We provide packages and support for FreeBSD.

Darwin is:

  • A multi-threaded C++ engine that runs security filters that work together to improve your network security
  • A collection of agents that use the DARWIN protocol to query the security filters accordingly

Darwin is still in an alpha stage, so few filters are available at this time.

Using the provided documentation and SDK you can develop your own Darwin Filters. We are seeking help! Testers and volunteers are welcome!

Advens (www.advens.fr) also provides commercial filters for Darwin !

Filters

Compilation

Note: This code part follows the C++14 standard. Compile with g++ version 8.3.0 or later.

To compile all the filters available, please enter the following:

cmake .
make -j4

To compile a specific filter:

cmake . -DFILTER=FILTER_NAME
make -j4

You can choose a filter from this list

You can also set a filter list:

cmake . -DFILTER="FILTER_NAME1;FILTER_NAME2"
make -j4

Don't forget to unset the FILTER variable if you want to compile all the filters available afterwards:

cmake . -UFILTER
make -j4

The compiled filter will be named darwin_filter_name (note: the name is displayed at the beginning of the compilation). You will find compilation and dependencies information for each filters in the Wiki.

Usage

Usage: ./darwin [-h] [-l [DEBUG|INFO|WARNING|ERROR|CRITICAL]] filter_name socket_path config_file monitoring_socket_path pid_file output next_filter_socket_path nb_thread cache_size threshold

Positional arguments:

  • filter_name Specify the name of this filter in the logs
  • socket_path Specify the path to the unix socket for the main connection
  • config_file Specify the path to the configuration file
  • monitoring_socket_path Specify the path to the monitoring unix socket
  • pid_file Specify the path to the file containing the pid of the process
  • output Specify the filter's output
  • next_filter_socket_path Specify the path to the next filter unix socket
  • nb_thread Integer specifying the number of treatment thread for this process
  • cache_size Integer specifying cache's size
  • threshold Integer specifying the filter's threshold (if behind 100, take the filter's default threshold)

OPTIONS:

  • -h Show help and exit
  • -l [DEBUG|INFO|WARNING|ERROR|CRITICAL|DEVELOPER] Set log level to DEBUG, INFO, WARNING (default), ERROR, CRITICAL or DEVELOPER. DEVELOPER mode does not create a daemon and log level is DEBUG.

Filter Manager

Python Version

Compatible with python 3.5.3 and later.

Usage

Usage: manager.py [-h] [-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}] config_file

Positional arguments:

config_file The config file to use.

Optional arguments:

-h, --help show this help message and exit

-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}, --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL} Set log level to DEBUG, INFO, WARNING (default), ERROR or CRITICAL.

Config File

The config file is JSON formatted and contains the filters information. They MUST be formatted as follow:

{
  "session_1": {
    "exec_path": "/home/darwin/filters/darwin_session",
    "config_file": "/var/sockets/redis/redis.sock",
    "output": "LOG",
    "next_filter": "",
    "nb_thread": 5,
    "threshold": 80,
    "log_level": "DEBUG",
    "cache_size": 0
  },
  "dga_1": {
    "exec_path": "/home/darwin/filters/darwin_dga",
    "config_file": "/home/darwin/conf/fdga/fdga.conf",
    "output": "LOG",
    "next_filter": "",
    "nb_thread": 5,
    "log_level": "DEBUG",
    "cache_size": 0
  }
}

You will find more information in the Wiki

The Service

In the service directory is a rc script named darwin that is the service script. It handles the following commands: start, stop, status and restart.

Usage

Use this for debug purpose only.

Usage: manager.py [-h] [-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}] config_file

Positional arguments:

config_file The config file to use.

Optional arguments:

-h, --help show this help message and exit

-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}, --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL} Set log level to DEBUG, INFO, WARNING (default), ERROR or CRITICAL.

darwin's People

Contributors

amol-soneji avatar dependabot[bot] avatar frikilax avatar gregoiregonzalez avatar hugososzynski avatar jjourdin avatar m4jr0 avatar mystique-orca avatar ns4nti avatar thibaudcartegnie avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

darwin's Issues

[IDEA] Implement Filters statistics

What is your feature about ?
Getting some neat stats about the running filters

If your request is about an existing filter/functionality
All the filters are concerned

Describe what you'd like
Some statistics about running Filters (number of inputs processed, valid/incorrect ones, mean processing time, number of matches ? etc...), when querying monitoring socket and/or written to file(s)

Additional context
Could have stats Class to pass along with atomic variables

[IDEA] File manager

What is your feature about ?
Need a way to manage concurrent file writing/reading in multi-threaded environment : A file manager

Describe what you'd like
A singleton, share by filters. All file operation pass through it, so it can manage writing/reading in a proper way.

Additional context
See the redis manager singleton.

[BUG] Manager stops on error when starting a filter

Describe the bug
Manager stops Darwin when one filter fails to start.
Occures basically when the executable given is not found (OS error)

Plateform (please complete the following information):

  • OS (version): every OS
  • Darwin version: all actual versions

To Reproduce
Steps to reproduce the behavior:

  1. Make a darwin.conf with several valid filters, but one with wrong executable path
  2. Follow darwin_manager.log
  3. Try to start the Darwin service
  4. See error when starting the filter with wrong path, and manager stop on error (also stopping all succesfully started filters)

Expected behavior
Manager should fail on startup for this filter, but continue for next ones and only generate error in logs.
Other filters should be started normally.

[BUG] Darwin error at service stop in Vulture OS

Describe the bug
When doing a service darwin stop, CRITICAL logs pop telling that some filters (not all) cannot open UNIX socket because of a bad weak pointer.

Plateform (please complete the following information):

  • OS (version): FreeBSD 12.0
  • Darwin version: 1.0
  • Vulture BASE version: 0.9.7
  • Vulture GUI version: 0.9.92

To Reproduce
Steps to reproduce the behavior:

  1. Install Vulture 4
  2. Create a new Darwin Policy containing:
  • fconnection
  • finspection
  • fdga
  • fhostlookup
  • ftanomaly
  1. Open logs in one terminal (tail -f /var/log/darwin/darwin.log)
  2. service darwin stop

Expected behavior
Darwin filters terminating without errors.

Screenshots/logs
Try 1:

{"date":"Wed Oct  2 12:48:49 2019","level":"CRITICAL","filter":"logs_1","message":"Core::run:: Cannot open unix socket: bad_weak_ptr"} 
{"date":"Wed Oct  2 12:48:49 2019","level":"CRITICAL","filter":"content_inspection_2","message":"Core::run:: Cannot open unix socket: bad_weak_ptr"} 
{"date":"Wed Oct  2 12:48:49 2019","level":"CRITICAL","filter":"tanomaly_2","message":"Core::run:: Cannot open unix socket: bad_weak_ptr"} 

Try 2:

{"date":"Wed Oct  2 12:49:34 2019","level":"CRITICAL","filter":"logs_1","message":"Core::run:: Cannot open unix socket: bad_weak_ptr"}
{"date":"Wed Oct  2 12:49:34 2019","level":"CRITICAL","filter":"content_inspection_2","message":"Core::run:: Cannot open unix socket: bad_weak_ptr"}
{"date":"Wed Oct  2 12:49:34 2019","level":"CRITICAL","filter":"dga_2","message":"Core::run:: Cannot open unix socket: bad_weak_ptr"}
{"date":"Wed Oct  2 12:49:34 2019","level":"CRITICAL","filter":"tanomaly_2","message":"Core::run:: Cannot open unix socket: bad_weak_ptr"}

Additional context
Using Rsyslog's mmdarwin module with an impcap listener.

[BUG] Update Fpython_example

Describe the bug
Fpython_example code is becoming outdated, needs an update.

Additional context
Some development has been done on the filters, and hasn't been reflected on this one, the code should be updated to reflect new underlying mechanics for developers.

[IDEA] Add threads

What is your feature about ?
Adding threads to the filters to speedup execution.

[IDEA] Rework CMakeList.txt

What is your feature about ?
Refactoring the current CMakeList.txt, splitting it into multiple files (if possible).

Describe what you'd like
A CMakeList.txt that calls / include other cmake files containing the cmake of the différent filters.

Each cmake file is in charge of checking it's filter's dependencies.

[BUG] Log lines not matching filter_name

Describe the bug
Looks like at least one filter have non corresponding log message and log filter_name (here fconnection message "DGA").
That is confusing when trying to resolve issues.

Plateform (please complete the following information):

  • OS (version): FreeBSD (12.0)
  • Darwin version: 1.0

To Reproduce
Steps to reproduce the behavior:

  1. Configure fconnection with a non existing configuration file
  2. Start darwin
  3. Read the logs in /var/log/darwin/darwin.log

Expected behavior
Log message matching filter_name.

Screenshots/logs

{"date":"Wed Sep 25 09:29:18 2019","level":"ERROR","filter":"connection_1","message":"DGA:: Generator:: Could not open the configuration file"}

[IDEA] Refactor & Merge Anomaly and TAnomaly

What is your feature about ?

Merging Anomaly & TAnomaly algorithm in one class (if possible).

Merging both filter into one and add the possibility to choose either Anomaly or TAnomaly behavior in configuration file.

Find solution for unique redis ping

Currently, we have for each Redis Manager a function "KeepAlive" that ping the redis, but when we have many filters that use this method, it lead to a lot of ping. Find a solution to "centralize" this function.

[IDEA] Test for fanomaly

What is your feature about ?
Need to add automated test for anomaly

Describe what you'd like
Test must be tested on Ubuntu and FreeBSD

[IDEA] Use Redis PUB/SUB instead of regular keys for log alerts

What is your feature about ?
change the way the Logs filter notifies alerts in Redis.

If your request is about an existing filter/functionality
Flogs

Describe what you'd like
Logs is currently logging alerts to redis using a regular key, this key is then fetched by third-parties at regular intervals for their workflows (Vulture, for example).
The enhancement would be to instead publish alerts in a Redis channel, using its PUB/SUB system to allow third-parties to be notified immediately of new alerts (instead of regular time-interval pollings).

Additional context
Flogs could have an additional parameter, redis_channel, that allows it to publish to redis on the given channel.
This could allow Flogs to work for regular Redis keys AND for Redis channels

[IDEA] Clean injection filter

What is your feature about ?
Remove the injection filter, because it's obsolete and unstable

Describe what you'd like
Remove all the code related to the injection filter

[IDEA] Replace `sprintf` by `snprintf` in Session.cpp

What is your feature about ?

  • Replacing sprintf by snprintf at line 451 of Session.cpp
sprintf(str, "%02x%02x%02x%02x-%02x%02x-%02x%02x-%02x%02x-%02x%02x%02x%02x%02x%02x",
              header.evt_id[0], header.evt_id[1], header.evt_id[2], header.evt_id[3],
              header.evt_id[4], header.evt_id[5], header.evt_id[6], header.evt_id[7],
              header.evt_id[8], header.evt_id[9], header.evt_id[10], header.evt_id[11],
              header.evt_id[12], header.evt_id[13], header.evt_id[14], header.evt_id[15]
);

build_archive.sh failure on debian buster

Hi,
First thanks for releasing the darwin code !

I'm trying to build it on debian buster,

Have installed needed deb packages

apt install libboost-all-dev libmaxminddb-dev libhiredis-dev libevent-dev pkg-config cmake build-essential

but build_archive.sh is failing

 ./build_archive.sh 
Building filters
-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29") 
Flags: -W -Wall -Wextra
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
CMake Warning at CMakeLists.txt:24 (find_package):
  By not providing "Findfaup.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "faup", but
  CMake did not find one.

  Could not find a package configuration file provided by "faup" with any of
  the following names:

    faupConfig.cmake
    faup-config.cmake

  Add the installation prefix of "faup" to CMAKE_PREFIX_PATH or set
  "faup_DIR" to a directory containing one of the above files.  If "faup"
  provides a separate development package or SDK, be sure it has been
  installed.


-- Checking for module 'libevent'
--   Found libevent, version 2.1.8-stable
-- Checking for module 'libevent_pthreads'
--   Found libevent_pthreads, version 2.1.8-stable
-- Checking for module 'hiredis'
--   Found hiredis, version 0.14.0
-- Boost version: 1.67.0
-- Checking for module 'libmaxminddb'
--   Found libmaxminddb, version 1.3.2
Generating CMake files for the filters: USER_AGENT;HOSTLOOKUP;SESSION;REPUTATION;DGA;CONNECTION;TANOMALY;ANOMALY;LOGS;END;CONTENT_INSPECTION
Creating lib: USER_AGENT
Lib created
Creating lib: HOSTLOOKUP
Lib created
Creating lib: SESSION
Lib created
Creating lib: REPUTATION
Lib created
Creating lib: DGA
Lib created
Creating lib: CONNECTION
Lib created
Creating lib: TANOMALY
Lib created
Creating lib: ANOMALY
Lib created
Creating lib: LOGS
Lib created
Creating lib: END
Lib created
Creating lib: CONTENT_INSPECTION
Lib created
Creating filter: USER_AGENT
Executable name: darwin_user_agent
Creating filter: HOSTLOOKUP
Executable name: darwin_hostlookup
Creating filter: SESSION
Executable name: darwin_session
Creating filter: REPUTATION
Executable name: darwin_reputation
Creating filter: DGA
Executable name: darwin_dga
Creating filter: CONNECTION
Executable name: darwin_connection
Creating filter: TANOMALY
Executable name: darwin_tanomaly
Creating filter: ANOMALY
Executable name: darwin_anomaly
Creating filter: LOGS
Executable name: darwin_logs
Creating filter: END
Executable name: darwin_end
Creating filter: CONTENT_INSPECTION
Executable name: darwin_content_inspection
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
ARMADILLO_LIBRARY
    linked by target "darwin_anomaly" in directory /home/darwin
    linked by target "darwin_tanomaly" in directory /home/darwin
MLPACK_INCLUDE_DIRS
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
   used as include directory in directory /home/darwin
MLPACK_LIBRARY
    linked by target "darwin_anomaly" in directory /home/darwin
    linked by target "darwin_tanomaly" in directory /home/darwin

-- Configuring incomplete, errors occurred!
See also "/home/darwin/build/CMakeFiles/CMakeOutput.log".
See also "/home/darwin/build/CMakeFiles/CMakeError.log".
root@openvas2:/home/darwin# cat /home/darwin/build/CMakeFiles/CMakeError.log
Determining if the pthread_create exist failed with the following output:
Change Dir: /home/darwin/build/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/make" "cmTC_7dd4b/fast"
/usr/bin/make -f CMakeFiles/cmTC_7dd4b.dir/build.make CMakeFiles/cmTC_7dd4b.dir/build
make[1] : on entre dans le répertoire « /home/darwin/build/CMakeFiles/CMakeTmp »
Building C object CMakeFiles/cmTC_7dd4b.dir/CheckSymbolExists.c.o
/usr/bin/cc    -o CMakeFiles/cmTC_7dd4b.dir/CheckSymbolExists.c.o   -c /home/darwin/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c
Linking C executable cmTC_7dd4b
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_7dd4b.dir/link.txt --verbose=1
/usr/bin/cc      CMakeFiles/cmTC_7dd4b.dir/CheckSymbolExists.c.o  -o cmTC_7dd4b 
/usr/bin/ld: CMakeFiles/cmTC_7dd4b.dir/CheckSymbolExists.c.o: in function `main':
CheckSymbolExists.c:(.text+0x1b): undefined reference to `pthread_create'
collect2: error: ld returned 1 exit status
make[1]: *** [CMakeFiles/cmTC_7dd4b.dir/build.make:87: cmTC_7dd4b] Error 1
make[1] : on quitte le répertoire « /home/darwin/build/CMakeFiles/CMakeTmp »
make: *** [Makefile:121: cmTC_7dd4b/fast] Error 2

File /home/darwin/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:
/* */
#include <pthread.h>

int main(int argc, char** argv)
{
  (void)argv;
#ifndef pthread_create
  return ((int*)(&pthread_create))[argc];
#else
  (void)argc;
  return 0;
#endif
}

Determining if the function pthread_create exists in the pthreads failed with the following output:
Change Dir: /home/darwin/build/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/make" "cmTC_a5e0c/fast"
/usr/bin/make -f CMakeFiles/cmTC_a5e0c.dir/build.make CMakeFiles/cmTC_a5e0c.dir/build
make[1] : on entre dans le répertoire « /home/darwin/build/CMakeFiles/CMakeTmp »
Building C object CMakeFiles/cmTC_a5e0c.dir/CheckFunctionExists.c.o
/usr/bin/cc   -DCHECK_FUNCTION_EXISTS=pthread_create   -o CMakeFiles/cmTC_a5e0c.dir/CheckFunctionExists.c.o   -c /usr/share/cmake-3.13/Modules/CheckFunctionExists.c
Linking C executable cmTC_a5e0c
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_a5e0c.dir/link.txt --verbose=1
/usr/bin/cc  -DCHECK_FUNCTION_EXISTS=pthread_create    CMakeFiles/cmTC_a5e0c.dir/CheckFunctionExists.c.o  -o cmTC_a5e0c -lpthreads 
/usr/bin/ld : ne peut trouver -lpthreads
collect2: error: ld returned 1 exit status
make[1]: *** [CMakeFiles/cmTC_a5e0c.dir/build.make:87: cmTC_a5e0c] Error 1
make[1] : on quitte le répertoire « /home/darwin/build/CMakeFiles/CMakeTmp »
make: *** [Makefile:121: cmTC_a5e0c/fast] Error 2

Any idea ?

Fix "both" mode with chained filters

When you configure a filter with a next filter, and you call it with packet_type "both", the next filter will want to "SendToSession", but the session was already closed because it had received it's response with the first filter.
It send "Broken Pipe" and crash the filter.
Need a fix.

[IDEA] Remove flogs for an "AlertManager" in the Core

What is your feature about ?
Removing the flogs filter because it does not comply with darwin philosophy.

Creating an "AlertManager" in the Core that is able to handle alert file, redis and maybe more.

Additional context
The "AlertManager" could use a list of output interfaces (IAlertRepository).

[IDEA] Core: Add Automated testing

What is your feature about ?

  • Adding automated test for the core component
  • Valgrind test for Linux
  • Correcting any issue that occurs during testing

Additional context
Automated testing with valgrind would be nice (look for XML output).

[IDEA] Rename Tanomaly's parameter "learning_mode"

What is your feature about ?
The FTanomaly filter's body parameter "learning_mode" doesn't represent a learning mode for the IA model, but just toggles the thread detecting every X minutes entries in Redis.

If your request is about an existing filter/functionality

  • TAnomaly

Describe what you'd like
This field should be named something like "activate_detection" or something.

[IDEA] FCONNECTION: Automated test & Hardened

What is your feature about ?

  • Adding automated test for the fconnection component
  • Valgrind test for Linux
  • Correcting any issue that occurs during testing

Additional context
Automated testing with valgrind would be nice (look for XML output).

[IDEA] FHOSTLOOKUP: Automated test & Hardened

What is your feature about ?

  • Adding automated test for the core component
  • Valgrind test for Linux
  • Correcting any issue that occurs during testing

Additional context
Automated testing with valgrind would be nice (look for XML output).

[BUG] Filter usage not matching parameter's description

Describe the bug
Filter's usage arguments does not match with the arguments description of the other lines.

Plateform (please complete the following information):

  • FreeBSD (12.0)
  • Darwin version: 1.0
  • Clients version (if applicable): Starting filter from shell

To Reproduce
Steps to reproduce the behavior:

  1. Go to the filter's folder
  2. ./darwin_any_filter_name

Expected behavior
Usage parameters matching with the later description of those.

Screenshots/logs

Usage: ./darwin filter_name socket_path config_file monitoring_socket_path pid_file output next_filter_socket_path max_thread min_spare_thread max_spare_thread [OPTIONS]
  filter_name	Specify the name of this filter in the logs
  socket_path	Specify the path to the unix socket for the main connection
  config_file	Specify the path to the configuration file
  monitoring_socket_path	Specify the path to the monitoring unix socket
  pid_file	Specify the path to the pid file
  output	Specify the filter's output
  next_filter_socket_path	Specify the path to the next filter unix socket
  nb_thread	Integer specifying the number of thread used
  cache_size	Integer specifying the cache's size
  threshold	Integer specifying the minimum certitude at which the filter will output a log.If it's over 100, take the filter's default threshold

OPTIONS
  -d	Debug mode, does not create a daemon, set log level to debug
  -i	Set log level to info
  -n	Set log level to notice
  -w	Set log level to warning (DEFAULT)
  -e	Set log level to error
  -c	Set log level to critical

[IDEA] Rework Log

What is your feature about ?
Our log system is not pleasant to use and not very effective

Describe what you'd like
Think of a way to rework the log system, maybe with streams ?

[BUG] Filter DGA crash when Generator is unable to configure the filter

Describe the bug
When the DGA's generator return False, the filter crash and didn't remove the PID file. Seen in the fix-dga branch because the Generator Configure is fixed to return False instead of crashing when bad token map is given

After investigation : At line 34 in Core.cpp, the generator Configure return false and the end of the function is reached. But line 42 in main.cpp, when we normally remove the PID file, is not reached. Maybe the raise(SIGTERM) at line 36 in the Core.cpp is in cause ?

See if the bug is same with others filters

Plateform (please complete the following information):

  • OS (version): FreeBSD 12.0
  • Darwin version: 1.1

To Reproduce
Steps to reproduce the behavior:

  1. Checkout in the fix-dga branch
  2. Call the dga filter with a Token Map with wrong format (a file with just a blank line is fine)

Expected behavior
The PID file is not removed.

In the log you should have :

{"date":"Fri Oct 18 14:05:31 2019","level":"CRITICAL","filter":"dga_2","message":"DGA:: LoadTokenMap:: Error when load token map : Blank lie"}
{"date":"Fri Oct 18 14:05:31 2019","level":"CRITICAL","filter":"dga_2","message":"Core:: Run:: Unable to configure the filter"}
{"date":"Fri Oct 18 14:05:31 2019","level":"DEBUG","filter":"dga_2","message":"Server::Handle:: Closing acceptor"}
{"date":"Fri Oct 18 14:05:31 2019","level":"INFO","filter":"dga_2","message":"Monitor::HandleAccept:: Acceptor closed, closing monitor..."}

But no log about removing the PID.

[IDEA] Add asynchrone mode tests

What is your feature about ?
Test if the filters react the way we want then they are called in an asynchronous

Describe what you'd like

  • Test LOGS output (maybe all output ?)
  • Test evt_id

[IDEA] Be able to store additional context/data in cache

What is your feature about ?
Be able to store additional context in cache.

Describe what you'd like
When filters store information (aka. certitude currently) to cache, they can't store additional context about the entry they just handled, being able to store additional metadata (in the form of a string/json/whatever) could be great to give more information in logs when entries have a high certitude level.

Additional context
Example : fyara_scan (first version currently on branch split_inspection) returns not only the certitude, but the matching yara rules (when applicable), tags, etc...
The current cache doesn't permit to store anything else than a number (?), so the matching entries will have context in log only once, and then only the certitude next times (even if the entry needs to be exactly the same, but that WILL happen).

darwin crashing when configuration doesn't have the right parameters

in Services.py, python crashes silently when a field is not present in the data parsed from the filter configuration file.
(load_conf [354-386]).
some checks should be made, or at least fields should be fetched, not assuming they are always present, and give a warning if they are required.

[BUG] Errors on socket write while connections/filters are closing

Describe the bug
Filters can get stuck when sending data to next filter, while inbound connection is closing or filter has received a SIGTERM.

Plateform (please complete the following information):

  • OS (version): Ubuntu and FreeBSD 12.0
  • Darwin version: 1.0.2 (+ threading modifications)

To Reproduce
Steps to reproduce the behavior:

  1. Launch a Content_inspection filter (either standalone or with darwin manager) with a next filter (either Logs or a simple unix socket with nc -Ulk) and type LOG, set several threads (around 5)
  2. Launch Rsyslog with impcap and mmdarwin, route mmdarwin output for content_inspection (see content inspection wiki), make sure to set the response_type to "darwin" or "both"
  3. Generate traffic (if necessary) to make all threads work (open/refresh several web pages att once, for example)
  4. send a SIGTERM to the filter (either with Ctrl-C when executed with -z, or with htop/top/kill/whatever)
  5. some of the threads should stop, but not all

Expected behavior
The filter should close cleanly.

Additional context
Seems the filter is hanging on the next filter synchronous write

[IDEA] Rework naming of filters

What is your feature about ?
Some filters have too general name (anomaly) or too heavy name (content_inspection), rethink it.

Describe what you'd like
More precise naming for filters, like for ftanomaly, tanomaly, content_inspection

Additional context
Idea for fanomaly : network_behavior ?

[IDEA] Multithread Test

What is your feature about ?
Do the test in parallel to improve testing time !

Describe what you'd like
One thread per filter test

[BUG] redis not flushed between tests

Describe the bug
"connection: new_connection_test" test of connection filter failing after successive calls to testing

Plateform (please complete the following information):

  • OS (version): [e.g. FreeBSD (12.0)]
  • Darwin version: [e.g. 1.0]

To Reproduce
Steps to reproduce the behavior:

  1. Make sure redis is running and listening on /var/sockets/redis/redis.sock
  2. Start the tests.py once
  3. Wait for connection tests to finish
  4. Stop the tests
  5. launch them a second time immediately
  6. Wait for first connection filter test
  7. Now test fails

Expected behavior
The test should be OK both times

Solution
The Redis cache is not cleared after/before the tests, so the previous entry (added by the filter on the first run) stays and pollutes the second run

Fix redis socket connection to master

Sometimes when you connect to the redis local socket, you can be on a slave.
We must verify if we are on it, else we switch to master.
We also need to care when the redis switch between slave and master and handle it

[BUG] FDGA: Crashes when `token_map_path` file has wrong format

Describe the bug
When the file contained in the "token_map_path" field of the config is not properly formatted the filter crashes without logging.

Platform (please complete the following information):

  • OS (version): FreeBSD (12.0)
  • Darwin version: 1.0

To Reproduce
Steps to reproduce the behavior:

  1. Generate a random file
  2. Set the field "token_map_path" to ths file
  3. Start fdga

Expected behavior
Log a CRITICAL error with the proper message and quit gracefully.

Additional context
Hint: Look around line 140-143 of fdga/Generator.cpp

[BUG] HeartBeat restarting filter when startup time is too long

Describe the bug
For some filters, when their starting time is long (more than a second), the socket will not be created right away and the HeartBeat will try to restart the filter.
The consequence is X concurrent filters with identical parameters running, and only one followed by the manager.

Plateform (please complete the following information):

  • OS (version): [e.g. FreeBSD (12.0)]
  • Darwin version: [e.g. 1.0]

To Reproduce
Steps to reproduce the behavior:

  1. Set a Darwin configuration with Hostlookup filter activated
  2. Assign a large (10+MB) db file to Hostlookup (database parameter), to slow down startup
  3. start Darwin

Expected behavior
the Darwin manager and filters should start, but additional Hostlookup filters should start every few seconds, with exactly the same parameters (use of top/htop can help).
If the manager's log level is below ERROR, WARNING logs should be appended in darwin_manager.log stating Heartbeat didn't find socket file associated to filter, and trying to restart it.

Additional context
Hostlookup filter's database should have a (very) large number of entries to slow down configuration to a couple of seconds

[IDEA] FLOGS: Automated Testing & Hardening

What is your feature about ?

  • Adding automated test for the flogs component
  • Valgrind test for Linux
  • Correcting any issue that occurs during testing

Additional context
Automated testing with valgrind would be nice (look for XML output).

[BUG] Tanomaly not responding/forwarding data to session/filters

Describe the bug
Tanomaly has no workflow to answer/forward entries to client/filters.

Plateform (please complete the following information):

  • e.g. FreeBSD (12.0)
  • Darwin version: [e.g. 1.0.1]
  • mmdarwin

To Reproduce
set darwin with Tanomaly filter
set rsyslog/impcap/mmdarwin to send data to Tanomaly, and request an answer (response="back"):

action(type="mmdarwin"
                key="darwin"
                socketpath="/var/sockets/darwin/tanomaly_1.sock"
                response="back"
                fields=["!impcap!net_src_ip", "!impcap!net_dst_ip", "!impcap!net_dst_port", "!impcap!IP_proto"])

Expected behaviour
mmdarwin should send data to Darwin, and receive a certitude of 0 (as Tanomaly cannot compute certitude for the specific entry).

Current behaviour
mmdarwin holds on the receiving part, and gets stuck as Tanomaly doesn't respond.

Screenshots/logs
image

Additional context
Seems a bit more complex than thought, as darwin-client-python and mmdarwin also needs to receive as much certitudes as entries sent (generally, bulk calls are not handled correctly in workflows for now, it seems)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.