threathunterx / nebula Goto Github PK

View Code? Open in Web Editor NEW

1.1K 53.0 387.0 16.9 MB

"星云"业务风控系统，主工程

Home Page: https://threathunter.cn/nebula.html

License: Apache License 2.0

Dockerfile 0.49% Shell 1.09% Lua 75.25% HTML 1.16% Perl 18.14% C 3.53% C++ 0.02% Roff 0.31%

nebula's Introduction

星云（TH-Nebula)业务风控系统介绍：

简介：

星云风控系统是一套互联网风控分析和检测平台，可以对企业遇到的各种业务风险场景进行细致的分析，找出威胁流量，帮助用户减少损失。星云采用旁路流量的方式进行数据采集，无需在业务逻辑上做数据埋点或侵入，同时支持本地私有化部署和Docker镜像云端部署。另外考虑到部分使用者风控经验不足，星云会提供基础的风控策略模板（基础内置五大风险场景：访客风险、帐号风险、支付风险、订单风险和营销风险），使用者可以结合业务实际情况，灵活的进行配置和调整。考虑到攻防对抗的时效性，策略调整之后实时生效，无需重新编译和上线。

产品特点：

1.轻量级部署

星云采用完全旁路流量解析的方式来采集业务信息，企业只需要与运维配合即可完成部署。值得一提的是，即使在业务增加、变化的情况下企业都可快速地获取到网络访问、登陆、注册、下单、参与活动等业务行为。

2.内置风险识别规则，简单易用

在“星云”上内置了大量业务场景下的攻防规则，并采用可视化规则编辑的方式，企业可以快速编辑策略并进行实际环境下的测试。

3.无埋点，无敏感数据泄漏风险

星云不需要企业研发埋点即可实现访问、登陆、注册、信息修改等的数据实时采集，无敏感数据外泄风险，更好的保护企业数据隐私。

解决问题：

风控系统的本质是为了能够让企业有能力主动发现业务风险，我们希望星云的开源能让企业能够快速的度过早期的基础建设阶段，进入到攻防效率提升阶。基于星云风控系统，企业可以针对不同的业务场景进行攻防对抗。

快速接入

使用手册

设计理念

二次开发

更新说明

** 商业版代码在持续更新中，并有以下提升：

增加api、logstash、rabbitmq等10+种流量捕获接入方式
线性扩展能力，以保证任何规模的数据量都可以处理
无单点故障，高可靠，以保证运营商级的服务
其他bug修复以及改进见下表

授权说明

威胁猎人团队2016年成立，核心团队成员均来自于国内一线互联网企业安全部门。我们的团队有来自情报分析、数据分析、业务风控、逆向、反欺诈等领域的资深专家，拥有多年黑产研究对抗经验。当您需要使用Github上的nebula代码时，建议您购买商业授权，获取商业授权后可以收到我们提供的nebula商业版全部源代码，以及我们专业安全团队的协助。购买商业授权为您节省大量开发、测试和完善时间，让您有更多时间用于创新及盈利。

负责人：卡卡
微信号：imakaka

需求定制

威胁猎人提供基于nebula的需求咨询与功能开发定制，即使您不懂技术，也可以根据您的需求为您定制成品

商业版代码更新内容

2019-7-25

4、修复mac上部署sniffer，bro驱动起不来的bug
3、修复部分界面展示异常bug
2、修复正常流量统计不实时bug
1、修复偶尔断流bug

nebula's People

Contributors

Stargazers

Watchers

Forkers

pandazheng cheetahsec xssddq webvul lstarby alivnhuang pf5512 npc7 ymero drtjordan 5up3rc xixingzhe loveleon ppayaoxing xuacker bluesky1055 r00tak alphisilaw zminjiao netkey 0x1ng danny-dong songfang nocarpe chandler1564 lilololi dacer250 kevinkerry madfroghe qsdj getwindow njcx marrygit lianghappy yanjlee kyodule redyoung86 garinyang adomore shellb0y johnnyttzhu oo00o whocare2014 cloudwafs sdgdsffdsfff raystyle wolfju baronnd a631381602 chensiqia attackanddefencesecuritylab luyishisi yuanxzhang cdpidan vf3ng jangocheng c0010 keyman9848 zjhiphop wawava fuckgitb darius-1024 liliaobin superdong0 dior222 xbzbing waterbolik haoirui free-guangzhou slumdunking r00tgrok csutong shellsec wulinzhizun gitsucce-zz atcom1 huangqiuhao m31n99 menglike nevernamed lua-study chensword weichaojie hnmadsec xi4od bradbann stayhungrystayfoolish0 emmaaii ronglaoxie sew1ng qqlover huangyuan666 k2vin longlycsu scannerkun w2n1ck fooying shellfeel allanxia lcweik

nebula's Issues

http://112.74.58.210:9001/ 挂了

测试环境挂了，端口提示无法连接

mac下用给的脚本构建景象有问题

linux下二进制或者源码安装时nebula主进程一直是Restarting状态

导致status查看状态报下面的错误：

风险命中数在大屏显示计数始终为0~~

风险事件管理无数据解决办法

风险事件管理无数据解决办法。

cd nebula
#停止
docker-compose down
#拉取新镜像
docker-compose pull nebula
#启动
docker-compose up -d
./ctrl.sh start //启动

缺少 threathunter.common 2.6.7 的源码

<dependency>
            <groupId>com.threathunter.common</groupId>
            <artifactId>threathunter.common</artifactId>
            <version>2.6.7</version>
</dependency>

用户和用户组创建失败

不管如何选择，在创建用户组的时候都是报失败，没有用户组，用户也创建不了~~

邮件报警配置后收取不到报警邮件解决办法

邮件报警配置后收取不到报警邮件。SMTP无法送记录。

收件人邮箱输入后敲回车，变成这样

别问我为什么会有这样的设定~~~~

按照部署文档部署后运行不起来

如题

http://112.74.58.210:9001 访问不了

还有新的体验地址嘛

已有流量pcap包文件的情况下是否方便对接到星云

已有流量pcap包文件的情况下是否方便对接到星云，

请问下在docker安装的时候,如何使用已有的mysql redis,这两个服务都有现成的没必要通过docker去重新构建

部分数据库没有创建表结构，却在源码中存在连接。

比如：db_mp_freqlog.t_frequency_campaign
配置页出现过一次：https://github.com/threathunterX/nebula/blob/d25b3bcd9409921e8abee3d94f05b1086eddd17a/src/go/src/malicious_prevent/malicious.conf
mysqlview.py中出现了一次：
https://github.com/threathunterX/nebula/blob/a4f535fdbbba31f84e91bf5a858256fb37b2a513/src/mysqlview/freqview.py
但是始终不知道这个数据库中的内容，
能否有一份完整的部署文档呢？

安装起来以后打开网站只有一个系统设置按钮,没有其他按钮

http://112.74.58.210:9001 访问不了

aerospike组建无法启动，报错如下：

link eth0 state up
link eth0 state up in 0
Jan 06 2020 09:40:37 GMT: FAILED ASSERTION (config): (cfg.c:1736) line 6 :: 'transaction-queues' is obsolete - please configure 'service-threads' carefully
Jan 06 2020 09:40:37 GMT: WARNING (as): (signal.c:161) SIGINT received, shutting down Aerospike Community Edition build 4.8.0.2 os debian9
Jan 06 2020 09:40:37 GMT: WARNING (as): (signal.c:164) startup was not complete, exiting immediately

虚拟机跑docker

利用星云项目怎么统计历史日志文件

sniffer无法抓取流量

目录下没有python依赖

COPY ./nebula_apps/python_lib /home/threathunter/nebula/python_lib

威胁猎人官方在阿里云上部署了一套星云系统，但是我测试的时候发现一直无法抓取力量，无论怎么请求都没有用

威胁猎人官方在阿里云上部署了一套星云系统，但是我测试的时候发现一直无法抓取力量，无论怎么请求都没有用，有没有详细教程

缺失的文档还会持续补充吗？

如题

CPU高达300%

容器启动失败

nebula版本：1.1.2
安装方式：二进制安装
问题：
nebula:incident_babel_db_writer和nebula:notice_babel_db_writer两个容器无法启动，状态：

nebula:incident_babel_db_writer FATAL Exited too quickly (process log may have details)
nebula:notice_babel_db_writer FATAL Exited too quickly (process log may have details)

看了下log

_sh: warning: setlocale: LC_ALL: cannot change locale (zh_CN.UTF-8)
Traceback (most recent call last):
File "db_writer.py", line 3, in
from gevent import monkey;monkey.patch_all()
File "/usr/lib64/python2.7/site-packages/gevent/monkey.py", line 966, in patch_all
_notify_patch(events.GeventWillPatchAllEvent(modules_to_patch, kwargs), _warnings)
File "/usr/lib64/python2.7/site-packages/gevent/monkey.py", line 168, in _notify_patch
notify_and_call_entry_points(event)
File "/usr/lib64/python2.7/site-packages/gevent/events.py", line 112, in notify_and_call_entry_points
subscriber = plugin.load()
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2345, in load
self.require(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2368, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 792, in resolve
new_requirements = dist.requires(req.extras)[::-1]
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2635, in requires
dm = self._dep_map
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2899, in _dep_map
self.__dep_map = self._compute_dependencies()
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2908, in _compute_dependencies
for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2890, in _parsed_pkg_info
metadata = self.get_metadata(self.PKG_INFO)
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 1798, in get_metadata
raise KeyError("No metadata except PKG-INFO is available")
KeyError: 'No metadata except PKG-INFO is available'sh: warning: setlocale: LC_ALL: cannot change locale (zh_CN.UTF-8)
Traceback (most recent call last):
File "db_writer.py", line 3, in
from gevent import monkey;monkey.patch_all()
File "/usr/lib64/python2.7/site-packages/gevent/monkey.py", line 966, in patch_all
_notify_patch(events.GeventWillPatchAllEvent(modules_to_patch, kwargs), _warnings)
File "/usr/lib64/python2.7/site-packages/gevent/monkey.py", line 168, in _notify_patch
notify_and_call_entry_points(event)
File "/usr/lib64/python2.7/site-packages/gevent/events.py", line 112, in notify_and_call_entry_points
subscriber = plugin.load()
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2345, in load
self.require(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2368, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 792, in resolve
new_requirements = dist.requires(req.extras)[::-1]
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2635, in requires
dm = self._dep_map
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2899, in _dep_map
self.__dep_map = self._compute_dependencies()
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2908, in _compute_dependencies
for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2890, in _parsed_pkg_info
metadata = self.get_metadata(self.PKG_INFO)
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 1798, in get_metadata
raise KeyError("No metadata except PKG-INFO is available")
KeyError: 'No metadata except PKG-INFO is available'

请帮忙看下啥问题，谢谢

系统配置页面输入的配置信息无法保存

如上图，系统配置页面，输入的邮箱信息，还有流量过滤等，输入完成后，点击保存，提示保存成功，但是刷新页面，配置全都消失了，请问是什么原因

BUILD FAILURE

当我执行./build.sh -u -v 1.1.0 --apps 出现如下的报错，请帮忙看一下,谢谢

环境：CentOS release 6.6 2.6.32-504.el6.x86_64
mvn -version
Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4; 2014-08-12T04:58:10+08:00)
Maven home: /opt/apache-maven-3.2.3
Java version: 1.8.0_201, vendor: Oracle Corporation
Java home: /opt/jdk1.8.0_201/jre
Default locale: en_US, platform encoding: UTF-8
Python 2.7.3
node v11.8.0

[INFO] Building babel 1.9.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ babel ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ babel ---
[INFO] Installing /export/servers/nebula/src/java_lib/babel_java/pom.xml to /root/.m2/repository/com/threathunter/babel/babel/1.9.0/babel-1.9.0.pom
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building babel-model 1.9.0
[INFO] ------------------------------------------------------------------------
[WARNING] The POM for com.threathunter.common:threathunter.common:jar:2.6.7 is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] babel .............................................. SUCCESS [ 0.754 s]
[INFO] babel-model ........................................ FAILURE [ 0.309 s]
[INFO] babel-util ......................................... SKIPPED
[INFO] babel-rmq .......................................... SKIPPED
[INFO] babel-redis ........................................ SKIPPED
[INFO] babel-rpc .......................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.367 s
[INFO] Finished at: 2019-04-11T15:45:01+08:00
[INFO] Final Memory: 6M/141M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project babel-model: Could not resolve dependencies for project com.threathunter.babel:babel-model:jar:1.9.0: Failure to find com.threathunter.common:threathunter.common:jar:2.6.7 in http://maven.aliyun.com/nexus/content/groups/public/ was cached in the local repository, resolution will not be reattempted until the update interval of alimaven has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :babel-model
====== install threathunter_basictools_java v1.1.0 ======
./build.sh: line 40: cd: java_lib/threathunter_basictools_java/: No such file or directory
====== install greatdant ======
./build.sh: line 43: cd: greatdant: No such file or directory
====== install labrador ======
./build.sh: line 46: cd: labrador: No such file or directory
====== install apiserver ======
./build.sh: line 49: cd: apiserver: No such file or directory
====== offline ======
./build.sh: line 52: cd: offline: No such file or directory
====== install greyhound ======
./build.sh: line 55: cd: greyhound: No such file or directory
====== install online ======
./build.sh: line 58: cd: online: No such file or directory
./build.sh: line 60: cd: BChart: No such file or directory
====== package BChart ======
mv: cannot stat `.git': No such file or directory
npm WARN saveError ENOENT: no such file or directory, open '/export/servers/nebula/src/java_lib/babel_java/package.json'
npm WARN enoent ENOENT: no such file or directory, open '/export/servers/nebula/src/java_lib/babel_java/package.json'
npm WARN babel_java No description
npm WARN babel_java No repository field.
npm WARN babel_java No README data
npm WARN babel_java No license field.

up to date in 1.509s
found 0 vulnerabilities

npm ERR! path /export/servers/nebula/src/java_lib/babel_java/package.json
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall open
npm ERR! enoent ENOENT: no such file or directory, open '/export/servers/nebula/src/java_lib/babel_java/package.json'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2019-04-11T07_45_06_865Z-debug.log
mv: cannot stat .git.bak': No such file or directory ====== package nebula_fe ====== cp: cannot stat BChart/BChart-1.1.16.tgz': No such file or directory
mv: cannot stat `.git': No such file or directory
npm WARN saveError ENOENT: no such file or directory, open '/export/servers/nebula/src/java_lib/package.json'
npm WARN enoent ENOENT: no such file or directory, open '/export/servers/nebula/src/java_lib/package.json'
npm WARN java_lib No description
npm WARN java_lib No repository field.
npm WARN java_lib No README data
npm WARN java_lib No license field.

up to date in 1.198s
found 0 vulnerabilities

npm ERR! path /export/servers/nebula/src/java_lib/BChart-1.1.16.tgz
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall stat
npm ERR! enoent ENOENT: no such file or directory, stat '/export/servers/nebula/src/java_lib/BChart-1.1.16.tgz'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2019-04-11T07_45_11_719Z-debug.log
./build.sh: line 74: cd: scripts: No such file or directory
mv: cannot stat .git.bak': No such file or directory ====== module copying ====== copy java module cp: cannot stat offline/target/nebula_offline_slot-prod.tar.gz': No such file or directory
cp: cannot stat online/nebula-onlineserver/target/nebula-onlineserver.tar.gz': No such file or directory cp: cannot stat apiserver/web-manager/target/java-web-release.tar.gz': No such file or directory
cp: cannot stat labrador/application/target/labrador-release.tar.gz': No such file or directory copy python module cp: cannot stat python_lib': No such file or directory
cp: cannot stat nebula_db_writer': No such file or directory cp: cannot stat nebula_query_web': No such file or directory
cp: cannot stat nebula_web': No such file or directory copy javascript module cp: cannot stat nebula_fe/build/nebula_fe.tar.gz': No such file or directory
./build.sh: line 97: cd: ../nebula_apps/: No such file or directory
====== uncompress ======
uncompress java-web
tar (child): java-web-release.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
uncompress labrador
tar (child): labrador-release.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
uncompress nebula-onlineserver
tar (child): nebula-onlineserver.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
uncompress nebula_offline_slot
tar (child): nebula_offline_slot-prod.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
uncompress nebula_fe
tar (child): ./nebula_fe.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now

IP风险分析地理位置未更新

监控大屏无风险事件数据

测试中风险事件数据都有了，但是投到大屏就没有这些数据，一开始以为浏览器问题，换了各种浏览器都没用~~

sniffer端接收不到流量

环境:centos7 + docker 18.03.1-ce docker-compose:1.16.1
刚才发的issue说虚拟环境不支持重装服务器。之后部署发现sniffer不成功。nebula端可以接收到别的机器传输的信号。本机的sniffer失败

星云增加可配置redis密码的需求

由于后期搭配redis密码使用的话，能否在sniffer的docker-compose配置中加入redis密码的参数
如
配置文件docker-compose.yml（直接修改此文件即可）
environment:
- REDIS_HOST=127.0.0.1 # 远程redisIP
- REDIS_PORT=16379 # 远程redis端口
- REDIS_PASS=password
- NEBULA_HOST=127.0.0.1 # 远程nebula服务IP
- NEBULA_PORT=9001 # 远程nebulaIP
- DRIVER_INTERFACE=eth0 # 监听网卡
- DRIVER_PORT=80,9001 # 业务服务端口