Giter Club home page Giter Club logo

Comments (4)

mihirsoni avatar mihirsoni commented on August 16, 2024

Hi @yangcaixing

Trying to understand on your use case , You want to aggregate for a given time period with all IP addresses and sum of all bytes. or just want to filter on specific host or static IP ? If you just want to monitor a single client_host you can create monitor using below query and modify with adding filter for client_host. If you want to get notified on if any of the client_host breaches threshold then you can create a monitor using sample query and write a Trigger in painless script.

Here is a sample query that you could you use as part of the creating a monitor using extraction query.You should modify below sample query as per your need adjust filters and apply time rage as necessary. This query will give you list of client_host with their sum(bytes)

{
  "size": 0, 
  "aggs": {
    "client_host": {
      "terms": {
        "field": "flow.client_hostname"
      },
      "aggs": {
       "byes":  {
          "sum": {
             "field": "flow.bytes"
          }
       }
      }
    }
  }
}

from alerting.

yangcaixing avatar yangcaixing commented on August 16, 2024

Hi @mihirsoni,
Thank very much, because my flow index are very huge, I want calculate all the IP ,and the sum bytes of each IP. when any client_host traffic breaches threshold, such as 1GB, I want trigger an alert as below:
IP:172.20.132.196, bytes: 2GB
IP:172.20.110.24, bytes: 3GB.
I try your query example, now I can get the IP and it's bytes info, but how to I write the trigger condition query, to monitor each IP and it's traffic, if meet the condition, will trigger the action, would you pleas help to me, thanks in advance.
open1
2
3

from alerting.

yangcaixing avatar yangcaixing commented on August 16, 2024

Hi @mihirsoni
And bellow are the query results, I paste the whole field, I already sorted the buckets as bytes "desc", so could you pleas help me, how to write the trigger condition for this buckets, and how to extract the "key" and "value" to action notification, I'm new to opendistro, thanks very much.

    "_shards": {
        "total": 25,
        "failed": 0,
        "successful": 25,
        "skipped": 0
    },
    "hits": {
        "hits": [],
        "total": {
            "value": 10000,
            "relation": "gte"
        },
        "max_score": null
    },
    "took": 72,
    "timed_out": false,
    "aggregations": {
        "client_host": {
            "doc_count_error_upper_bound": -1,
            "sum_other_doc_count": 977161,
            "buckets": [
                {
                    "doc_count": 12544,
                    "bytes": {
                        "value": 1949235700
                    },
                    "key": "172.20.120.104"
                },
                {
                    "doc_count": 697,
                    "bytes": {
                        "value": 1856866682
                    },
                    "key": "172.20.150.82"
                },
                {
                    "doc_count": 1960,
                    "bytes": {
                        "value": 1683565579
                    },
                    "key": "172.20.120.151"
                },
                {
                    "doc_count": 865,
                    "bytes": {
                        "value": 1091118606
                    },
                    "key": "172.20.144.81"
                },
                {
                    "doc_count": 1469,
                    "bytes": {
                        "value": 1032784474
                    },
                    "key": "172.20.150.67"
                },
                {
                    "doc_count": 92,
                    "bytes": {
                        "value": 830436284
                    },
                    "key": "172.20.60.14"
                },
                {
                    "doc_count": 1126,
                    "bytes": {
                        "value": 696536181
                    },
                    "key": "172.20.170.67"
                },
                {
                    "doc_count": 1111,
                    "bytes": {
                        "value": 577392383
                    },
                    "key": "10.10.101.2"
                },
                {
                    "doc_count": 1379,
                    "bytes": {
                        "value": 522637517
                    },
                    "key": "172.20.80.27"
                },
                {
                    "doc_count": 580,
                    "bytes": {
                        "value": 459864599
                    },
                    "key": "172.20.80.82"
                }
            ]
        }
    }
}

from alerting.

mihirsoni avatar mihirsoni commented on August 16, 2024

Hi @yangcaixing

To solve your problem, you can modify query in a such way that would give you buckets only if they're breaching threshold using bucket_selector aggregation in Elasticsearch.

Sample Query

{
  "size": 0, 
  "aggs": {
    "client_host": {
      "terms": {
        "field": "flow.client_hostname"
      },
      "aggs": {
       "total_byes":  {
          "sum": {
             "field": "flow.bytes"
          }
       },
       "bytes_bucket_filter": {
          "bucket_selector": {
            "buckets_path": {
              "totalBytes":"total_bytes"
            },
            "script": "params.totalBytes > 300"
          }
        }
      }
    }
  }
}

Above query will result buckets of client_host only if they're breaching threshold mentioned inside script otherwise it will be empty. Now you can write simple trigger condition on length of buckets, if it is more than 0 that means there are client_hosts who is breaching threshold and then can be printed in your action message. For more details on how to please refer documentation.

P.S : Please ensure the performance of query and tune it accordingly.

I am closing this issue for now, feel free to re-open / create new issue if you find any bugs. I would suggest to use official forum for questions instead of using Github. This will help us to manage our Github strictly for issues and feature requests.

from alerting.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.