Giter Club home page Giter Club logo

elasticsearch-fluentd-kibana's Introduction

elasticsearch-fluentd-kibana

Elasticsearch, FluentD and Kibana

Description

Get the sources:

$ git clone https://github.com/bekkerstacks/elasticsearch-fluentd-kibana
$ cd elasticsearch-fluentd-kibana

Push logs from your containers to elasticsearch with fluentd.

Usage

Create a overlay network:

$ docker network create --driver overlay private

Deploy the elasticsearch-fluentd-kibana stack:

$ docker stack deploy -c docker-compose.yml efk
Creating config efk_fluent-elasticsearch-conf.v1
Creating service efk_fluentd-elasticsearch
Creating service efk_kibana
Creating service efk_elasticsearch

Deploy the sample application that logs info to stdout every 5 seconds:

$ docker stack deploy -c docker-compose-logger.yml efk
Creating service efk_demo-logger

View the elasticsearch nodes:

$ curl http://localhost:9200/_cat/nodes?v
ip       heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.0.0.7           34          95   7    0.42    0.43     0.26 mdi       *      es-node.1.docker-desktop

View the elasticsearch indices:

$ curl http://localhost:9200/_cat/indices?v
health status index              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana_1          Kxq95NMkR-6m9AMWWfzEvg   1   0          0            0       230b           230b
yellow open   fluentd-2019.10.09 fnxKL5I2Ty6vneP1bLWnhA   5   1          3            0       21kb           21kb

View the documents from the fluentd index:

$ > curl http://localhost:9200/fluentd-2019.10.09/_search?pretty
{
  "took" : 108,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 6,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "fluentd-2019.10.09",
        "_type" : "access_log",
        "_id" : "CToQsW0BdVrjXZ_xF5Hd",
        "_score" : 1.0,
        "_source" : {
          "source" : "stdout",
          "log" : "The date is: 1570633394",
          "container_id" : "de7883816f60d768f0c743c39eeb351c67a8eeefde867f3d013b628a1b1cd575",
          "container_name" : "/efk_demo-logger.1.tt85yi5sv2yosj9w1n5jaaqoj",
          "hostname" : "0ce8d311101c",
          "tag" : "docker.efk.demo-logger",
          "stack_name" : "efk",
          "service_name" : "demo-logger",
          "fluentd_hostname" : "docker-desktop",
          "@timestamp" : "2019-10-09T15:05:19.000000000+00:00",
          "@log_name" : "docker.efk.demo-logger"
        }
      },
...
}

Head over to kibana on http://localhost:5601/, select "Management" and "Index Patterns", provide the pattern "fluentd-" as shown below:

image

Then select "@timestamp" as the "Time Filter field name" and click "Create Index Pattern"

Click on "Discover" and you will see the log events as shown below:

image

elasticsearch-fluentd-kibana's People

Contributors

bekkerstacks avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.