shazchaudhry / docker-elastic Goto Github PK

Deploy Elastic stack in a Docker Swarm cluster. Ship application logs and metrics using beats & GELF plugin to Elasticsearch

Shell 100.00%

logging gelf logstash metricbeat kibana travis filebeat log-aggregation elasticsearch jenkins-container

docker-elastic's Introduction

User story

As a DevOps team member, I want to install Elastic Stack (v7.9.1 by default) so that all application and system logs are collected centrally for searching, visualizing, analyzing and reporting purpose

Assumptions

Infrastructre is setup in Docker swarm mode
All containerized custom applications are designed to start with GELF log driver in order to send logs to Elastic Stack
NOTE: for cases where filebeat is to be run in "Docker for AWS": you will need to turn off auditd module in the filebeat config. Otherwise, filebeat service will fail to run

Architecture

The architecture used is shown in the table below

High level design	In scope	Not in scope
	Only beats for log files and metrics are used. All logs and metrics are shipped to elasticsearch directly in this repo. 2x Elasticsearch, 1x apm-server and 1x Kibana are used.	Ingest nodes are not used
	All containerized custom applications are designed to start with GELF log driver in order to send logs to Elastic Stack	-

For the full list of free features that are included in the basic license, see: https://www.elastic.co/subscriptions

Prerequisite

One docker swarm mode cluster allocated to running Elastic Stack. This cluster must have at least two nodes; 1x master and 1x worker. On each Elasticsearch cluster node, maximum map count check should be set to as follows: (required to run Elasticsearch)
- sudo sysctl -w vm.max_map_count=262144
- sudo echo 'vm.max_map_count=262144' >> /etc/sysctl.conf (to persist reboots)
One docker swarm mode cluster allocated to running containerized custom applications. This cluster must have at least on node; 1x master

Get docker compose files

You will need these files to deploy Eleasticsearch, Logstash, Kibana, and Beats. So, first SSH in to the master node of the Docker Swarm cluster allocated to running Elastic Stack and clone this repo by following these commands:

alias git='docker run -it --rm --name git -u $(id -u ${USER}):$(id -g ${USER}) -v $PWD:/git -w /git alpine/git' (This alias is only required if git is not already installed on your machine. This alias will allow you to clone the repo using a git container)
git version
git clone https://github.com/shazChaudhry/docker-elastic.git
cd docker-elastic

Deploy Elastic Stack

SSH in to the master node of the Docker Swarm cluster allocated to running Elastic Stack. Deploy Elastic stack by running the following commands:
- export ELASTIC_VERSION=7.9.1
- export ELASTICSEARCH_USERNAME=elastic
- export ELASTICSEARCH_PASSWORD=changeme
- export INITIAL_MASTER_NODES=node1 (See Important discovery and cluster formation settings)
- export ELASTICSEARCH_HOST=node1 (node1 is default value if you are creating VirtualBox with the provided Vagrantfile. Otherwise, change this value to one of your VMs in the swarm cluster)
- docker network create --driver overlay --attachable elastic
- docker stack deploy --compose-file docker-compose.yml elastic
  - You will need to be a little patient and wait for about 5 mins to allow stack to be ready
  - Assuming you have only two VMs, this will deploy a reverse proxy, logstash, Kibana and 2x Elasticsearch instances in Master / data nodes configuration. Please note that Elasticsearch is configured to start as a global service which means elasticsearch data nodes will be scalled out automatically as soon as new VMs are added to the Swarm cluster. Here is an explaination on various Elasticsearch cluster nodes
Check status of the stack services by running the following commands:
- docker stack services elastic
- docker stack ps --no-trunc elastic (address any error reported at this point)
- curl -XGET -u ${ELASTICSEARCH_USERNAME}:${ELASTICSEARCH_PASSWORD} ${ELASTICSEARCH_HOST}':9200/_cat/health?v&pretty' (Inspect cluster health status which should be green. It should also show 2x nodes in total assuming you only have two VMs in the cluster)
If in case beats are also desired to be installed in this very docker swarm cluster, then use the instructions provided in the next section

Deploy Beats

SSH in to the master node of the Docker Swarm cluster allocated to running containerized custom applications and beats. Clone this repo and change directory as per the instructions above.

Execute the following commands to deploy filebeat and metricbeat:

export ELASTIC_VERSION=7.9.1
export ELASTICSEARCH_USERNAME=elastic
export ELASTICSEARCH_PASSWORD=changeme
export ELASTICSEARCH_HOST=node1 (node1 is default value if you are creating VirtualBox with the provided Vagrantfile. Otherwise, change this value to your Elasticsearch host)
export KIBANA_HOST=node1 (node1 is default value if you are creating VirtualBox with the provided Vagrantfile. Otherwise, change this value to your Kibana host)
docker network create --driver overlay --attachable elastic

Filebeat

docker stack deploy --compose-file filebeat-docker-compose.yml filebeat (Filebeat starts as a global service on all docker swarm nodes. It is only configured to pick up container logs for all services at '/var/lib/docker/containers/*/*.log' (container stdout and stderr logs) and forward them to Elasticsearch. These logs will then be available under filebeat index in Kibana. You will need to add additional configurations for other log locations. You may wish to read Docker Reference Architecture: Docker Logging Design and Best Practices)
Running the following command should print elasticsearch index and one of the rows should have filebeat-*
- curl -XGET -u ${ELASTICSEARCH_USERNAME}:${ELASTICSEARCH_PASSWORD} ${ELASTICSEARCH_HOST}':9200/_cat/indices?v&pretty'

Metricbeat

docker stack deploy --compose-file metricbeat-docker-compose.yml metricbeat (Metricbeat starts as a global service on all docker swarm nodes. It sends system and docker stats from each node to Elasticsearch. These stats will then be available under metricbeat index in Kibana)
Running the following command should print elasticsearch index and one of the rows should have metricbeat-*
- curl -XGET -u ${ELASTICSEARCH_USERNAME}:${ELASTICSEARCH_PASSWORD} ${ELASTICSEARCH_HOST}':9200/_cat/indices?v&pretty'

Testing

Wait until all stacks above are started and are up and running and then run jenkins container where filebeat is running:

docker container run -d --rm --name jenkins -p 8080:8080 jenkinsci/blueocean
Login at http://[KIBANA_HOST] which should show Management tab
- username = elastic
- password = changeme
On the Kibana Management tab, configure an index pattern (if not already done automatically)
- Index name or pattern = filebeat-*
- Time-field name = @timestamp
Click on Kibana Discover tab to view containers' console logs (including Jenkins) under filebeat-* index. Here is a screenshot showing Jenkins container logs:

Sending messages to Logstash over gelf

Logstash pipeline is configured to accept messages with gelf log driver. Gelf is one of the plugin mentioned in Docker Reference Architecture: Docker Logging Design and Best Practices. Start an application which sends messages with gelf. An example could be as follows:

Stop the Jenkins container started earlier:
- docker container stop jenkins
Start Jenkins container again but with gelf log driver this time:
- export LOGSTASH_HOST=node1
- docker container run -d --rm --name jenkins -p 8080:8080 --log-driver=gelf --log-opt gelf-address=udp://${LOGSTASH_HOST}:12201 jenkinsci/blueocean
- Note that --log-driver=gelf --log-opt gelf-address=udp://${LOGSTASH_HOST}:12201 sends container console logs to Elastic stack
On the Kibana Management tab, configure an index pattern
- Index name or pattern = logstash-*
- Time-field name = @timestamp
Click on Discover tab and select logstash-* index in order to see logs sent to Elasticsearch via Logstash. Here is a screenshot showing Jenkins container logs:

Here is another example:

docker container run --rm -it --log-driver=gelf --log-opt gelf-address=udp://${LOGSTASH_HOST}:12201 alpine ping 8.8.8.8
Login to Kibana and you should see traffic coming into Elasticsearch under logstash-* index
You can use syslog as well as TLS if you wish to add in your own certs

Testing with APM Java Agent

Follow these instructions to build a java app that we will use for APM:

WIP

References

elastic/examples - Home for Elasticsearch examples available to everyone. It's a great way to get started
Elasticsearch sizing and capacity planning
Unifying APM, Logs, and Metrics for Deeper Operational Visibility
Machine Learning in the Elastic Stack - YouTube
Security and Threat Detection with the Elastic Stack

docker-elastic's People

Contributors

Stargazers

Watchers

Forkers

hanhongyuan mubuja vannazux jockdarock ycyr mittyok kentk87 kmcdeogracias emailmukeshn breitsmiley phungxuananh jymun adriancarayol afandylamusu sangkyunyoon z4c munisodomgula capsulated jcaromiq gipxy pallav8990 tiago-rocha-gomes kosmeen datphan310 123guanxing mcezzare ruanbekker sky3hao lucenarenato woxinqidai databill86 mbahoshy vsoltys meloning robinchap getfloret redhorse158 jesuittreason leshibily rschweda crenom yshulin sherwher jakesylvestre jason1242000 k8sdevopscloud annalitvin grillos karimfadl thkong octavioranieri alexeykostenko nfhook k8scat hack-fang selllami tofbabs kaznoda milosm claranobre moisesbas amitkumarsinghdev gemmacert-jobs feirenk sce anaselhajjaji starlessboi fagnersilva tchiring08 lennyridel itmseanlin uamartins abhijd9602 nielsolie baiyongzhen vamshisiddarth batzgroup 1-1is0 noyb34 alp1396 kefirzhang andremoah naristo bhargrah snirgu kykini estebang13 sezarmab ssteck hranitely2k lewismc look4regev alcefilho cmc-kumar mijara juanfcosanz ioanniskoumarelas kshailen d2life2005 igarciar

docker-elastic's Issues

Upgrade Elastic Stack to v6.4

How to add elastic cluster monitoring the right way?

I have tried with the below module configuration.

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

processors:
- add_cloud_metadata:

output.elasticsearch:
  hosts: ['elasticsearch:9200'] # Moonitoring cluster
  protocol: "http"

setup.kibana:
  host: "http://kibana:5601/monitoringkibana" #Monitoring KIbana
  protocol: "http"
  ssl.enabled: false
  
xpack.monitoring.enabled: true
setup.dashboards.enabled: true

metricbeat.modules:
- module: elasticsearch
  metricsets:
    - node
    - node_stats
    - index
    - index_summary
    - shard
  period: 20s
  hosts: ["http://192.168.0.1:9201"] #Production Elasticseaarch which I want to create monitoring for

I am getting only OS analytics

Permission denied when writing to /etc/sysctl.conf

Hi. Thanks for a really useful repository.

I saw you wrote this in README.md:
sudo echo 'vm.max_map_count=262144' >> /etc/sysctl.conf (to persist reboots)
However, as far as I research,

You can't use sudo to affect output redirection; > and >> (and, for completeness, <) are effected with the privilege of the calling user, because redirection is done by the calling shell, not the called subprocess.

You can see more details in this question.
So I think the command should be:
sudo sh -c 'echo "vm.max_map_count=262144" >> /etc/sysctl.conf'

Thank you.

Elastisearch cluster fails when change data path

Hello,

When I launch stack with this volumen path works fine

    volumes:
      - elasticsearch:/usr/share/elasticsearch/data

If I change the volume path to another location.

    volumes:
      - ./elasticsearch:/usr/share/elasticsearch/data

Shows me this error and never creates the cluster.

{"type": "server", "timestamp": "2020-05-28T07:35:05,570Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "DevOps", "node.name": "ig-dockerdev-01", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node1] to bootstrap a cluster: have discovered [{ig-dockerdev-01}{yCtLfa_IS_OOL6gvDnMnsA}{f8BlMJRGQ7GJ7ChMnUZjMA}{10.0.15.145}{10.0.15.145:9300}{dimrt}{xpack.installed=true, transform.node=true}, {ig-dockerdev-03}{KoHiSiuITV2YmVGkEXxtIQ}{b95IUP-jRBuhyR-mvyetAQ}{10.0.15.143}{10.0.15.143:9300}{dimrt}{xpack.installed=true, transform.node=true}, {ig-dockerdev-02}{OVrgsX9NRp6OIh2Jo5FcQQ}{Ysl3vSx4SBWwT_Mq3U4Lkw}{10.0.15.144}{10.0.15.144:9300}{dimrt}{xpack.installed=true, transform.node=true}]; discovery will continue using [10.0.15.144:9300, 10.0.15.143:9300] from hosts providers and [{ig-dockerdev-01}{yCtLfa_IS_OOL6gvDnMnsA}{f8BlMJRGQ7GJ7ChMnUZjMA}{10.0.15.145}{10.0.15.145:9300}{dimrt}{xpack.installed=true, transform.node=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }

Need to filter the message section of logs

@shazChaudhry Hope you are doing good.

I am working on configuring ELK with filebeat on docker. I have all the logs available on Kinbana dashboard, which are coming from filebeat docker.

Now my query is how to filter the message content of a log file which is coming from another server.

I have included the following lines in logstash.conf file which is in ELK stack server.

filter {

#/var/log/xxx/error.log
if ([log][file][path] =~ "/logs/error.log") {
grok {
match => { "message" => "%{DATE:date} %{TIME:time} | %{LOGLEVEL:loglevel} | %{IP:client_ip} [%{NUMBER:bytes}] %{WORD:method} /%{NOTSPACE:request_page} HTTP/%{NUMBER:http_version} | %{GREEDYDATA:logmessage}" }
}
}
}

However, it is not working.

here is my filebeat-docker.yml

filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false

processors:

add_cloud_metadata: ~

filebeat.inputs:

type: log
enabled: true
paths:
- "/var/log/apache2/*.log"
  exclude_files: ['.gz$']
  json.message_key: log
  include_lines: ['^ERR', '^WARN']

output.elasticsearch:
hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'

Can you please suggest on this?

Sabil.

Upgrade Elastic Stack to v7.9.1

At the time of creating this issue, the version is 7.9.1

https://www.elastic.co/guide/index.html

Filebeat on “Docker for AWS” fails to start

Issue has been logged on filebeat discussion board - https://discuss.elastic.co/t/filebeat-on-docker-for-aws-fails-to-start/110345

solutions mentioned in docker-archive/for-aws#123 has not worked in my case and filebeat is unable to locate /var/lib/docker/containers/ path. The error I get is:

"invalid mount config for type "bind": bind source path does not exist"

Add Application Performance Monitoring (APM) to the stack

Docs:

Webinars:

https://www.elastic.co/webinars/agumenting-logs-and-metrics-with-apm

not working

Hi,

I am trying to use this with 7.17.9, and it is not working any more.

It seems like the master were not able to discover. Can you please help with this as it helps a lot?

ELB does not appear to support UDP protocole

The solution in this repository expect docker containers to send UDP messages to Logstash. However, it appears that UDP is not supported on AWS Elastic Load Balancer. Extremely annoying.....

Currently, I am using GELF log driver with docker services to send UDP messages to Logstash. Investigate what other log driver options are available

Add an input buffer like redis, rabbitmq, or kafka

Currently, messages are sent directly to Elasticsearch.

Deploying Beats - Why are indices not created?

Hey, i'm trying out your code, but don't see the "filebeat-" or "metricbeat-" or any other beat when I curl for indices.. (this line: Running the following command should print elasticsearch index and one of the rows should have filebeat-*). Any Ideas why this could happen?

elastic search module fails to start.

I have started experimenting with this ,and i am facing something which seems to be a trivial issue ,just hard to solve for me.

"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: IndexFormatTooNewException[Format version is not supported (resource BufferedChecksumIndexInput(SimpleFSIndexInput(path=\"/usr/share/elasticsearch/data/nodes/0/_state/segments_2\"))): 10 (needs to be between 7 and 9)];",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.7.0.jar:7.7.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.7.0.jar:7.7.0]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.7.0.jar:7.7.0]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.7.0.jar:7.7.0]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.7.0.jar:7.7.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.7.0.jar:7.7.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.7.0.jar:7.7.0]",
"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
"at org.elasticsearch.node.Node.<init>(Node.java:638) ~[elasticsearch-7.7.0.jar:7.7.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:264) ~[elasticsearch-7.7.0.jar:7.7.0]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.7.0.jar:7.7.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.7.0.jar:7.7.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.7.0.jar:7.7.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.7.0.jar:7.7.0]",
"... 6 more",

Only the elastic search docker fails to start. Everything else starting up and waiting for elasticsearch to start.
Does anyone has any idea what am I missing?

Auditbeat in swarm mode

https://www.elastic.co/guide/en/beats/auditbeat/current/running-on-docker.html#_special_requirements

PID and CAP_ADD options are ignored as they are not yet available in swarm mode. Might be available in mid 2019

Unsupported config option for services.filebeat: 'configs'

How to start elasticsearch 7.0 in global mode in docker swarm?

I have posted a question in these forums:

could you help me out on a couple of things please

Hello,
I'm trying to deploy your cluster in the hope that I can learn more about this ELK.
I have some questions:
1- can you please explain this:
alias git='docker run -it --rm --name git -v $PWD:/git -w /git indiehosters/git git'
2- I am unable to create this ( I did not install beats.):
On the Kibana Management tab, configure an index pattern, Index name or pattern = filebeat-*
3- Edit "filebeat-docker-compose.yml" file. Change environment variables for Kibana and Elasticseaerch
hosts
- Does this need the actual IP address?

Thanks

Adding SSL to kibana for alerts

I'm trying to add alerts but first I need to activate transport layer security. I have generated my self signed certificated using this bash


#!/bin/bash

# Generate Root Key rootCA.key with 2048
openssl genrsa -passout pass:"$1" -des3 -out rootCA.key 2048

# Generate Root PEM (rootCA.pem) with 1024 days validity.
openssl req -passin pass:"$1" -subj "/C=US/ST=Random/L=Random/O=Global Security/OU=IT Department/CN=Local Certificate"  -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.pem

# Add root cert as trusted cert
if [[ "$OSTYPE" == "linux-gnu"* ]]; then
        # Linux
        yum -y install ca-certificates
        update-ca-trust force-enable
        cp rootCA.pem /etc/pki/ca-trust/source/anchors/
        update-ca-trust
        #meeting ES requirement
        sysctl -w vm.max_map_count=262144
elif [[ "$OSTYPE" == "darwin"* ]]; then
        # Mac OSX
        security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain rootCA.pem
else
        # Unknown.
        echo "Couldn't find desired Operating System. Exiting Now ......"
        exit 1
fi

# Generate Kib01 Cert
openssl req -subj "/C=US/ST=Random/L=Random/O=Global Security/OU=IT Department/CN=localhost"  -new -sha256 -nodes -out kib01.csr -newkey rsa:2048 -keyout kib01.key
openssl x509 -req -passin pass:"$1" -in kib01.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out kib01.crt -days 500 -sha256 -extfile  <(printf "subjectAltName=DNS:localhost,DNS:kib01")

I have added the following SSL variables in Kibana Service:

  - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=Vss86whNwQrKjA3D8aKTCRN6SnZLX4rv
  - SERVER_SSL_ENABLED=false
  - SERVER_SSL_KEY=config/certs/kib01.key
  - SERVER_SSL_CERTIFICATE=config/certs/kib01.crt
  - SERVER_SSL_KEYPASSPHRASE=testest123
  - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/rootCA.pem

docker.compose.yml

version: "3.8"

# 10 Things to Consider When Planning Your Elasticsearch Project: https://ecmarchitect.com/archives/2015/07/27/4031
# Using Apache JMeter to Test Elasticsearch: https://ecmarchitect.com/archives/2014/09/02/3915

services:

  swarm-listener:
    image: dockerflow/docker-flow-swarm-listener:latest
    hostname: swarm-listener
    networks:
      - elastic
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
    environment:
      - DF_NOTIFY_CREATE_SERVICE_URL=http://proxy:8080/v1/docker-flow-proxy/reconfigure
      - DF_NOTIFY_REMOVE_SERVICE_URL=http://proxy:8080/v1/docker-flow-proxy/remove
    deploy:
      placement:
        constraints: [node.role == manager]

  proxy:
    image: dockerflow/docker-flow-proxy:latest
    hostname: proxy
    ports:
      - "80:80"
      - "443:443"
      - "9200:9200"
      - "8200:8200"
    networks:
      - elastic
    environment:
      - LISTENER_ADDRESS=swarm-listener
      - MODE=swarm
      - BIND_PORTS=9200,8200
    deploy:
      replicas: 2

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION:-7.7.0}
    environment:
      # https://github.com/docker/swarmkit/issues/1951
      - node.name={{.Node.Hostname}}
      - discovery.seed_hosts=elasticsearch
      - cluster.initial_master_nodes=${INITIAL_MASTER_NODES:-node1}
      - cluster.name=DevOps
      - ELASTIC_PASSWORD=${ELASTICSEARCH_PASSWORD:-changeme}
      - xpack.security.enabled=true
      - xpack.monitoring.collection.enabled=true
      - xpack.security.audit.enabled=true
      - xpack.license.self_generated.type=trial
      - network.host=0.0.0.0
    networks:
      - elastic
    volumes:
      - elasticsearch:/usr/share/elasticsearch/data
    deploy:
      mode: 'global'
      endpoint_mode: dnsrr
      labels:
        - com.df.notify=true
        - com.df.distribute=true
        - com.df.servicePath=/
        - com.df.port=9200
        - com.df.srcPort=9200

  logstash:
    image: docker.elastic.co/logstash/logstash:${ELASTIC_VERSION:-7.7.0}
    hostname: "{{.Node.Hostname}}-logstash"
    environment:
      - XPACK_MONITORING_ELASTICSEARCH_URL=http://elasticsearch:9200
      - XPACK_MONITORING_ELASTICSEARCH_USERNAME=${ELASTICSEARCH_USERNAME:-elastic}
      - XPACK_MONITORING_ELASTICSEARCH_PASSWORD=${ELASTICSEARCH_PASSWORD:-changeme}
    ports:
      - "12201:12201/udp"
    networks:
      - elastic
    configs:
      - source: ls_config
        target: /usr/share/logstash/pipeline/logstash.conf

  kibana:
    image: docker.elastic.co/kibana/kibana:${ELASTIC_VERSION:-7.7.0}
    hostname: "{{.Node.Hostname}}-kibana"
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=${ELASTICSEARCH_USERNAME:-elastic}
      - ELASTICSEARCH_PASSWORD=${ELASTICSEARCH_PASSWORD:-changeme}
      - SERVER_NAME="{{.Node.Hostname}}-kibana"
      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=Vss86whNwQrKjA3D8aKTCRN6SnZLX4rv
      - SERVER_SSL_ENABLED=false
      - SERVER_SSL_KEY=config/certs/kib01.key
      - SERVER_SSL_CERTIFICATE=config/certs/kib01.crt
      - SERVER_SSL_KEYPASSPHRASE=testest123
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/rootCA.pem
    configs:
      - source: key_config
        target: /usr/share/kibana/config/certs/kib01.key
      - source: crt_config
        target: /usr/share/kibana/config/certs/kib01.crt
      - source: root_config
        target: /usr/share/kibana/config/certs/rootCA.pem
    networks:
      - elastic
    volumes:
      - kibana:/usr/share/kibana/data
    deploy:
      labels:
        - com.df.notify=true
        - com.df.distribute=true
        - com.df.servicePath=/
        - com.df.port=5601
        - com.df.srcPort=80

  apm-server:
    image: docker.elastic.co/apm/apm-server:${ELASTIC_VERSION:-7.7.0}
    hostname: "{{.Node.Hostname}}-apm-server"
    networks:
      - elastic
    command: >
        --strict.perms=false -e
        -E apm-server.rum.enabled=true
        -E setup.kibana.host=kibana:5601
        -E setup.kibana.username=${ELASTICSEARCH_USERNAME}
        -E setup.kibana.password=${ELASTICSEARCH_PASSWORD}
        -E setup.template.settings.index.number_of_replicas=0
        -E apm-server.kibana.enabled=true
        -E apm-server.kibana.host=kibana:5601
        -E apm-server.kibana.username=${ELASTICSEARCH_USERNAME}
        -E apm-server.kibana.password=${ELASTICSEARCH_PASSWORD}
        -E output.elasticsearch.hosts=["elasticsearch:9200"]
        -E output.elasticsearch.username=${ELASTICSEARCH_USERNAME}
        -E output.elasticsearch.password=${ELASTICSEARCH_PASSWORD}
        -E xpack.monitoring.enabled=true
    deploy:
      labels:
        - com.df.notify=true
        - com.df.distribute=true
        - com.df.servicePath=/
        - com.df.port=8200
        - com.df.srcPort=8200

networks:
    elastic:
      external: true

volumes:
  elasticsearch:
  kibana:

configs:
  ls_config:
    file: $PWD/elk/logstash/config/pipeline/logstash.conf
  ||key_config:
    file: $PWD/keyskeys/kib01.key
  crt_config:
    file: $PWD/keyskeys/kib01.crt
  root_config:
    file: $PWD/keyskeys/rootCA.pem

0 results until now.

Can you please advise as to how I configure kibana to work with SSL?

By the way thx @shazChaudhry for the repo. It is very useful to me :)

Thanks in advance.

Use a better logstash pipeline filter

elk/logstash/config/pipeline/logstash.conf

# The following filter is a hack! The "de_dot" filter would be better.
filter {
  ruby {
    code => "
      event.to_hash.keys.each { |k| event[ k.gsub('.','_') ] = event.remove(k) if k.include?'.' }
    "
  }
}

Using https connexions don't work

Hello,

In the process of upgrading the elk stack from 6.X to 7.X i tried to follow your config, using dnsrr (to avoid issue with the ingress address)
The cluster seems up, (i can curl from the elastic container)
But other service can't access it (like kibana)

I use authentication for the connection, passing by the https. Did you try with it?

Regards

Is it possible to start Packetbeat in Docker swarm mode?

Network capabilities required to start packetbeat in docker are ignored by docker swarm mode.

Have reached out to the community for help: https://discuss.elastic.co/t/run-packetbeat-in-docker-swarm-mode/129937

Create APM client

APM agent is needed to send metrics to APM server that was stood up in issue #34

Deployment not working Host not found

Hi,

I hope you can help me with the issue to get the version up and running. I used docker 18.06.3-ce and followed your instruction to start up the environment. I changed the node1 entry in the environment variables to the host name of my swarm master.

But I get the following error:
{"type": "server", "timestamp": "2019-11-18T09:25:15,179+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "DevOps", "node.name": "vala-n6", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [vala-n1] to bootstrap a cluster: have discovered []; discovery will continue using [] from hosts providers and [{vala-n6}{40UudO_vSM-tjXJt69uBMQ}{fjbzNvJaTDSmDmLKbufmEA}{10.0.4.15}{10.0.4.15:9300}{ml.machine_memory=16691216384, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2019-11-18T09:25:15,186+0000", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "DevOps", "node.name": "vala-n6", "message": "failed to resolve host [elasticsearch]" ,

Also the Proxy produces the following error:
2019/11/18 09:37:54 Error: Fetching config from swarm listener failed: Get http://swarm-listener:8080/v1/docker-flow-swarm-listener/get-services: dial tcp: lookup swarm-listener on 127.0.0.11:53: no such host. Will retry in 5 seconds.

Did you have an idea what the issue could be and how I can solve this. I have 6 nodes. Named vala-n1 to vala-n6.

BR and Thanks for your help.

Use docker secret management to hide credentials

Currently, username and passwords are in plain text. These credentials will need to be locked using docker secrets

Configure filebeat to fetch from local filesystem

I am running a non-containerized application locally which produces timeseries logs in a text file (in addition to executing other application functionality). I wanted to containerize all of the ELK stack for analyzing the timeseries logs so this project is just what I needed. I then wanted to use filebeat to feed the local log4j text log to logstash and into Elasticsearch.

Having deployed everything as follows

$ docker stack services elastic
ID                  NAME                     MODE                REPLICAS            IMAGE                                                 PORTS
hwv9mjnnbady        elastic_kibana           replicated          1/1                 docker.elastic.co/kibana/kibana:7.9.1
k2yjhizxkcdd        elastic_logstash         replicated          1/1                 docker.elastic.co/logstash/logstash:7.9.1             *:12201->12201/udp
m3xqajcqmopx        elastic_swarm-listener   replicated          1/1                 dockerflow/docker-flow-swarm-listener:latest
vuvk78117tmj        elastic_elasticsearch    global              1/1                 docker.elastic.co/elasticsearch/elasticsearch:7.9.1
y76njf0lta3q        elastic_apm-server       replicated          1/1                 docker.elastic.co/apm/apm-server:7.9.1
yl3f8ha2frzt        elastic_proxy            replicated          2/2                 dockerflow/docker-flow-proxy:latest                   *:80->80/tcp, *:443->443/tcp, *:8200->8200/tcp, *:9200->9200/tcp

and then filebeat

docker stack services filebeat
ID                  NAME                MODE                REPLICAS            IMAGE                                    PORTS
51di10aflys8        filebeat_filebeat   global              1/1                 docker.elastic.co/beats/filebeat:7.9.1

The problem I am having is configuring the application log directory for filebeat to fetch. For example, my logs are written to /Users/$name/some/directory/application.log.

Can you please advise as to how I configure filebeat-docker-compose.yml to pick up the local logs?