Giter Club home page Giter Club logo

docker-elk-filebeat's Introduction

Docker Elasticsearch with Kibana and Filebeat

Run the latest version of the ELK (Elasticsearch, Filebeat, Kibana) stack with Docker and Docker Compose.

It will give you the ability to analyze any data set by using the searching/aggregation capabilities of Elasticsearch and the visualization power of Kibana.

Based on the official Docker images:

Note: Other branches in this project are available:

Requirements

Host setup

  1. Install Docker version 1.10.0+
  2. Install Docker Compose version 1.6.0+
  3. Clone this repository

SELinux

On distributions which have SELinux enabled out-of-the-box you will need to either re-context the files or set SELinux into Permissive mode in order for docker-elk to start properly. For example on Redhat and CentOS, the following will apply the proper context:

$ chcon -R system_u:object_r:admin_home_t:s0 docker-elk/

Usage

Bringing up the stack

Note: In case you switched branch or updated a base image - you may need to run docker-compose build first

Start the ELK stack using docker-compose:

$ docker-compose -f docker-compose-elk.yml up

You can also choose to run it in background (detached mode):

$ docker-compose -f docker-compose-elk.yml up -d

Give Kibana a few seconds to initialize, then access the Kibana web UI by hitting http://localhost:5601 with a web browser.

By default, the stack exposes the following ports:

  • 9200: Elasticsearch HTTP
  • 9300: Elasticsearch TCP transport
  • 5601: Kibana

Notice: Kibana will ask you for basic auth username/password saved in filebeat.yml

WARNING: If you're using boot2docker, you must access it via the boot2docker IP address instead of localhost.

WARNING: If you're using Docker Toolbox, you must access it via the docker-machine IP address instead of localhost.

Initial setup

Default Kibana index pattern creation

When Kibana launches for the first time, it is not configured with any index pattern.

Via the Kibana web UI

NOTE: You need to inject data into Logstash before being able to configure a Logstash index pattern via the Kibana web UI. Then all you have to do is hit the Create button.

Refer to Connect Kibana with Elasticsearch for detailed instructions about the index pattern configuration.

Configuration

NOTE: Configuration is not dynamically reloaded, you will need to restart the stack after any change in the configuration of a component.

How can I tune the filebeat configuration?

The Filebeat configuration is stored in filebeat/filebeat.yml.

How can I tune the Elasticsearch configuration?

The Elasticsearch configuration is stored in elasticsearch/config/elasticsearch.yml.

How can I scale out the Elasticsearch cluster?

Storage

How can I persist Elasticsearch data?

The data stored in Elasticsearch will be persisted after container reboot but not after container removal.

In order to persist Elasticsearch data even after removing the Elasticsearch container, you'll have to mount a volume on your Docker host. Update the elasticsearch service declaration to:

elasticsearch:

  volumes:
    - /path/to/storage:/usr/share/elasticsearch/data

This will store Elasticsearch data inside /path/to/storage.

docker-elk-filebeat's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

docker-elk-filebeat's Issues

Is this stack working on Docket Toolbox for Windows ?

Hi,

Since we need to use VirtualBox on another project on our computers, we're limited to the Docket Toolbox for Windows to run Docker.

After cloning your repo and running docker-compoes, we got this message :

Status: Downloaded newer image for docker.elastic.co/beats/filebeat:6.2.2
Creating docker-elk-filebeat_elasticsearch_1 ... error

ERROR: for docker-elk-filebeat_elasticsearch_1  Cannot start service elasticsearch: b'OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \\"rootfs_linux.go:58: mounting \\\\\\"/c/cygwin64/home/nam/projects/docker-elk/elasticsearch/config/elasticsearch.yml\\\\\\" to rootfs \\\\\\"/mnt/sda1/var/lib/docker/overlay2/d90a866bdb9168e2a67cc1f679bd427408c462d249b8fff1b7113dc39b1d824f/merged\\\\\\" at \\\\\\"/mnt/sda1/var/lib/docker/overlay2/d90a866bdb9168e2a67cc1f679bd427408c462d249b8fff1b7113dc39b1d824f/merged/usr/share/elasticsearch/config/elasticsearch.yml\\\\\\" caused \\\\\\"not a directory\\\\\\"\\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type'

ERROR: for elasticsearch  Cannot start service elasticsearch: b'OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \\"rootfs_linux.go:58: mounting \\\\\\"/c/cygwin64/home/nam/projects/docker-elk/elasticsearch/config/elasticsearch.yml\\\\\\" to rootfs \\\\\\"/mnt/sda1/var/lib/docker/overlay2/d90a866bdb9168e2a67cc1f679bd427408c462d249b8fff1b7113dc39b1d824f/merged\\\\\\" at \\\\\\"/mnt/sda1/var/lib/docker/overlay2/d90a866bdb9168e2a67cc1f679bd427408c462d249b8fff1b7113dc39b1d824f/merged/usr/share/elasticsearch/config/elasticsearch.yml\\\\\\" caused \\\\\\"not a directory\\\\\\"\\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type'
ERROR: Encountered errors while bringing up the project.

I'd just like to know if people got this to work on Windows before or if it's known to not be supported on our current scenario before putting too much time to figure out things out.

Thanks!

elasticsearch and filebeat failed to start

[root@HKAPPDLV010 elasticsearch]# docker logs -f docker-elk-filebeat_elasticsearch_1
2019-08-30 08:20:45,396 main ERROR No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
Exception in thread "main" SettingsException[Failed to load settings from /usr/share/elasticsearch/config/elasticsearch.yml]; nested: AccessDeniedException[/usr/share/elasticsearch/config/elasticsearch.yml];
at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:101)
at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:95)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85)
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/config/elasticsearch.yml
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
at java.nio.file.Files.newInputStream(Files.java:152)
at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1162)
at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:99)
... 6 more

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.