Giter Club home page Giter Club logo

elasticsearch-templates's Introduction

elasticsearch-templates

The repository contains scripts and sources to generate Elasticsearch templates that comply with Common Data Model.

Build Status

Problem Statement

We are trying to solve the problem of conflicts and inconsistencies in log data as collected by, and from, different subsystems stored together as a unified data set under one warehouse.

Namespace hierarchy

Namespace hierarchy on the log metadata is the key concept. We use the Elasticsearch index templates and document mapping to cast the common metadata keys into usable documents.

Namespace corresponds to a top-level JSON key of Elasticsearch document. Namespace is usually defined per individual app or subsystem, so that different applications/subsystems not conflict in various metadata fields.

Adding new namespace

Create a namespace definition file in namespaces/ folder.

Adding new Elasticsearch template

Create a sub-folder in templates/ folder, named as the desired template. (Alternatively copy/modify one of the existing template folders)

Add/modify template.yml definition file to include proper namespace definitions. See for example templates/openshift/README.md for the details.

Elasticsearch versions support

Support for multiple Elasticsearch versions has been added. Resulting files (ie. index-templates or index-patterns) are generated for each supported version of Elasticsearch. Target version of ES is encoded into the file name.

List of currently supported ES versions can be find in scripts/supported_versions.py.

The idea is that all the input file templates and data are formatted according to the latest supported ES version and scripts handle backward data and format conversions for older ES versions. As part of unit testing the generated data is compared to released common data files (automatically downloaded from GitHub during tests).

Generating documentation

Use the makefile in the templates/ folder.

Alternatively, run the following command: python ./scripts/generate_template.py (path to template in templates/) namespaces/ --doc.

The generated file looks like "xxx.asciidoc".

Viewing the documentation

Install the asciidoc viewer in web browser.

Open the local path to the asciidoc file "xxx.asciidoc" in your browser.

elasticsearch-templates's People

Contributors

larsks avatar lukas-vlcek avatar lzap avatar richm avatar spbnick avatar sradco avatar t0ffel avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.