triforkse / logstash-mesos Goto Github PK

Make sure that we don't looks any log entries when streaming log files form other containers.

Example:

• How do we handle rolling log files
• What happens if the Logstash Framework restarts and intermediate log entries have been produced.

The scheduler must tell the executors what Elastic Search endpoint to use.

Currently testing any changes takes a long time since the entire cluster has to run before you can see any results.

We are looking into running everything from java to save the cost of running docker compose.

While stream files from in the executor, if a stream fails. The executor should try to restart it.

Update the README to point to an installation guide.

The guide should container:

1. How to install the framework
2. How configuration files look (parsing/tags/etc)
3. How to set up an Elastic Search connection to store the logs.

The scheduler should monitor a folder for config files and propagate config files to executors.

It would be help to have a feature roadmap.

Currently we need to build images containing both logstash and mesos (i.e for the executor).

Instead we should be able to expose the docker host's mesos-libraries as a volume so that we avoid having to build the mesos files into the image.

We can do the same thing for the logstash files. I.e put them in a volume, map them into the container at runtime and then use a "vanilla" java8 image to run the executor's tasks.

Imporove the README.md to get new developers up and running quickly.

I can imagine a Logstash framework working a half dozen different ways.

A design doc would helpful.