Support for X-Pack version 6 about elkx-docker HOT 24 CLOSED

Here's a quick and dirty way to get started.

Create the following docker-compose.yml file.

elkx:
  image: sebp/elkx
  ports:
    - "5601:5601"
    - "9200:9200"
    - "5044:5044"
  environment:
    - ELASTIC_BOOTSTRAP_PASSWORD="changeme"

Start a container using docker-compose.

$ docker-compose up
Creating elkxdocker_elkx_1
Attaching to elkxdocker_elkx_1
elkx_1  | ERROR: Setting [bootstrap.pass] does not exist in the keystore.
elkx_1  |  * Starting periodic command scheduler cron
elkx_1  |    ...done.
elkx_1  |  * Starting Elasticsearch Server
elkx_1  |    ...done.
elkx_1  | waiting for Elasticsearch to be up (1/30)
...

In another shell, open a bash session in the running container (replacing <name of the running container> with the right value), and use X-Pack's setup-passwords tool to set the passwords for the built-in users.

$ docker exec -it <name of the running container> bash
# $ES_HOME/bin/x-pack/setup-passwords interactive
Initiating the setup of reserved user elastic,kibana,logstash_system passwords.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y


Enter password for [elastic]: changeme
Reenter password for [elastic]: changeme
Enter password for [kibana]: changeme
Reenter password for [kibana]: changeme
Enter password for [logstash_system]: changeme
Reenter password for [logstash_system]: changeme
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [elastic]

Now configure monitoring for Logstash with the right users:

# cat >> /opt/logstash/config/logstash.yml
xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: "changeme"
^D

Stop the container, then edit the docker-compose.yml as follows:

elkx:
  image: sebp/elkx
  ports:
    - "5601:5601"
    - "9200:9200"
    - "5044:5044"
  environment:
    - ELASTICSEARCH_USER=elastic
    - ELASTICSEARCH_PASSWORD=changeme
    - LOGSTASH_USER=elastic
    - LOGSTASH_PASSWORD=changeme
    - KIBANA_USER=kibana
    - KIBANA_PASSWORD=changeme

Then start the container again using docker-compose up.

from elkx-docker.

Hello Sebastien, i'm really really noob in all this, i maneged to run the container using the compose way you describe.

I'm getting the following error that i can't figure out, could you please help me with this?

the line 2 and 3 repeats for ever.

elkx_1 | [2017-11-22T16:43:24,019][ERROR][logstash.inputs.metrics ] Monitoring is not available: License information is currently unavailable. Please make sure you have added your production elasticsearch connection info in the xpack.monitoring.elasticsearch settings.
elkx_1 | [2017-11-22T16:43:24,026][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
elkx_1 | [2017-11-22T16:43:24,042][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}

from elkx-docker.

@luisvalleGH @spujadas I'm actually getting the same error

from elkx-docker.

Just merged #22, thanks.

from elkx-docker.

@yannouchou I've just pushed code for elkx:600-rc2 (build in progress).

After a fair amount of tweaking to have all the authentication bits working, everything now appears to be starting up and running properly (even though it's slightly more involved than in version 5, as it now requires a specific bootstrap procedure to enable the passwords to be initially created).

If you want to give it a go, your feedback would be most welcome.

from elkx-docker.

Seems to work smoothly for me, thanks a lot!

The only observation I have is that it seems that logs aren't being written to /var/log/elasticsearch/$CLUSTER_NAME.log for some reason. The file is present but remains desperately empty. Maybe that's caused by one of my customizations but I don't see why that would be the case...

Do you have a recommended way to check if the bootstrapping needs to be run for a given container? In my startup script I've used a custom file in the root of the elasticsearch data folder as a semaphore to mark the bootstrapping done, but there might be a better way from querying elastic itself?

Thanks for the great work!

from elkx-docker.

The only observation I have is that it seems that logs aren't being written to /var/log/elasticsearch/$CLUSTER_NAME.log for some reason. The file is present but remains desperately empty. Maybe that's caused by one of my customizations but I don't see why that would be the case...

Hmmm, you're right, Elasticsearch doesn't write any logs at all in /var/log/elasticsearch with the default set-up.

I've figured out that if I add xpack.security.audit.enabled: true to elasticsearch.yml then the (very chatty) audit logs will be written to /var/log/elasticsearch/elasticsearch_access.log, so at the very least it's using the right path.

Copying the log4j configuration from /etc/elasticsearch/log4j2.properties to /etc/elasticsearch/x-pack/log4j2.properties has no effect… so perhaps not writing the logs is actually an X-Pack feature. 🤔
Not an ELKX expert, so can't tell unfortunately.

Do you have a recommended way to check if the bootstrapping needs to be run for a given container? In my startup script I've used a custom file in the root of the elasticsearch data folder as a semaphore to mark the bootstrapping done, but there might be a better way from querying elastic itself?

Again, not an expert so I'd say that your method is probably just as good as any other one.

Another method I can think of would be to use ES's keystore:

• If the /etc/elasticsearch/elasticsearch.keystore file doesn't exist, then the container has never been started in bootstrap mode. => bootstrap needed

• If /opt/elasticsearch/bin/elasticsearch-keystore list shows bootstrap.pass then a bootstrap password has been set for ES. => elastic's password must be set

• Otherwise (i.e. keystore exists and no bootstrap.pass entry in it) => elastic's password has been changed after the bootstrap process.

Don't know what the actual best practice is though.

from elkx-docker.

I found the ES logs, they're actually under $ES_HOME, in weirdly-named files:
${sys:es.logs}.log
${sys:es.logs}_deprecation.log
etc...
Might be just a bug in the rc2?

from elkx-docker.

Apparently there was a change in the logging settings that was documented since 5.3 (https://www.elastic.co/guide/en/elasticsearch/reference/5.3/settings.html vs https://www.elastic.co/guide/en/elasticsearch/reference/5.2/settings.html), and became breaking in 6.0.0-rc2 (elastic/elasticsearch@fc3280b#diff-791b9967b12e6e58642504a148f441e2).
I've updated the Dockerfile from sebp/elk to use the official jvm.options and log4j2.properties, which restores the expected behaviour. Will commit/push and trigger the builds.

from elkx-docker.

OK, one more problem - not sure if we should keep using the same issue.
Apparently in production mode (which is dictated by the license that's imported in ES), ES has to use https.
This means that, among other things, the startup script will consistently fail as the http polling url will not respond.
I have a fix (involving passing en env variable to set the protocol), but don't have time to raise a PR now, might do over the weekend...

from elkx-docker.

Hehe, one more thing... :)
Looks like it's now expecting jvm.options to be in /etc/elasticsearch instead of /opt/config/elasticsearch...

from elkx-docker.

Looks like it's now expecting jvm.options to be in /etc/elasticsearch instead of /opt/config/elasticsearch...

Indeed! (regression introduced by spujadas/elk-docker@2b475f3, fixed by spujadas/elk-docker@4f6abb3 and spujadas/elk-docker@89481a6)

from elkx-docker.

Hello,
I tried to start the container but it gives this error. How can I solve this?

You must set the ELASTICSEARCH_USER and ELASTICSEARCH_PASSWORD environment
variables

from elkx-docker.

@Jonnifernandes Version 6 of X-Pack introduces a whole new way of managing user credentials, please see the README at https://github.com/spujadas/elkx-docker for pointers on the changes in the official documentation, and for guidance on how to first bootstrap a container to set a password for elastic, then set passwords for other users, then start the image using these passwords and the env vars mentioned in the error you got.

from elkx-docker.

I read the documentation, but im still not getting. Can you give me any hints?

docker run -it -e ELASTICSEARCH_PASSWORD=changeme -e ELASTICSEARCH_USER=elastic -e KIBANA_USER=kibana -e KIBANA_PASSWORD=kibana 7442af323a41

touch: cannot touch '/var/log/elasticsearch/{"error":{"root_cause":[{"type":"security_exception","reason":"failedtoauthenticateuser[elastic]","header":{"WWW-Authenticate":"Basicrealm="security"charset="UTF-8""}}],"type":"security_exception","reason":"failedtoauthenticateuser[elastic]","header":{"WWW-Authenticate":"Basicrealm="security"charset="UTF-8""}},"status":401}.log': File name too long
tail: cannot open '/var/log/elasticsearch/{"error":{"root_cause":[{"type":"security_exception","reason":"failedtoauthenticateuser[elastic]","header":{"WWW-Authenticate":"Basicrealm="security"charset="UTF-8""}}],"type":"security_exception","reason":"failedtoauthenticateuser[elastic]","header":{"WWW-Authenticate":"Basicrealm="security"charset="UTF-8""}},"status":401}.log' for reading: File name too long

from elkx-docker.

@spujadas Thank you for that detailed example!

Apparently I have a fundamental misunderstanding of docker. Once you issue the docker-compose up command for the second time (after adding the updated environmental variables), wouldn't that create a new container and discard the password changes made earlier?

from elkx-docker.

@matthew-hickok somewhat confusingly (especially in contrast to the standard docker run commands which would behave as you mention, and to docker start which can't be used to change environment variables), docker-compose up actually starts the previously created container (like docker start, i.e. preserving the contents of the container) whilst also enabling the env vars to be changed (like docker run and unlike docker start).

from elkx-docker.

@spujadas That's interesting. I had no idea compose behaved that way. This worked perfect. Thank you!

from elkx-docker.

from elkx-docker.

Hello Sebastien, i'm really really noob in all this, i maneged to run the container using the compose way you describe.

I'm getting the following error that i can't figure out, could you please help me with this?

the line 2 and 3 repeats for ever.

elkx_1 | [2017-11-22T16:43:24,019][ERROR][logstash.inputs.metrics ] Monitoring is not available: License information is currently unavailable. Please make sure you have added your production elasticsearch connection info in the xpack.monitoring.elasticsearch settings.
elkx_1 | [2017-11-22T16:43:24,026][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
elkx_1 | [2017-11-22T16:43:24,042][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}

+1

from elkx-docker.

@dickyangyi If @yannouchou's suggestion above doesn't help, then I'm going to recommend heading over to https://discuss.elastic.co/c/x-pack for help as this is most likely an X-Pack configuration issue rather than an actual problem with the image.

from elkx-docker.

this thread saved tons of my time because i don't know neither how to update the docker environment variables using docker run.

I would suggest to update document to add more instructions related to "docker-compose up" as you posted above, that's just so clear and convenient.

from elkx-docker.

@siben168 Good idea => done!

from elkx-docker.

@luisvalleGH @dickyangyi I've also experienced my changes to logstash.yml intermittently being dropped after restarting the container.

@spujadas I just submitted a PR (#22) for your review that has helped resolve this behavior on my end.

from elkx-docker.