The example.pillar shows that it is that way:
discovery.zen.ping.unicast.hosts: ["host1", "host2"]

But then it will be rendered as:

discovery.zen.ping.unicast.hosts:
- host1
- host2

The cluster will work afterwards but only because "discovery.zen.ping.unicast.hosts:" is actually empty and will use multicast for discovery which has its own problems according to the elasticsearch-documentation.

States execute, but service stops. I think this is due to a new check in 5.0.1.

Relevant log:

[2016-11-27T18:15:38,231][WARN ][o.e.b.BootstrapCheck     ] [I9Z5Rn8] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

We may need a state that sets this.

Elasticsearch docs support multiple data paths.

The path.data settings can be set to multiple paths, in which case all paths will be used to store data (although the files belonging to a single shard will all be stored on the same data path):

path:
  data:
    - /mnt/elasticsearch_1
    - /mnt/elasticsearch_2
    - /mnt/elasticsearch_3

See: https://www.elastic.co/guide/en/elasticsearch/reference/master/path-settings.html

Attempting to use a list produces the following output:

[CRITICAL] Rendering SLS 'base:elasticsearch.config' failed: while constructing a mapping
  in "<unicode string>", line 1, column 1:
    include:
    ^
found unacceptable key [ScalarNode(tag='tag:yaml.org,2002:str', value='/usr/shar
  in "<unicode string>", line 17, column 1:
    ['/usr/share/elasticsearch/data' ... 
    ^
local:
    Data failed to compile:
----------
    Rendering SLS 'base:elasticsearch.config' failed: while constructing a mappi
  in "<unicode string>", line 1, column 1:
    include:
    ^
found unacceptable key [ScalarNode(tag='tag:yaml.org,2002:str', value='/usr/shar
  in "<unicode string>", line 17, column 1:
    ['/usr/share/elasticsearch/data' ... 
    ^

On systems with systemd (e.g. Ubuntu Xenial/16.04), system limits must be specified via a systemd service file (official docs). I made PR #38 to do that, so the configuration can be set similar to sysconfig or jvm_opts. Not super confident with salt formulas yet, so please let me know if the PR needs some more work.

It would be nice to not remove the useful comments from /etc/default/elasticsearch

Am used to using the ini state module https://docs.saltstack.com/en/latest/ref/states/all/salt.states.ini_manage.html to get this, I'll submit a patch as soon as I have something working.

Hi

When installing ES 5.4.0 on Ubuntu in an LXD container you need to install with the following command

export ES_SKIP_SET_KERNEL_PARAMETERS=true; apt install elasticsearch

The ES_SKIP_SET_KERNEL_PARAMETERS=true env variable is necesarl because there are kernel parameters that are tuned on installation that are not available to the container and installation fails.

The license of this salt formula is missing.

more information:
https://www.elastic.co/guide/en/elasticsearch/reference/current/setting-system-settings.html#sysconfig
https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html

formula should be able to control /etc/elasticsearch/jvm.options.

@cmclaughlin said in #26:

file.replace was added to set JAVA_HOME in the sysconfig file, but that
results in the file changing every time the state is applied... and
then ElasticSearch restarts every time too.

On a debian jessie I end up with elasticsearch 1.0.3 installed (https://packages.debian.org/jessie/elasticsearch)

Maybe some sort of pinning has to be deployed or fromrepo used ?

ES 6.3 won't start without a keystore file existing. On many systems the user/group/context that ES runs in does not, and should not, have permission to create files in /etc/elasticsearch. Formula should create a default keystore in default location if one does not exist.

Jul 20 14:05:11 box systemd[1]: Starting Elasticsearch...
Jul 20 14:05:12 box elasticsearch[85725]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: org.elasticsearch.cli.UserException: unable to create temporary keystore at [/etc/elasticsearch/elasticsearch.keystore.tmp], please check filesystem permissions
Jul 20 14:05:12 box elasticsearch[85725]: Likely root cause: java.nio.file.AccessDeniedException: /etc/elasticsearch/elasticsearch.keystore.tmp
Jul 20 14:05:12 box elasticsearch[85725]: at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
Jul 20 14:05:12 box elasticsearch[85725]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
Jul 20 14:05:12 box elasticsearch[85725]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
Jul 20 14:05:12 box elasticsearch[85725]: at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
Jul 20 14:05:12 box systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE

When I change the repo from 7 to 8 then the both repositories exist in the source file.
This ends in an inconsistent state where both repositories are specified. It looks like the intention in code is to empty the file
but this isn't happening.

The init.sls file is pointing to files prefixed with elasticsearch, but the original files does not include this prefix.

This is related to #7 in the sense that I'm poking at some assumptions about how the formula should apply configuration and what it covers. I would like to open a discussion on these.

Right now the formula takes a number of directives for elasticsearch.yml, looks for these in the template and insets them into the rendered elasticsearch.yml.

Anything else (including node.master and node.data) are viewed as separate config, nested in elasticsearch.config.custom_options. These are passed along as is.

My question is if it makes sense to make this distinction - why not just pass along everything in elasticsearch.config directly to elasticsearch.yml?

Allow managing the logging config.

The directories specified via elasticsearch:config:path should be managed so that they exist with the correct permissions.

I view shield as a core functionality of Elasticsearch and think it makes sense to have a discussion about how this formula could/should support the configuration.

Most other plugins can be configured via elatsicsearch.yml, but Shield requires more than what is available today, ranging from distribution of roles.yml to installation of the json license.

Should this functionality be separated into it's own state, e.g. elasticsearch.shield (x-pack from version 5 onward), or does it make sense to include it in the elasticsearch.config state?

Salt 2018.3.3
elasticsearch.major_version: 6

      ID: apt-transport-https
Function: pkg.installed
  Result: True
 Comment: All specified packages are already installed
 Started: 17:19:08.751183
Duration: 32.439 ms
 Changes:
Warnings: The following package(s) are "virtual package" names: apt-
          transport-https. These will no longer be supported as of the
          Fluorine release. Please update your SLS file(s) to use the actual
          package name.

Fix to come when tested.

Setting

elasticsearch:
    sysconfig:
      ES_STARTUP_SLEEP_TIME: 30
     ES_JAVA_OPTS: -Xms2g -Xmx2g

ends up as ES_JAVA_OPTS=-Xms2g -Xmx2g which is not valid in shell/sysconfig and causes a service start error:

[ec2-user@ip-10-0-3-52 ~]$ sudo service elasticsearch start
/etc/sysconfig/elasticsearch: line 4: -Xmx2g: command not found

Is it just me or does state.highstate test=True always say the package will be upgraded even when it's up to date?

      ID: elasticsearch_pkg
Function: pkg.installed
    Name: elasticsearch
  Result: None
 Comment: The following packages would be installed/updated: elasticsearch=5.*
 Started: 23:23:50.533340
Duration: 1.268 ms
 Changes:

What do you think about adding support for pinning the minor version so test=True applies cleanly? I'd be happy to open a PR... also let me know if you have any other solutions in mind.

Im currently investigating the cause, but running this formula on aws linux results in:

0:01:48     amazon-ebs: [ERROR   ] Command '/sbin/chkconfig --list elasticsearch' failed with return code: 1
20:01:48     amazon-ebs: [ERROR   ] stderr: service elasticsearch supports chkconfig, but is not referenced in any runlevel (run 'chkconfig --add elasticsearch')
20:01:48     amazon-ebs: [ERROR   ] retcode: 1
20:03:17     amazon-ebs: [ERROR   ] Command '/sbin/service elasticsearch start' failed with return code: 1
20:03:17     amazon-ebs: [ERROR   ] output: Starting elasticsearch: [FAILED]
20:03:17     amazon-ebs: [ERROR   ] Command '/sbin/chkconfig --list elasticsearch' failed with return code: 1
20:03:17     amazon-ebs: [ERROR   ] stderr: service elasticsearch supports chkconfig, but is not referenced in any runlevel (run 'chkconfig --add elasticsearch')
20:03:17     amazon-ebs: [ERROR   ] retcode: 1
20:03:17     amazon-ebs: [ERROR   ] {'elasticsearch': True}
20:03:49     amazon-ebs: local:

Running it twice results in a success.

Hi, is there any chance to support Docker?

Elasticsearch doesn't exist in Bionic so installation fails when determining version.

- Rendering SLS 'base:elasticsearch.plugins' failed: Jinja variable dict object has no element 0
  /var/tmp/.salt_1a862b_salt/running_data/var/cache/salt/minion/files/base/elasticsearch/settings.sls(20):
  ---
  [...]
    {% if pillar_major_version %}
      {% set major_version = pillar_major_version %}
    {% else %}
      {% set pkg_version = salt['pkg.version'](elasticsearch_map.pkg) %}
      {% if pkg_version %}
        {% set major_version = pkg_version[0] | int %}    <======================
      {% else %}
        {% set major_version = False %}
      {% endif %}
    {% endif %}
  {% endif %}

So this has been something I've been thinking about for a while. I'm opening this issue to both note this down and maybe start a discussion if anyone else could be interested in a feature like this.

So what's this about?

Say I want to set cluster.routing.allocation.awareness when spinning up a new cluster. Maybe I want to inject a default template. Or set any other thing available via the Elasticsearch API in a managed way.

Great thing is that YAML defined in pillar translates easily to JSON and vice versa.

But is Saltstack able to do this in an easy way? I guess we're looking at something like salt.utils.http (info here: https://docs.saltstack.com/en/latest/topics/tutorials/http.html, https://www.youtube.com/watch?v=m9AQxcSDkaE)

Thoughts?

On Debian (specifically Ubuntu 18.04.1) the postinstall of elasticsearch 6.x requires Java to be installed.
This formula doesn't manage this dependency.

See also https://github.com/saltstack-formulas/logstash-formula, which does.

I cannot see any elasticsearch.plugins state. It is possible that you forgot to add the sls to dee6a41 ?

That is causing the state not to compile:

sudo salt server state.sls elasticsearch
server:
    Data failed to compile:
----------
    Specified SLS elasticsearch.plugins in saltenv base is not available on the salt master or through a configured fileserver

https://www.elastic.co/guide/en/elasticsearch/reference/7.16/advanced-configuration.html#set-jvm-options