Discuss: Stack versions as a resource to match stack packs about cloud-on-k8s HOT 4 CLOSED

I think we have settled on an approach for now. Good to close? @nkvoll

from cloud-on-k8s.

Before discussing we should come up with more concrete options re how we can solve.

from cloud-on-k8s.

Need for override

There are several things we might need to override compared to what the official ES Docker image contains:

• elasticsearch.yml settings (can also be passed as env var)
• plugins for our own use (eg. S3 repository plugin)
• extra plugins defined in the stack CRD (eg. hebrew language)
• keystore credentials
• (tweak JVM options?)

Ways to override

1. Build our own Docker image

We can build our own Docker image, inheriting from the official one, and do whatever we need in the start script that we maintain.

# cloud.elastic.co/operator/elasticsearch:6.4.2-1
FROM cloud.elastic.co/elasticsearch/elasticsearch:6.4.2

COPY plugins/ plugins/
COPY "start-script.sh"
CMD ["./start-script.sh"]

This is basically what we do in Cloud with Stack Packs.

Pros:

• Flexibility to do whatever we need before ES starts
• Pre-load usual plugins to avoid waiting for download to complete

Cons:

• Need to maintain lifecycle of these Docker images. One image per ES official Docker image, that we may need to upgrade if we change the content of start-script.sh.

2. Inject a start script

Instead of maintaining our own Docker image, we could "inject" the script we need to be executed beforehand.
This can be done through a ConfigMap:

kubectl create configmap init-script --from-file /path/to/init-script.sh

# pod spec
spec:
    volumeMounts:
        - name: init-script
          mountPath: /scripts
volumes:
      - name: init-script
        configMap:
          name: init-script
          defaultMode: 0744 # execution right

2.1 In the ES container

We can choose to inject this script within the ES container directly, and tweak the main command to not run Elasticsearch:

command: ["/scripts/init-script.sh"]

2.2 In an init container

We can use an init container to perform whatever needs to before running the official ES container.
By reusing the same ES official Docker image as the init container image, we can ensure the use of the appropriate versioned tool (eg. bin/elasticsearch-plugin).

For changes to be persisted between the init containers and the ES container, we can rely on emptyDir volumes.
That is, we would need volumes for the following:

• config/ (including elasticsearch.yml and plugin config directories - see elasticsearch-plugin source code)
• plugins/
• bin/ (for plugin scripts... but also includes ES binaries)
• etc?

For simplicity, that could be the entire ES directory from the init container that we copy over there.

Pros:

• No need to maintain a lifecycle of Docker Images corresponding to a particular ES version, we can reuse the same ES Docker image.
• A bit hacky but simpler to get started
• (with the init container:) we can inject some secrets in the init container, that are not injected in the ES container (eg. TLS stuff).

Cons:

• Need to wait for all plugins to be downloaded, since they're not part of the official image.
• Less flexibility in what we can do: limited to copying a shell or python script (that's probably enough though?)
• (with the init container:) shared volumes complexity
• Init script lifecycle management?
• ConfigMap needs to be in the same namespace as ES pods

from cloud-on-k8s.

Taking 1) sounds to me to be the right way to go longer term, but perhaps taking 2) might be a good way to progress right now with fewer deps and learning more/quicker about our requirements for what we'd want 1) to do vs what we'd want a pod-level sidecar to do.

For 2, the init script potentially needs to be potentially versioned, but if we go with supporting a limited version range for now it will probably be the same, so we have a trivial way of starting.

Init container (2.2) sounds the way to go to initialize the pod rather than replacing the command for the ES container -- less reaching into an official image to change stuff at runtime. E.g in the init containers, by using the ES container image, install plugins and copy stuff into different emptyDirs shared with the container images, facilitating shared-disk/volumes between the ES container image and a potential sidecar.

We do need to consider what we'd want the interface between the ES image and the rest of the Pod to be in terms of what volumes we're using, what paths we mount them to (e.g should we always assume /usr/share/elasticsearch/plugins ? Sounds like potentially an env variable?). Not that we need to make everything completely configurable/pluggable straight away, but it is something to consider for down the line.

from cloud-on-k8s.