Pods status go into CrashLoopBackOff about kubernetes-elasticsearch-cluster HOT 11 CLOSED

kubectl logs <pod>

from kubernetes-elasticsearch-cluster.

@tspacek this may be something else, namely the default value for ES_HEAP_SIZE. I will be validating that ASAP.

from kubernetes-elasticsearch-cluster.

I'm not sure if this is ok to make the master node also a client and a data plus make the data a client as well

Elasticsearch documentation disrecommend this practice.

Other than that, we need to see pod logs to understand why it's crashing.

from kubernetes-elasticsearch-cluster.

ye but a single server that perform all roles actually work just fine,
I was more wondering whether the fabric8io unicast discovery should work when multiple roles are activated at once,

There are no errors.....

The only thing I see is this:

kubectl describe pod es-data-2599720219-u9axv
  2h    8m  25  {kubelet 172.17.8.102}  spec.containers{es-data}    Normal  Pulling     pulling image "quay.io/pires/docker-elasticsearch-kubernetes:2.3.4"
  2h    8m  25  {kubelet 172.17.8.102}  spec.containers{es-data}    Normal  Pulled      Successfully pulled image "quay.io/pires/docker-elasticsearch-kubernetes:2.3.4"
  8m    8m  1   {kubelet 172.17.8.102}  spec.containers{es-data}    Normal  Created     Created container with docker id 817f72bd6eea
  8m    8m  1   {kubelet 172.17.8.102}  spec.containers{es-data}    Normal  Started     Started container with docker id 817f72bd6eea
  1h    12s 81  {kubelet 172.17.8.102}                  Warning FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "es-data" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=es-data pod=es-data-2599720219-u9axv_default(330fb913-5f22-11e6-be3c-080027e9ede8)"

  2h    12s 182 {kubelet 172.17.8.102}  spec.containers{es-data}    Warning BackOff Back-off restarting failed docker container

Any suggestions how to debug further?

Many thanks.

from kubernetes-elasticsearch-cluster.

yea well, same thing, nothing that seems to be suspicious (same errors as in describe)

I also tried to take a look at /data/log/.. but nothing,

Anyway, I keep this issue opened for a while, meanwhile I'll try to investigate it a bit more,

Thanks.

from kubernetes-elasticsearch-cluster.

@asaf Did you get to the bottom of this? I get the same. I suspect because I am using https://github.com/kubernetes/minikube

Here's an example of the pod logs:

➜  kubernetes-elasticsearch-cluster git:(master) kubectl logs es-master-3368190183-uhift
[2016-08-24 11:33:00,471][INFO ][node                     ] [Sluk] version[2.3.5], pid[11], build[90f439f/2016-07-27T10:36:52Z]
[2016-08-24 11:33:00,472][INFO ][node                     ] [Sluk] initializing ...
[2016-08-24 11:33:01,644][INFO ][plugins                  ] [Sluk] modules [reindex, lang-expression, lang-groovy], plugins [cloud-kubernetes], sites []
[2016-08-24 11:33:01,688][INFO ][env                      ] [Sluk] using [1] data paths, mounts [[/data (tmpfs)]], net usable_space [644.7mb], net total_space [896.2mb], spins? [no], types [tmpfs]
[2016-08-24 11:33:01,689][INFO ][env                      ] [Sluk] heap size [503.6mb], compressed ordinary object pointers [true]
Killed

from kubernetes-elasticsearch-cluster.

I can confirm this happens, pods are killed. Need to make sure why, though. My gut feeling is OOM.

My environment is 4 CoreOS alpha VM's, 1 vCPU + 4GB RAM, Kubernetes 1.3.5.

from kubernetes-elasticsearch-cluster.

Can confirm it's stable if I set ES_HEAP_SIZE such that the sum of all pods' is lower than Minikube's default 1gb.

from kubernetes-elasticsearch-cluster.

@tspacek can confirm as well that's the case. I will be closing this by setting a very low value, 256MB and adding a note on docs.

from kubernetes-elasticsearch-cluster.

@asaf try again and let us know!

from kubernetes-elasticsearch-cluster.

Just want to share my experience with this,
had the same issue, 4 hyper-v vm's with 1gb RAM each. Had random crashes all over the es cluster. I allocated 2gb Ram to each machine, crashes stop and the es cluster runs stable.

from kubernetes-elasticsearch-cluster.