Comments (28)
jasonkuhrt
commented on May 18, 2024
Also error with lb-node:
[2015-02-03T15:58:08.370596540Z] [2015-02-03 15:58:08,370][WARN ][io.fabric8.elasticsearch.discovery.k8s.K8sUnicastHostsProvider] [Vision] Exception caught during discovery javax.ws.rs.ProcessingException : java.net.SocketException: SocketException invoking http://10.23.243.201:443/api/v1beta1/pods: Connection reset
[2015-02-03T15:58:08.377486396Z] [2015-02-03 15:58:08,377][WARN ][org.apache.cxf.phase.PhaseInterceptorChain] Interceptor for {http://10.23.243.201:443}WebClient has thrown exception, unwinding now
[2015-02-03T15:58:08.377486396Z] org.apache.cxf.interceptor.Fault: Could not send Message.
[2015-02-03T15:58:08.377486396Z] at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)
[2015-02-03T15:58:08.377486396Z] at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
[2015-02-03T15:58:08.377486396Z] at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:619)
[2015-02-03T15:58:08.377486396Z] at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:674)
[2015-02-03T15:58:08.377486396Z] at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:224)
[2015-02-03T15:58:08.377486396Z] at com.sun.proxy.$Proxy28.getPods(Unknown Source)
[2015-02-03T15:58:08.377486396Z] at io.fabric8.kubernetes.api.KubernetesHelper.getPodMap(KubernetesHelper.java:310)
[2015-02-03T15:58:08.377486396Z] at io.fabric8.elasticsearch.discovery.k8s.K8sUnicastHostsProvider.buildDynamicNodes(K8sUnicastHostsProvider.java:101)
[2015-02-03T15:58:08.377486396Z] at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.sendPings(UnicastZenPing.java:316)
[2015-02-03T15:58:08.377486396Z] at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.ping(UnicastZenPing.java:224)
[2015-02-03T15:58:08.377486396Z] at org.elasticsearch.discovery.zen.ping.ZenPingService.ping(ZenPingService.java:146)
[2015-02-03T15:58:08.377486396Z] at org.elasticsearch.discovery.zen.ping.ZenPingService.pingAndWait(ZenPingService.java:124)
[2015-02-03T15:58:08.377486396Z] at org.elasticsearch.discovery.zen.ZenDiscovery.findMaster(ZenDiscovery.java:941)
[2015-02-03T15:58:08.377486396Z] at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:337)
[2015-02-03T15:58:08.377486396Z] at org.elasticsearch.discovery.zen.ZenDiscovery.access$6000(ZenDiscovery.java:80)
[2015-02-03T15:58:08.377486396Z] at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1318)
[2015-02-03T15:58:08.377486396Z] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[2015-02-03T15:58:08.377486396Z] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[2015-02-03T15:58:08.377486396Z] at java.lang.Thread.run(Thread.java:745)
[2015-02-03T15:58:08.377486396Z] Caused by: java.net.SocketException: SocketException invoking http://10.23.243.201:443/api/v1beta1/pods: Connection reset
[2015-02-03T15:58:08.377486396Z] at sun.reflect.GeneratedConstructorAccessor9.newInstance(Unknown Source)
[2015-02-03T15:58:08.377486396Z] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[2015-02-03T15:58:08.377486396Z] at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
[2015-02-03T15:58:08.377486396Z] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1359)
[2015-02-03T15:58:08.377486396Z] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1343)
[2015-02-03T15:58:08.377486396Z] at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
[2015-02-03T15:58:08.377486396Z] at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:638)
[2015-02-03T15:58:08.377486396Z] at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
[2015-02-03T15:58:08.377486396Z] ... 18 more
[2015-02-03T15:58:08.377486396Z] Caused by: java.net.SocketException: Connection reset
[2015-02-03T15:58:08.377486396Z] at java.net.SocketInputStream.read(SocketInputStream.java:189)
[2015-02-03T15:58:08.377486396Z] at java.net.SocketInputStream.read(SocketInputStream.java:121)
[2015-02-03T15:58:08.377486396Z] at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
[2015-02-03T15:58:08.377486396Z] at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
[2015-02-03T15:58:08.377486396Z] at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
[2015-02-03T15:58:08.377486396Z] at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:703)
[2015-02-03T15:58:08.377486396Z] at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
[2015-02-03T15:58:08.377486396Z] at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:674)
[2015-02-03T15:58:08.377486396Z] at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1535)
[2015-02-03T15:58:08.377486396Z] at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
[2015-02-03T15:58:08.377486396Z] at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
[2015-02-03T15:58:08.377486396Z] at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:266)
[2015-02-03T15:58:08.377486396Z] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1557)
[2015-02-03T15:58:08.377486396Z] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1527)
[2015-02-03T15:58:08.377486396Z] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1330)
[2015-02-03T15:58:08.377486396Z] ... 21 more
from kubernetes-elasticsearch-cluster.
pires
commented on May 18, 2024
This is some problem accessing your kubernetes api.
from kubernetes-elasticsearch-cluster.
jasonkuhrt
commented on May 18, 2024
hmm, yeah.
from kubernetes-elasticsearch-cluster.
jasonkuhrt
commented on May 18, 2024
What business does Elasticsearch have using the Kubernetes API?
from kubernetes-elasticsearch-cluster.
pires
commented on May 18, 2024
It uses kubernetes api for discovering other elasticsearch nodes based on some labels (in this case component:elasticsearch) .
from kubernetes-elasticsearch-cluster.
jasonkuhrt
commented on May 18, 2024
@pires Ah ok. I'm going to be back on this tomorrow and will dig in again.
from kubernetes-elasticsearch-cluster.
jasonkuhrt
commented on May 18, 2024
e.g. https://github.com/pires/kubernetes-elasticsearch-cluster/blob/master/node-data/elasticsearch.yml#L14 right?
from kubernetes-elasticsearch-cluster.
jasonkuhrt
commented on May 18, 2024
I noticed that it is trying to make a request like so: http://10.23.243.201:443/api/v1beta1/pods. I sshed into my k8s machines to try and replicate the request with curl but no luck:
asonkuhrt@k8s-littlecloud-3-node-2:~$ curl -X GET http://10.23.243.201:443/api/v1beta1/pods -v
* About to connect() to 10.23.243.201 port 443 (#0)
* Trying 10.23.243.201...
* connected
* Connected to 10.23.243.201 (10.23.243.201) port 443 (#0)
> GET /api/v1beta1/pods HTTP/1.1
> User-Agent: curl/7.26.0
> Host: 10.23.243.201:443
> Accept: */*
>
* additional stuff not fine transfer.c:1037: 0 0
* Recv failure: Connection reset by peer
* Closing connection #0
curl: (56) Recv failure: Connection reset by peer
I feel like until I can create a successful HTTP request myself with curl then of course elasticsearch-kubernetes is not going to work.
from kubernetes-elasticsearch-cluster.
jasonkuhrt
commented on May 18, 2024
I was able to successfully access my Kubernetes Cluster's pods from my compute by navigating to https://104.154.53.226:443/api/v1beta1/pods in my browser (had to accept the insecure warning, and then enter my kubernetes auth credentials).
This means that the API is working at this level. Why its not working when elasticsearch-cloud-kubernetes tries to access it I do not know yet.
I do not understand the IP elasticsearch-cloud-kubernetes is trying to use. The private IP of my Kubernetes Master node is 10.240.16.28 which I believe is where the HTTP API lives, the same API that elasticsearch-cloud-kubernetes is trying to access, right? If so, why is it trying the IP of 10.23.243.201? Also I find http:// combined with 443 suspicious but not sure about that yet.
from kubernetes-elasticsearch-cluster.
pires
commented on May 18, 2024
This has something to do with your Kubernetes master configuration and the environment variables injected into the container(s). Can you ssh into a minion, run docker ps, choose one container ID and..
docker exec 685030acfc2c env
You should see something like:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=a96d1b26-a95d-11e4-9459-0800272d7481
KUBERNETES_RO_SERVICE_HOST=10.244.208.218
KUBERNETES_RO_SERVICE_PORT=80
KUBERNETES_RO_PORT=tcp://10.244.208.218:80
KUBERNETES_RO_PORT_80_TCP=tcp://10.244.208.218:80
KUBERNETES_RO_PORT_80_TCP_PROTO=tcp
KUBERNETES_RO_PORT_80_TCP_PORT=80
KUBERNETES_RO_PORT_80_TCP_ADDR=10.244.208.218
KUBERNETES_SERVICE_HOST=10.244.205.77
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT=tcp://10.244.205.77:443
KUBERNETES_PORT_443_TCP=tcp://10.244.205.77:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.244.205.77
ELASTICSEARCH_SERVICE_HOST=10.244.85.250
ELASTICSEARCH_SERVICE_PORT=9200
ELASTICSEARCH_PORT=tcp://10.244.85.250:9200
ELASTICSEARCH_PORT_9200_TCP=tcp://10.244.85.250:9200
ELASTICSEARCH_PORT_9200_TCP_PROTO=tcp
ELASTICSEARCH_PORT_9200_TCP_PORT=9200
ELASTICSEARCH_PORT_9200_TCP_ADDR=10.244.85.250
LOGSTASH_SERVICE_HOST=10.244.159.44
LOGSTASH_SERVICE_PORT=5043
LOGSTASH_PORT=tcp://10.244.159.44:5043
LOGSTASH_PORT_5043_TCP=tcp://10.244.159.44:5043
LOGSTASH_PORT_5043_TCP_PROTO=tcp
LOGSTASH_PORT_5043_TCP_PORT=5043
LOGSTASH_PORT_5043_TCP_ADDR=10.244.159.44
KIBANA_SERVICE_HOST=10.244.63.95
KIBANA_SERVICE_PORT=80
KIBANA_PORT=tcp://10.244.63.95:80
KIBANA_PORT_80_TCP=tcp://10.244.63.95:80
KIBANA_PORT_80_TCP_PROTO=tcp
KIBANA_PORT_80_TCP_PORT=80
KIBANA_PORT_80_TCP_ADDR=10.244.63.95
HOME=/root
JAVA_HOME=/usr/lib/jvm/java-8-oracle
Can you point out the values for KUBERNETES_RO_SERVICE_HOST and KUBERNETES_RO_SERVICE_PORT=80?
from kubernetes-elasticsearch-cluster.
jasonkuhrt
commented on May 18, 2024
master on node-2
jasonkuhrt@k8s-littlecloud-3-node-2:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3c497873afd2 pires/elasticsearch:master "/usr/bin/runsvdir - 4 minutes ago Up 4 minutes k8s_elasticsearch-master.4d0ce765_5a667e72-ae1b-11e4-bfb6-42010af0101c.default.api_5a667e72-ae1b-11e4-bfb6-42010af0101c_e341e1b8
8aac55311653 kubernetes/pause:go "/pause" 4 minutes ago Up 4 minutes k8s_net.5569bb00_5a667e72-ae1b-11e4-bfb6-42010af0101c.default.api_5a667e72-ae1b-11e4-bfb6-42010af0101c_b1b7e583
e16dcb68e0ca quay.io/littlebits/bitlab-voteview:latest "npm start" 6 days ago Up 6 days k8s_bitlab-voteview.90dfc967_bitlab-voteview.default.api_65193013-a8ea-11e4-bfb6-42010af0101c_640854bc
6e2f229887c2 google/cadvisor:0.7.1 "/usr/bin/cadvisor" 6 days ago Up 6 days k8s_cadvisor.a53216ec_cadvisor-agent.file-6bb810db-k8s-littlecloud-3-node-2.file_51a7b786a2968ad5480eee8221752019_627bf517
1c0bf2ff432c kubernetes/pause:go "/pause" 6 days ago Up 6 days 0.0.0.0:4194->8080/tcp k8s_net.beddb979_cadvisor-agent.file-6bb810db-k8s-littlecloud-3-node-2.file_51a7b786a2968ad5480eee8221752019_c26bb937
3de8825ce842 kubernetes/pause:go "/pause" 6 days ago Up 6 days k8s_net.3e0f95f3_bitlab-voteview.default.api_65193013-a8ea-11e4-bfb6-42010af0101c_343520fb
jasonkuhrt@k8s-littlecloud-3-node-2:~$ docker exec 3c497873afd2 env
FATA[0000] Post http:///var/run/docker.sock/v1.16/containers/3c497873afd2/exec: dial unix /var/run/docker.sock: permission denied. Are you trying to connect to a TLS-enabled daemon without TLS?
jasonkuhrt@k8s-littlecloud-3-node-2:~$ sudo docker exec 3c497873afd2 env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=5a667e72-ae1b-11e4-bfb6-42010af0101c
SKYDNS_PORT=udp://10.23.240.10:53
SKYDNS_PORT_53_UDP_PROTO=udp
ELASTICSEARCH_PORT_9200_TCP_PROTO=tcp
CLOUD_USER_DEBUG_SERVICE_PORT=80
KUBERNETES_RO_SERVICE_PORT=80
KUBERNETES_RO_PORT=tcp://10.23.255.235:80
KUBERNETES_RO_PORT_80_TCP=tcp://10.23.255.235:80
KUBERNETES_PORT_443_TCP=tcp://10.23.243.201:443
CLOUD_API_STREAM_SERVICE_PORT=443
CLOUD_USER_DEBUG_PORT_80_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
SKYDNS_SERVICE_HOST=10.23.240.10
ELASTICSEARCH_SERVICE_HOST=10.23.247.62
ELASTICSEARCH_SERVICE_PORT=9200
ELASTICSEARCH_PORT=tcp://10.23.247.62:9200
ELASTICSEARCH_PORT_9200_TCP_PORT=9200
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_RO_SERVICE_HOST=10.23.255.235
SKYDNS_PORT_53_UDP=udp://10.23.240.10:53
SKYDNS_PORT_53_UDP_ADDR=10.23.240.10
CLOUD_API_STREAM_PORT=tcp://10.23.248.123:443
CLOUD_USER_DEBUG_PORT_80_TCP_ADDR=10.23.252.127
KUBERNETES_PORT=tcp://10.23.243.201:443
KUBERNETES_RO_PORT_80_TCP_PORT=80
KUBERNETES_RO_PORT_80_TCP_ADDR=10.23.255.235
SKYDNS_SERVICE_PORT=53
CLOUD_API_STREAM_PORT_443_TCP=tcp://10.23.248.123:443
CLOUD_API_STREAM_PORT_443_TCP_PORT=443
KUBERNETES_SERVICE_HOST=10.23.243.201
KUBERNETES_PORT_443_TCP_ADDR=10.23.243.201
SKYDNS_PORT_53_UDP_PORT=53
ELASTICSEARCH_PORT_9200_TCP=tcp://10.23.247.62:9200
ELASTICSEARCH_PORT_9200_TCP_ADDR=10.23.247.62
CLOUD_USER_DEBUG_SERVICE_HOST=10.23.252.127
CLOUD_USER_DEBUG_PORT=tcp://10.23.252.127:80
CLOUD_USER_DEBUG_PORT_80_TCP_PORT=80
CLOUD_API_STREAM_SERVICE_HOST=10.23.248.123
CLOUD_API_STREAM_PORT_443_TCP_ADDR=10.23.248.123
CLOUD_USER_DEBUG_PORT_80_TCP=tcp://10.23.252.127:80
KUBERNETES_SERVICE_PORT=443
KUBERNETES_RO_PORT_80_TCP_PROTO=tcp
CLOUD_API_STREAM_PORT_443_TCP_PROTO=tcp
HOME=/root
JAVA_HOME=/usr/lib/jvm/java-8-oracle
ES_PKG_NAME=elasticsearch-1.4.2
load-balancer on node-1
jasonkuhrt@k8s-littlecloud-3-node-1:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4c0d82476c72 pires/elasticsearch:lb "/usr/bin/runsvdir - 7 minutes ago Up 7 minutes k8s_elasticsearch-lb.422b07fc_61a64258-ae1b-11e4-bfb6-42010af0101c.default.api_61a64258-ae1b-11e4-bfb6-42010af0101c_50743064
2eb582eda477 kubernetes/pause:go "/pause" 7 minutes ago Up 7 minutes k8s_net.71ffdf04_61a64258-ae1b-11e4-bfb6-42010af0101c.default.api_61a64258-ae1b-11e4-bfb6-42010af0101c_52a19dc3
4cc2904d8d76 gcr.io/littlebits_electronics/cloud-user-debug-nginx:9 "nginx -g 'daemon of 2 days ago Up 2 days k8s_cloud-user-debug-nginx.e23ed9a1_cloud-user-debug.default.api_5a058695-abef-11e4-bfb6-42010af0101c_7eb3cbbe
2d2d87a7e1ef gcr.io/littlebits_electronics/cloud-user-debug:latest "npm start" 2 days ago Up 2 days k8s_cloud-user-debug.e794de9e_cloud-user-debug.default.api_5a058695-abef-11e4-bfb6-42010af0101c_b276175e
38bd44436aee kubernetes/pause:go "/pause" 2 days ago Up 2 days k8s_net.a1e820_cloud-user-debug.default.api_5a058695-abef-11e4-bfb6-42010af0101c_55e64d84
c63975643a84 google/cadvisor:0.7.1 "/usr/bin/cadvisor" 5 days ago Up 5 days k8s_cadvisor.a53216ec_cadvisor-agent.file-6bb810db-k8s-littlecloud-3-node-1.file_f8d6d295b710e6b51f6d49667533769e_1d7ff0cd
fac961e3ff04 kubernetes/pause:go "/pause" 5 days ago Up 5 days 0.0.0.0:4194->8080/tcp k8s_net.beddb979_cadvisor-agent.file-6bb810db-k8s-littlecloud-3-node-1.file_f8d6d295b710e6b51f6d49667533769e_da90e6bc
jasonkuhrt@k8s-littlecloud-3-node-1:~$ sudo docker exec 4c0d82476c72 env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=61a64258-ae1b-11e4-bfb6-42010af0101c
KUBERNETES_RO_PORT_80_TCP_PORT=80
SKYDNS_SERVICE_PORT=53
ELASTICSEARCH_PORT_9200_TCP=tcp://10.23.247.62:9200
CLOUD_USER_DEBUG_SERVICE_HOST=10.23.252.127
CLOUD_USER_DEBUG_PORT_80_TCP_ADDR=10.23.252.127
KUBERNETES_RO_SERVICE_PORT=80
CLOUD_API_STREAM_PORT_443_TCP_ADDR=10.23.248.123
ELASTICSEARCH_PORT=tcp://10.23.247.62:9200
SKYDNS_PORT_53_UDP=udp://10.23.240.10:53
SKYDNS_PORT_53_UDP_PORT=53
CLOUD_API_STREAM_PORT_443_TCP=tcp://10.23.248.123:443
CLOUD_USER_DEBUG_PORT_80_TCP_PORT=80
SKYDNS_SERVICE_HOST=10.23.240.10
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT_443_TCP_PORT=443
SKYDNS_PORT_53_UDP_ADDR=10.23.240.10
CLOUD_API_STREAM_SERVICE_PORT=443
ELASTICSEARCH_PORT_9200_TCP_ADDR=10.23.247.62
CLOUD_USER_DEBUG_PORT_80_TCP=tcp://10.23.252.127:80
CLOUD_USER_DEBUG_PORT_80_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP=tcp://10.23.243.201:443
KUBERNETES_PORT_443_TCP_ADDR=10.23.243.201
CLOUD_USER_DEBUG_SERVICE_PORT=80
CLOUD_USER_DEBUG_PORT=tcp://10.23.252.127:80
KUBERNETES_SERVICE_HOST=10.23.243.201
KUBERNETES_PORT=tcp://10.23.243.201:443
ELASTICSEARCH_SERVICE_PORT=9200
ELASTICSEARCH_PORT_9200_TCP_PROTO=tcp
KUBERNETES_RO_PORT_80_TCP_ADDR=10.23.255.235
ELASTICSEARCH_PORT_9200_TCP_PORT=9200
KUBERNETES_RO_PORT=tcp://10.23.255.235:80
KUBERNETES_RO_PORT_80_TCP=tcp://10.23.255.235:80
SKYDNS_PORT=udp://10.23.240.10:53
CLOUD_API_STREAM_SERVICE_HOST=10.23.248.123
CLOUD_API_STREAM_PORT=tcp://10.23.248.123:443
CLOUD_API_STREAM_PORT_443_TCP_PROTO=tcp
CLOUD_API_STREAM_PORT_443_TCP_PORT=443
SKYDNS_PORT_53_UDP_PROTO=udp
ELASTICSEARCH_SERVICE_HOST=10.23.247.62
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_RO_SERVICE_HOST=10.23.255.235
KUBERNETES_RO_PORT_80_TCP_PROTO=tcp
HOME=/root
JAVA_HOME=/usr/lib/jvm/java-8-oracle
ES_PKG_NAME=elasticsearch-1.4.2
from kubernetes-elasticsearch-cluster.
jasonkuhrt
commented on May 18, 2024
So according to the environment variable values inside the container KUBERNETES_RO_SERVICE_HOST is 10.23.255.235 in both lb and master. On the other hand the error from Elasticsearch is:
master
2015-02-06T16:27:41.178303447Z [2015-02-06 16:27:41,178][WARN ][io.fabric8.elasticsearch.discovery.k8s.K8sUnicastHostsProvider] [Sweetface] Exception caught during discovery javax.ws.rs.ProcessingException : java.net.SocketException: SocketException invoking http://10.23.243.201:443/api/v1beta1/pods: Connection reset
2015-02-06T16:27:42.685670186Z [2015-02-06 16:27:42,685][WARN ][org.apache.cxf.phase.PhaseInterceptorChain] Interceptor for {http://10.23.243.201:443}WebClient has thrown exception, unwinding now
load-balancer
2015-02-06T16:28:39.474594981Z [2015-02-06 16:28:39,474][WARN ][io.fabric8.elasticsearch.discovery.k8s.K8sUnicastHostsProvider] [Sweetface] Exception caught during discovery javax.ws.rs.ProcessingException : java.net.SocketException: SocketException invoking http://10.23.243.201:443/api/v1beta1/pods: Connection reset
2015-02-06T16:28:39.480620340Z [2015-02-06 16:28:39,480][WARN ][org.apache.cxf.phase.PhaseInterceptorChain] Interceptor for {http://10.23.243.201:443}WebClient has thrown exception, unwinding now
--edit So to be clear, in both cases:
Elasticsearch is trying to use 10.23.243.201 but KUBERNETES_RO_SERVICE_HOST equals 10.23.255.235.
from kubernetes-elasticsearch-cluster.
jasonkuhrt
commented on May 18, 2024
Based on the IP Elasticsearch is trying to use (10.23.243.201) here are the container-ENV vars that have the same value as the IP being accessed by Elasticsearch:
KUBERNETES_PORT_443_TCP=
tcp://10.23.243.201:443
KUBERNETES_PORT_443_TCP_ADDR=
10.23.243.201
KUBERNETES_SERVICE_HOST=
10.23.243.201
KUBERNETES_PORT=
tcp://10.23.243.201:443
from kubernetes-elasticsearch-cluster.
pires
commented on May 18, 2024
This could be related to https://groups.google.com/forum/#!topic/google-containers/4yH5xBTbu2U
from kubernetes-elasticsearch-cluster.
jasonkuhrt
commented on May 18, 2024
@pires Question. Should io.fabric8.elasticsearch.discovery.k8s.K8sUnicastHostsProvider be trying to use KUBERNETES_SERVICE_HOST or KUBERNETES_RO_SERVICE_HOST?
My understanding is that io.fabric8.elasticsearch.discovery.k8s.K8sUnicastHostsProvider should use the RO variant. But it is not. This would mean a bug with ... https://github.com/fabric8io/elasticsearch-cloud-kubernetes, no?
from kubernetes-elasticsearch-cluster.
pires
commented on May 18, 2024
Well, in principle both approaches should work. I think it's a kube bug as I've pointed out before. Let's wait to see if Cullen's PR will fix it.
from kubernetes-elasticsearch-cluster.
cjcullen
commented on May 18, 2024
Yeah. Hitting the RO endpoint should probably give elasticsearch what it needs, but it also should work how it is.
from kubernetes-elasticsearch-cluster.
cjcullen
commented on May 18, 2024
Here's the issue I filed to hopefully help others who are searching for similar problems:
kubernetes/kubernetes#4216
I'm working on a PR now.
from kubernetes-elasticsearch-cluster.
pires
commented on May 18, 2024
@cjcullen thanks for letting us know.
@jdyson or @jstrachan want to comment on the Fabric8 Kubernetes API side of things, namely why you chose to read from the read-write service instead of read-only?
from kubernetes-elasticsearch-cluster.
jasonkuhrt
commented on May 18, 2024
Regarding fix being released to GKE, quoting @cjcullen from mailing list:
If everything goes smoothly, this should be fixed in GKE next week.
from kubernetes-elasticsearch-cluster.
pires
commented on May 18, 2024
Awesome, @jasonkuhrt please close this when you're able to test said fix.
from kubernetes-elasticsearch-cluster.
cjcullen
commented on May 18, 2024
Updating my original statement: Kubernetes only had a patch release this week, so the next minor release (with this fix) won't be live in GKE until next week :(.
from kubernetes-elasticsearch-cluster.
pires
commented on May 18, 2024
@cjcullen thanks for the update. Since this is a P1, will you cut a 0.10.2 release anytime soon?
from kubernetes-elasticsearch-cluster.
pires
commented on May 18, 2024
@jasonkuhrt can you confirm this is fixed with 0.11.0?
from kubernetes-elasticsearch-cluster.
jasonkuhrt
commented on May 18, 2024
@pires Ah ok. I will try this in the next few days.
from kubernetes-elasticsearch-cluster.
pires
commented on May 18, 2024
It's working :-)
from kubernetes-elasticsearch-cluster.
jasonkuhrt
commented on May 18, 2024
@pires Awesome!
from kubernetes-elasticsearch-cluster.
heytrav
commented on May 18, 2024
I'm currently seeing the behaviour above using the v0.11.0-dirty binaries.
from kubernetes-elasticsearch-cluster.
Related Issues (20)
- Deprecated service HOT 1
- Missing required field "selector" in io.k8s.api.apps.v1.StatefulSetSpec HOT 2
- ingest-attachment does not work in Cluster HOT 1
- Master pods refuse to start with memory limits set on AWS EKS HOT 2
- Question: Memory requirements HOT 2
- enable discovery, other nodes to join the cluster HOT 1
- best way to add synonyms to an index
- Reloadable secure settings or restart master
- Unable to set MEMORY_LOCK to true HOT 1
- Statefulset not working with resize of pvc
- Re-election takes over 30 seconds when deleting master pod (but fast when killing the process directly) HOT 2
- how to set attributes HOT 1
- es-data nodes exceeding Xmx memory HOT 2
- Java heap space error for es-client and es-data when we specify memory limit
- plugin install ?
- ES production configuration HOT 2
- ES_PLUGINS_INSTALL breaks boot sequence HOT 2
- Why this project if we have it in an oficial ElasticSearch as Helm package? HOT 5
- Data nodes failing to restart
- es-ingest and es-data not connecting after restarting es-master
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kubernetes-elasticsearch-cluster.