Hi All,
I am trying to deploy OpenSearch in kubernetes environment.
below is the helm chart I am using it,
https://github.com/opendistro-for-elasticsearch/opendistro-build/tree/main/helm/opendistro-es
Helm version
[LRB_346_PCAA@k8s-rmp-master-0 ~]$ helm3 version
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/LRB_346_PCAA/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/LRB_346_PCAA/.kube/config
version.BuildInfo{Version:"v3.3.4", GitCommit:"a61ce5633af99708171414353ed49547cf05013d", GitTreeState:"clean", GoVersion:"go1.14.9"}
Kubernetes version
[LRB_346_PCAA@k8s-rmp-master-0 ~]$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-18T01:10:43Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-18T01:02:01Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
I have changed the images of kibana and elasticsearch in values.yaml and I used the opensearch 1.0.0 for elasticsearch and opensearch-dashboards 1.0.0 for kibana.
I have deployed the chart using the below command,
sudo helm3 install elasticsearch ./ -n t253-u000265
below is the pod status,
opendistro-es]$ kubectl get pods -w
NAME READY STATUS RESTARTS AGE
elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8 1/1 Running 0 26s
elasticsearch-opendistro-es-data-0 1/1 Running 0 26s
elasticsearch-opendistro-es-kibana-77cb457bf7-hsj8n 1/1 Running 0 26s
elasticsearch-opendistro-es-master-0 1/1 Running 0 26s
But when I checked the client pod logs below is the error,
[2021-09-23T18:49:45,018][INFO ][o.o.b.BootstrapChecks ] [elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2021-09-23T18:49:57,848][WARN ][o.o.d.HandshakingTransportAddressConnector] [elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:449b]:9300]] completed handshake with [{elasticsearch-opendistro-es-master-0}{3TSgi__ETdOp5ateCv832g}{alO-X1KpQk-ID3XAaTlS1A}{127.0.0.1}{127.0.0.1:9300}{mr}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [elasticsearch-opendistro-es-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8}{jMECGhb0RrKGeYhk7tmoqQ}{gw8c0vhXTT-FLbha8zr1UA}{127.0.0.1}{127.0.0.1:9300}{ir}
at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:405) ~[opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:170) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:492) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:482) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:288) [opensearch-security-1.0.0.0.jar:1.0.0.0]
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1207) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:266) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:260) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:697) [opensearch-1.0.0.jar:1.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-23T18:49:58,394][WARN ][o.o.d.HandshakingTransportAddressConnector] [elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:449b]:9300]] completed handshake with [{elasticsearch-opendistro-es-master-0}{3TSgi__ETdOp5ateCv832g}{alO-X1KpQk-ID3XAaTlS1A}{127.0.0.1}{127.0.0.1:9300}{mr}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [elasticsearch-opendistro-es-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8}{jMECGhb0RrKGeYhk7tmoqQ}{gw8c0vhXTT-FLbha8zr1UA}{127.0.0.1}{127.0.0.1:9300}{ir}
at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:405) ~[opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:170) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:492) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:482) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:288) [opensearch-security-1.0.0.0.jar:1.0.0.0]
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1207) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:266) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:260) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:697) [opensearch-1.0.0.jar:1.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-23T18:49:59,394][WARN ][o.o.d.HandshakingTransportAddressConnector] [elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:449b]:9300]] completed handshake with [{elasticsearch-opendistro-es-master-0}{3TSgi__ETdOp5ateCv832g}{alO-X1KpQk-ID3XAaTlS1A}{127.0.0.1}{127.0.0.1:9300}{mr}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [elasticsearch-opendistro-es-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8}{jMECGhb0RrKGeYhk7tmoqQ}{gw8c0vhXTT-FLbha8zr1UA}{127.0.0.1}{127.0.0.1:9300}{ir}
at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:405) ~[opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:170) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:492) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:482) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:288) [opensearch-security-1.0.0.0.jar:1.0.0.0]
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1207) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:266) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:260) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:697) [opensearch-1.0.0.jar:1.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-23T18:50:00,323][WARN ][o.o.d.HandshakingTransportAddressConnector] [elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:449b]:9300]] completed handshake with [{elasticsearch-opendistro-es-master-0}{3TSgi__ETdOp5ateCv832g}{alO-X1KpQk-ID3XAaTlS1A}{127.0.0.1}{127.0.0.1:9300}{mr}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [elasticsearch-opendistro-es-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8}{jMECGhb0RrKGeYhk7tmoqQ}{gw8c0vhXTT-FLbha8zr1UA}{127.0.0.1}{127.0.0.1:9300}{ir}
at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:405) ~[opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:170) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:492) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:482) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:288) [opensearch-security-1.0.0.0.jar:1.0.0.0]
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1207) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:266) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:260) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:697) [opensearch-1.0.0.jar:1.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-23T18:50:01,318][WARN ][o.o.d.HandshakingTransportAddressConnector] [elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:449b]:9300]] completed handshake with [{elasticsearch-opendistro-es-master-0}{3TSgi__ETdOp5ateCv832g}{alO-X1KpQk-ID3XAaTlS1A}{127.0.0.1}{127.0.0.1:9300}{mr}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [elasticsearch-opendistro-es-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8}{jMECGhb0RrKGeYhk7tmoqQ}{gw8c0vhXTT-FLbha8zr1UA}{127.0.0.1}{127.0.0.1:9300}{ir}
at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:405) ~[opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:170) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:492) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:482) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:288) [opensearch-security-1.0.0.0.jar:1.0.0.0]
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1207) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:266) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:260) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:697) [opensearch-1.0.0.jar:1.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-23T18:50:02,240][WARN ][o.o.d.HandshakingTransportAddressConnector] [elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:449b]:9300]] completed handshake with [{elasticsearch-opendistro-es-master-0}{3TSgi__ETdOp5ateCv832g}{alO-X1KpQk-ID3XAaTlS1A}{127.0.0.1}{127.0.0.1:9300}{mr}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [elasticsearch-opendistro-es-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8}{jMECGhb0RrKGeYhk7tmoqQ}{gw8c0vhXTT-FLbha8zr1UA}{127.0.0.1}{127.0.0.1:9300}{ir}
at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:405) ~[opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:170) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:492) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:482) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:288) [opensearch-security-1.0.0.0.jar:1.0.0.0]
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1207) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:266) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:260) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:697) [opensearch-1.0.0.jar:1.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-23T18:50:03,313][WARN ][o.o.d.HandshakingTransportAddressConnector] [elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:449b]:9300]] completed handshake with [{elasticsearch-opendistro-es-master-0}{3TSgi__ETdOp5ateCv832g}{alO-X1KpQk-ID3XAaTlS1A}{127.0.0.1}{127.0.0.1:9300}{mr}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [elasticsearch-opendistro-es-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8}{jMECGhb0RrKGeYhk7tmoqQ}{gw8c0vhXTT-FLbha8zr1UA}{127.0.0.1}{127.0.0.1:9300}{ir}
at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:405) ~[opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:170) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:492) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:482) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:288) [opensearch-security-1.0.0.0.jar:1.0.0.0]
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1207) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:266) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:260) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:697) [opensearch-1.0.0.jar:1.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-23T18:50:04,310][WARN ][o.o.d.HandshakingTransportAddressConnector] [elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:449b]:9300]] completed handshake with [{elasticsearch-opendistro-es-master-0}{3TSgi__ETdOp5ateCv832g}{alO-X1KpQk-ID3XAaTlS1A}{127.0.0.1}{127.0.0.1:9300}{mr}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [elasticsearch-opendistro-es-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8}{jMECGhb0RrKGeYhk7tmoqQ}{gw8c0vhXTT-FLbha8zr1UA}{127.0.0.1}{127.0.0.1:9300}{ir}
at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:405) ~[opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:170) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:492) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:482) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:288) [opensearch-security-1.0.0.0.jar:1.0.0.0]
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1207) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:266) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:260) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:697) [opensearch-1.0.0.jar:1.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-23T18:50:05,296][WARN ][o.o.d.HandshakingTransportAddressConnector] [elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:449b]:9300]] completed handshake with [{elasticsearch-opendistro-es-master-0}{3TSgi__ETdOp5ateCv832g}{alO-X1KpQk-ID3XAaTlS1A}{127.0.0.1}{127.0.0.1:9300}{mr}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [elasticsearch-opendistro-es-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8}{jMECGhb0RrKGeYhk7tmoqQ}{gw8c0vhXTT-FLbha8zr1UA}{127.0.0.1}{127.0.0.1:9300}{ir}
at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:405) ~[opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:170) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:492) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:482) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:288) [opensearch-security-1.0.0.0.jar:1.0.0.0]
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1207) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:266) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:260) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:697) [opensearch-1.0.0.jar:1.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-23T18:50:06,247][WARN ][o.o.d.HandshakingTransportAddressConnector] [elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:449b]:9300]] completed handshake with [{elasticsearch-opendistro-es-master-0}{3TSgi__ETdOp5ateCv832g}{alO-X1KpQk-ID3XAaTlS1A}{127.0.0.1}{127.0.0.1:9300}{mr}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [elasticsearch-opendistro-es-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8}{jMECGhb0RrKGeYhk7tmoqQ}{gw8c0vhXTT-FLbha8zr1UA}{127.0.0.1}{127.0.0.1:9300}{ir}
at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:405) ~[opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:170) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:492) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:482) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:288) [opensearch-security-1.0.0.0.jar:1.0.0.0]
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1207) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:266) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:260) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:697) [opensearch-1.0.0.jar:1.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-23T18:50:07,251][WARN ][o.o.d.HandshakingTransportAddressConnector] [elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:449b]:9300]] completed handshake with [{elasticsearch-opendistro-es-master-0}{3TSgi__ETdOp5ateCv832g}{alO-X1KpQk-ID3XAaTlS1A}{127.0.0.1}{127.0.0.1:9300}{mr}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [elasticsearch-opendistro-es-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8}{jMECGhb0RrKGeYhk7tmoqQ}{gw8c0vhXTT-FLbha8zr1UA}{127.0.0.1}{127.0.0.1:9300}{ir}
at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:405) ~[opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:170) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:492) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:482) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:288) [opensearch-security-1.0.0.0.jar:1.0.0.0]
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1207) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:266) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:260) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:697) [opensearch-1.0.0.jar:1.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-23T18:50:08,250][WARN ][o.o.d.HandshakingTransportAddressConnector] [elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:449b]:9300]] completed handshake with [{elasticsearch-opendistro-es-master-0}{3TSgi__ETdOp5ateCv832g}{alO-X1KpQk-ID3XAaTlS1A}{127.0.0.1}{127.0.0.1:9300}{mr}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [elasticsearch-opendistro-es-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8}{jMECGhb0RrKGeYhk7tmoqQ}{gw8c0vhXTT-FLbha8zr1UA}{127.0.0.1}{127.0.0.1:9300}{ir}
at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:405) ~[opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:170) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:492) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:482) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:288) [opensearch-security-1.0.0.0.jar:1.0.0.0]
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1207) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:266) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:260) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:697) [opensearch-1.0.0.jar:1.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-23T18:50:08,304][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [elasticsearch-opendistro-es-client-5dd6c59b74-l6lr8] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:369) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:133) ~[?:?]
at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637) ~[?:?]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:282) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1387) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1282) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1329) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.59.Final.jar:4.1.59.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.59.Final.jar:4.1.59.Final]
at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: javax.crypto.BadPaddingException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
at sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1895) ~[?:?]
at sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:240) ~[?:?]
at sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:197) ~[?:?]
at sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:160) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:110) ~[?:?]
... 27 more
Similarly in data logs I am getting the same error,
[2021-09-23T18:49:56,058][INFO ][o.o.n.Node ] [elasticsearch-opendistro-es-data-0] initialized
[2021-09-23T18:49:56,059][INFO ][o.o.n.Node ] [elasticsearch-opendistro-es-data-0] starting ...
[2021-09-23T18:49:56,066][DEPRECATION][o.o.d.c.s.Settings ] [elasticsearch-opendistro-es-data-0] [node.data] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-23T18:49:56,066][DEPRECATION][o.o.d.c.s.Settings ] [elasticsearch-opendistro-es-data-0] [node.ingest] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-23T18:49:56,066][DEPRECATION][o.o.d.c.s.Settings ] [elasticsearch-opendistro-es-data-0] [node.master] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-23T18:49:56,069][DEPRECATION][o.o.d.c.s.Settings ] [elasticsearch-opendistro-es-data-0] [node.master] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-23T18:49:56,204][INFO ][o.o.t.TransportService ] [elasticsearch-opendistro-es-data-0] publish_address {127.0.0.1:9300}, bound_addresses {[::]:9300}
[2021-09-23T18:49:56,205][DEPRECATION][o.o.d.c.s.Settings ] [elasticsearch-opendistro-es-data-0] [node.data] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-23T18:49:56,205][DEPRECATION][o.o.d.c.s.Settings ] [elasticsearch-opendistro-es-data-0] [node.ingest] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-23T18:49:56,205][DEPRECATION][o.o.d.c.s.Settings ] [elasticsearch-opendistro-es-data-0] [node.master] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-23T18:49:56,208][DEPRECATION][o.o.d.c.s.Settings ] [elasticsearch-opendistro-es-data-0] [node.data] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-23T18:49:56,208][DEPRECATION][o.o.d.c.s.Settings ] [elasticsearch-opendistro-es-data-0] [node.ingest] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-23T18:49:56,208][DEPRECATION][o.o.d.c.s.Settings ] [elasticsearch-opendistro-es-data-0] [node.master] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-23T18:49:56,208][DEPRECATION][o.o.d.c.s.Settings ] [elasticsearch-opendistro-es-data-0] [node.master] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-23T18:49:56,208][DEPRECATION][o.o.d.c.s.Settings ] [elasticsearch-opendistro-es-data-0] [node.data] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-23T18:49:56,209][DEPRECATION][o.o.d.c.s.Settings ] [elasticsearch-opendistro-es-data-0] [node.ingest] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-23T18:49:56,209][DEPRECATION][o.o.d.c.s.Settings ] [elasticsearch-opendistro-es-data-0] [node.master] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-23T18:49:56,221][DEPRECATION][o.o.d.c.s.Settings ] [elasticsearch-opendistro-es-data-0] [node.master] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-23T18:49:56,352][DEPRECATION][o.o.d.c.s.Settings ] [elasticsearch-opendistro-es-data-0] [node.data] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-23T18:49:56,352][DEPRECATION][o.o.d.c.s.Settings ] [elasticsearch-opendistro-es-data-0] [node.ingest] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-23T18:49:56,352][DEPRECATION][o.o.d.c.s.Settings ] [elasticsearch-opendistro-es-data-0] [node.master] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-23T18:49:56,370][INFO ][o.o.b.BootstrapChecks ] [elasticsearch-opendistro-es-data-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2021-09-23T18:49:58,374][WARN ][o.o.d.HandshakingTransportAddressConnector] [elasticsearch-opendistro-es-data-0] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:449b]:9300]] completed handshake with [{elasticsearch-opendistro-es-master-0}{3TSgi__ETdOp5ateCv832g}{alO-X1KpQk-ID3XAaTlS1A}{127.0.0.1}{127.0.0.1:9300}{mr}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [elasticsearch-opendistro-es-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {elasticsearch-opendistro-es-data-0}{KDD8dHGwQ6yhMWtbVi-PFQ}{sxMuB9g2T6eI22diUqKyCA}{127.0.0.1}{127.0.0.1:9300}{dr}
at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:405) ~[opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:170) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:492) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:482) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:288) [opensearch-security-1.0.0.0.jar:1.0.0.0]
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1207) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:266) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:260) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:697) [opensearch-1.0.0.jar:1.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-23T18:49:59,736][WARN ][o.o.d.HandshakingTransportAddressConnector] [elasticsearch-opendistro-es-data-0] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:449b]:9300]] completed handshake with [{elasticsearch-opendistro-es-master-0}{3TSgi__ETdOp5ateCv832g}{alO-X1KpQk-ID3XAaTlS1A}{127.0.0.1}{127.0.0.1:9300}{mr}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [elasticsearch-opendistro-es-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {elasticsearch-opendistro-es-data-0}{KDD8dHGwQ6yhMWtbVi-PFQ}{sxMuB9g2T6eI22diUqKyCA}{127.0.0.1}{127.0.0.1:9300}{dr}
at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:405) ~[opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:170) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:492) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:482) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:288) [opensearch-security-1.0.0.0.jar:1.0.0.0]
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1207) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:266) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:260) [opensearch-1.0.0.jar:1.0.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:697) [opensearch-1.0.0.jar:1.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-23T18:49:59,759][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [elasticsearch-opendistro-es-data-0] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:369) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:133) ~[?:?]
at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637) ~[?:?]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:282) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1387) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1282) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1329) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.59.Final.jar:4.1.59.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.59.Final.jar:4.1.59.Final]
at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: javax.crypto.BadPaddingException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
Below is the master pod logs but I could see cluster state is RED
[2021-09-23T18:49:57,044][INFO ][o.o.c.s.MasterService ] [elasticsearch-opendistro-es-master-0] elected-as-master ([1] nodes joined)[{elasticsearch-opendistro-es-master-0}{3TSgi__ETdOp5ateCv832g}{alO-X1KpQk-ID3XAaTlS1A}{127.0.0.1}{127.0.0.1:9300}{mr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{elasticsearch-opendistro-es-master-0}{3TSgi__ETdOp5ateCv832g}{alO-X1KpQk-ID3XAaTlS1A}{127.0.0.1}{127.0.0.1:9300}{mr}]}
[2021-09-23T18:49:57,091][INFO ][o.o.c.c.CoordinationState] [elasticsearch-opendistro-es-master-0] cluster UUID set to [Di2UaiCBQe2aKWQICmVIwg]
[2021-09-23T18:49:57,138][INFO ][o.o.c.s.ClusterApplierService] [elasticsearch-opendistro-es-master-0] master node changed {previous [], current [{elasticsearch-opendistro-es-master-0}{3TSgi__ETdOp5ateCv832g}{alO-X1KpQk-ID3XAaTlS1A}{127.0.0.1}{127.0.0.1:9300}{mr}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2021-09-23T18:49:57,173][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [elasticsearch-opendistro-es-master-0] Config override setting update called with empty string. Ignoring.
[2021-09-23T18:49:57,220][INFO ][o.o.h.AbstractHttpServerTransport] [elasticsearch-opendistro-es-master-0] publish_address {127.0.0.1:9200}, bound_addresses {[::]:9200}
[2021-09-23T18:49:57,221][INFO ][o.o.n.Node ] [elasticsearch-opendistro-es-master-0] started
[2021-09-23T18:49:57,221][INFO ][o.o.s.OpenSearchSecurityPlugin] [elasticsearch-opendistro-es-master-0] Node started
[2021-09-23T18:49:57,222][INFO ][o.o.s.c.ConfigurationRepository] [elasticsearch-opendistro-es-master-0] Will attempt to create index .opendistro_security and default configs if they are absent
[2021-09-23T18:49:57,223][INFO ][o.o.s.OpenSearchSecurityPlugin] [elasticsearch-opendistro-es-master-0] 0 OpenSearch Security modules loaded so far: []
[2021-09-23T18:49:57,223][INFO ][o.o.s.c.ConfigurationRepository] [elasticsearch-opendistro-es-master-0] Background init thread started. Install default config?: true
[2021-09-23T18:49:57,267][INFO ][o.o.g.GatewayService ] [elasticsearch-opendistro-es-master-0] recovered [0] indices into cluster_state
[2021-09-23T18:49:57,446][INFO ][o.o.c.m.MetadataCreateIndexService] [elasticsearch-opendistro-es-master-0] [.opendistro_security] creating index, cause [api], templates [], shards [1]/[1]
[2021-09-23T18:49:57,461][INFO ][o.o.c.r.a.AllocationService] [elasticsearch-opendistro-es-master-0] Cluster health status changed from [YELLOW] to [RED] (reason: [index [.opendistro_security] created]).
[2021-09-23T18:50:27,512][INFO ][o.o.s.c.ConfigurationRepository] [elasticsearch-opendistro-es-master-0] Index .opendistro_security created?: true
[2021-09-23T18:50:27,513][INFO ][o.o.s.c.ConfigurationRepository] [elasticsearch-opendistro-es-master-0] Node started, try to initialize it. Wait for at least yellow cluster state....
[2021-09-23T18:50:57,169][INFO ][o.o.i.i.ManagedIndexCoordinator] [elasticsearch-opendistro-es-master-0] Performing move cluster state metadata.
[2021-09-23T18:51:57,170][INFO ][o.o.i.i.ManagedIndexCoordinator] [elasticsearch-opendistro-es-master-0] Performing move cluster state metadata.
[2021-09-23T18:52:57,171][INFO ][o.o.i.i.ManagedIndexCoordinator] [elasticsearch-opendistro-es-master-0] Performing move cluster state metadata.
[2021-09-23T18:53:57,171][INFO ][o.o.i.i.ManagedIndexCoordinator] [elasticsearch-opendistro-es-master-0] Performing move cluster state metadata.
[2021-09-23T18:53:57,172][INFO ][o.o.i.i.MetadataService ] [elasticsearch-opendistro-es-master-0] Move Metadata succeed, set finish flag to true. Indices failed to get indexed: {}
[2021-09-23T18:54:56,878][INFO ][o.o.j.s.JobSweeper ] [elasticsearch-opendistro-es-master-0] Running full sweep
[2021-09-23T18:54:57,172][INFO ][o.o.i.i.ManagedIndexCoordinator] [elasticsearch-opendistro-es-master-0] Cancel background move metadata process.
[2021-09-23T18:54:57,172][INFO ][o.o.i.i.ManagedIndexCoordinator] [elasticsearch-opendistro-es-master-0] Performing move cluster state metadata.
[2021-09-23T18:54:57,172][INFO ][o.o.i.i.MetadataService ] [elasticsearch-opendistro-es-master-0] Move metadata has finished.
[2021-09-23T18:59:56,881][INFO ][o.o.j.s.JobSweeper ] [elasticsearch-opendistro-es-master-0] Running full sweep
[2021-09-23T19:04:56,883][INFO ][o.o.j.s.JobSweeper ] [elasticsearch-opendistro-es-master-0] Running full sweep
So I tried exec into the pod to see the cluster status but I am getting the below response,
kubectl exec -it elasticsearch-opendistro-es-master-0 bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[opensearch@elasticsearch-opendistro-es-master-0 ~]$
[opensearch@elasticsearch-opendistro-es-master-0 ~]$ curl -k -u admin:admin https://elasticsearch-opendistro-es-client-service:9200/_cluster/health?pretty=true
OpenSearch Security not initialized.[opensearch@elasticsearch-opendistro-es-master-0 ~]$
Attached the helm package with values.yaml
opendistro-es.zip
Please correct me If I am doing anything wrong.
Thanks,
Ganeshbabu R