zookeeper cluster cannot communicate with each other about seldon-server HOT 6 CLOSED

Hi,

Can you let us know how you are running kubernetes.
Are you using minikube or another method?

• Gurminder

from seldon-server.

Hi Gurminder,
I think my kubernet is not the minikube
I am running kubernets like these(one mater, and one minion, only using minion node to run docker containers):
master node:
(flannel network 172.17.0.0/16)
etcd, kube-apiserver, kube-controller-manager, kube-scheduler, and so on
minion node:
kube-proxy, kubelet, flannel(flannel network 172.17.28.0/24 (subnet), docker0: 172.17.28.1)
when running seldon-up.sh the services and pods created successfully,
there are three zookeeper pods, zookeeper-1, zookeeper-2, zookeeper-3
take zookeeper-1 for example, its cluster ip is 10.254.168.191 and endpoint address is 172.17.28.3
run cmd on master : kubectl exec zookeeper-1 -- bash -c "echo srvr | nc 172.17.28.3 2181"
I can get the message "This ZooKeeper instance is not currently serving requests"
on the minion node, I run the command, echo srvr | nc 10.254.168.191 2181 also get the reply
"This ZooKeeper instance is not currently serving requests"
this means that I can access the service via kube-proxy.
the problem is:
when I run command "kubectl exec zookeeper-2 -- bash -c "echo srvr | nc 10.254.168.191 2181"
there is no reply, this means that in zookeeper-2 cannot access the address 10.254.168.191
when I run command "kubectl exec zookeeper-2 -- bash -c "echo srvr | nc zookeeper-1 2181"
reply "cannot found hostname"
SO, my question is how can the zookeeper node communicate with each other?
Does there some iptables rule needed?
Another question, which seldon pods communicate with each other using kube-dns or using environment variable?

can you give me some guidethat you installed your kubernets?
thanks very much

from seldon-server.

Hi,

This looks like a dns issue.
The zookeeper servers are unable to find each other.

Latest version of kubernetes should have the dns service built in.
see http://kubernetes.io/docs/admin/dns/

Which version of kubernetes are you using?

from seldon-server.

Hi Gurminder,
thanks for you reply, I am using kubernets v1.2.0
Client Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.0", GitCommit:"ec7364b6e3b155e78086018aa644057edbe196e5", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.0", GitCommit:"ec7364b6e3b155e78086018aa644057edbe196e5", GitTreeState:"clean"}

Indeed, there are two problems, cannot know hostname might be dns problem, but when I using cluster ip address, still not work, there might be network routing problems.

By the way, the kube-dns is required by seldon, right?
Thanks very much again~

from seldon-server.

Hi,

kubernets v1.2.0 would not have the dns built in, you would to manually add that.

It would be best to use the latest version of kubernetes or at least v1.3.

from seldon-server.

@gsunner
Hi Gurminder,
Thank you very much for your help, I have fix the problem that zookeepers cannot communicate with each other by services ip address. Just add some rules in iptables can resolve it. It's the configuration problem of k8s. The only problem is kube-dns now, thanks

from seldon-server.