Comments (15)
yprokule
commented on August 23, 2024
@AsherShoshan the same error on all three masters?
from dev-scripts.
karmab
commented on August 23, 2024
i see the following in the etcd-member container
[root@kni1-c13u23 ~]# crictl logs 6445797c307d7
2019-04-15 08:39:58.065116 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd
2019-04-15 08:39:58.065235 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd-member-kni1-c13u23.cloud.lab.eng.bos.redhat.com
2019-04-15 08:39:58.065281 I | etcdmain: etcd Version: 3.3.10
2019-04-15 08:39:58.065285 I | etcdmain: Git SHA: a0e62b4
2019-04-15 08:39:58.065288 I | etcdmain: Go Version: go1.10.3
2019-04-15 08:39:58.065290 I | etcdmain: Go OS/Arch: linux/amd64
2019-04-15 08:39:58.065294 I | etcdmain: setting maximum number of CPUs to 56, total number of available CPUs is 56
2019-04-15 08:39:58.065381 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2019-04-15 08:39:58.065409 I | embed: peerTLS: cert = /etc/ssl/etcd/system:etcd-peer:etcd-c13u23.kni1.cloud.lab.eng.bos.redhat.com.crt, key = /etc/ssl/etcd/system:etcd-peer:etcd-c13u23.kni1.cloud.lab.eng.bos.redhat.com.key, ca = , trusted-ca = /etc/ssl/etcd/ca.crt, client-cert-auth = true, crl-file =
2019-04-15 08:39:58.083361 I | embed: listening for peers on https://0.0.0.0:2380
2019-04-15 08:39:58.083459 I | embed: listening for client requests on 0.0.0.0:2379
2019-04-15 08:39:58.576308 I | etcdserver: recovered store from snapshot at index 3500036
2019-04-15 08:39:58.576476 I | mvcc: restore compact to 3189759
2019-04-15 08:39:58.616289 I | etcdserver: name = etcd-member-kni1-c13u23.cloud.lab.eng.bos.redhat.com
2019-04-15 08:39:58.616315 I | etcdserver: data dir = /var/lib/etcd
2019-04-15 08:39:58.616334 I | etcdserver: member dir = /var/lib/etcd/member
2019-04-15 08:39:58.616337 I | etcdserver: heartbeat = 100ms
2019-04-15 08:39:58.616341 I | etcdserver: election = 1000ms
2019-04-15 08:39:58.616344 I | etcdserver: snapshot count = 100000
2019-04-15 08:39:58.616356 I | etcdserver: advertise client URLs = https://10.19.138.11:2379
2019-04-15 08:39:59.819216 I | etcdserver: restarting member 18647d3de1022d1f in cluster 7d05ee670e95b1f6 at commit index 3570264
2019-04-15 08:39:59.828661 I | raft: 18647d3de1022d1f became follower at term 2
2019-04-15 08:39:59.828735 I | raft: newRaft 18647d3de1022d1f [peers: [18647d3de1022d1f,25b94689aa475203,e81accfec2d09820], term: 2, commit: 3570264, applied: 3500036, lastindex: 3570266, lastterm: 2]
2019-04-15 08:39:59.828934 I | etcdserver/api: enabled capabilities for version 3.3
2019-04-15 08:39:59.828969 I | etcdserver/membership: added member 18647d3de1022d1f [https://etcd-c13u23.kni1.cloud.lab.eng.bos.redhat.com:2380] to cluster 7d05ee670e95b1f6 from store
2019-04-15 08:39:59.829520 I | etcdserver/membership: added member 25b94689aa475203 [https://etcd-c13u27.kni1.cloud.lab.eng.bos.redhat.com:2380] to cluster 7d05ee670e95b1f6 from store
2019-04-15 08:39:59.829558 I | etcdserver/membership: added member e81accfec2d09820 [https://etcd-c13u25.kni1.cloud.lab.eng.bos.redhat.com:2380] to cluster 7d05ee670e95b1f6 from store
2019-04-15 08:39:59.829578 I | etcdserver/membership: set the cluster version to 3.3 from store
2019-04-15 08:39:59.830840 I | mvcc: restore compact to 3189759
2019-04-15 08:39:59.869860 W | auth: simple token is not cryptographically signed
2019-04-15 08:39:59.871048 I | rafthttp: starting peer 25b94689aa475203...
2019-04-15 08:39:59.871075 I | rafthttp: started HTTP pipelining with peer 25b94689aa475203
2019-04-15 08:39:59.871341 I | rafthttp: started streaming with peer 25b94689aa475203 (writer)
2019-04-15 08:39:59.871896 I | rafthttp: started streaming with peer 25b94689aa475203 (writer)
2019-04-15 08:39:59.872076 I | rafthttp: started peer 25b94689aa475203
2019-04-15 08:39:59.872092 I | rafthttp: added peer 25b94689aa475203
2019-04-15 08:39:59.872100 I | rafthttp: starting peer e81accfec2d09820...
2019-04-15 08:39:59.872110 I | rafthttp: started HTTP pipelining with peer e81accfec2d09820
2019-04-15 08:39:59.872160 I | rafthttp: started streaming with peer 25b94689aa475203 (stream MsgApp v2 reader)
2019-04-15 08:39:59.872331 I | rafthttp: started streaming with peer 25b94689aa475203 (stream Message reader)
2019-04-15 08:39:59.872371 I | rafthttp: started streaming with peer e81accfec2d09820 (writer)
2019-04-15 08:39:59.872522 I | rafthttp: started streaming with peer e81accfec2d09820 (writer)
2019-04-15 08:39:59.872758 I | rafthttp: started peer e81accfec2d09820
2019-04-15 08:39:59.872787 I | rafthttp: added peer e81accfec2d09820
2019-04-15 08:39:59.872809 I | etcdserver: starting server... [version: 3.3.10, cluster version: 3.3]
2019-04-15 08:39:59.872849 I | rafthttp: started streaming with peer e81accfec2d09820 (stream MsgApp v2 reader)
2019-04-15 08:39:59.873028 I | rafthttp: started streaming with peer e81accfec2d09820 (stream Message reader)
2019-04-15 08:39:59.888159 I | embed: ClientTLS: cert = /etc/ssl/etcd/system:etcd-server:etcd-c13u23.kni1.cloud.lab.eng.bos.redhat.com.crt, key = /etc/ssl/etcd/system:etcd-server:etcd-c13u23.kni1.cloud.lab.eng.bos.redhat.com.key, ca = , trusted-ca = /etc/ssl/etcd/ca.crt, client-cert-auth = true, crl-file =
2019-04-15 08:39:59.890783 I | embed: listening for metrics on https://0.0.0.0:9978
2019-04-15 08:40:00.786122 I | rafthttp: peer 25b94689aa475203 became active
2019-04-15 08:40:00.786141 I | rafthttp: established a TCP streaming connection with peer 25b94689aa475203 (stream MsgApp v2 writer)
2019-04-15 08:40:00.786255 I | rafthttp: established a TCP streaming connection with peer 25b94689aa475203 (stream Message writer)
2019-04-15 08:40:00.787367 I | rafthttp: established a TCP streaming connection with peer 25b94689aa475203 (stream MsgApp v2 reader)
2019-04-15 08:40:00.787475 I | rafthttp: established a TCP streaming connection with peer 25b94689aa475203 (stream Message reader)
2019-04-15 08:40:01.629932 I | raft: 18647d3de1022d1f is starting a new election at term 2
2019-04-15 08:40:01.629951 I | raft: 18647d3de1022d1f became candidate at term 3
2019-04-15 08:40:01.629980 I | raft: 18647d3de1022d1f received MsgVoteResp from 18647d3de1022d1f at term 3
2019-04-15 08:40:01.629990 I | raft: 18647d3de1022d1f [logterm: 2, index: 3570266] sent MsgVote request to 25b94689aa475203 at term 3
2019-04-15 08:40:01.629996 I | raft: 18647d3de1022d1f [logterm: 2, index: 3570266] sent MsgVote request to e81accfec2d09820 at term 3
2019-04-15 08:40:02.223503 I | raft: 18647d3de1022d1f [term: 3] received a MsgVote message with higher term from 25b94689aa475203 [term: 6]
2019-04-15 08:40:02.223522 I | raft: 18647d3de1022d1f became follower at term 6
2019-04-15 08:40:02.223543 I | raft: 18647d3de1022d1f [logterm: 2, index: 3570266, vote: 0] cast MsgVote for 25b94689aa475203 [logterm: 2, index: 3570271] at term 6
2019-04-15 08:40:02.224402 I | raft: raft.node: 18647d3de1022d1f elected leader 25b94689aa475203 at term 6
2019-04-15 08:40:02.226746 I | etcdserver: published {Name:etcd-member-kni1-c13u23.cloud.lab.eng.bos.redhat.com ClientURLs:[https://10.19.138.11:2379]} to cluster 7d05ee670e95b1f6
2019-04-15 08:40:02.226764 I | embed: ready to serve client requests
2019-04-15 08:40:02.228477 I | embed: serving client requests on [::]:2379
2019-04-15 08:40:02.236510 I | embed: rejected connection from "127.0.0.1:53050" (error "tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage", ServerName "")
WARNING: 2019/04/15 08:40:02 Failed to dial 0.0.0.0:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.
2019-04-15 08:40:04.872213 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.071570082s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:40:04.872323 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.071897182s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:40:04.872904 W | rafthttp: health check for peer e81accfec2d09820 could not connect: dial tcp 10.19.138.12:2380: connect: connection refused (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:40:04.873093 W | rafthttp: health check for peer e81accfec2d09820 could not connect: dial tcp 10.19.138.12:2380: connect: connection refused (prober "ROUND_TRIPPER_RAFT_MESSAGE")
proto: no coders for int
proto: no encoder for ValueSize int [GetProperties]
2019-04-15 08:40:09.873107 W | rafthttp: health check for peer e81accfec2d09820 could not connect: dial tcp 10.19.138.12:2380: connect: connection refused (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:40:09.873234 W | rafthttp: health check for peer e81accfec2d09820 could not connect: dial tcp 10.19.138.12:2380: connect: connection refused (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:40:14.873259 W | rafthttp: health check for peer e81accfec2d09820 could not connect: dial tcp 10.19.138.12:2380: connect: connection refused (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:40:14.873375 W | rafthttp: health check for peer e81accfec2d09820 could not connect: dial tcp 10.19.138.12:2380: connect: connection refused (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:40:16.124143 I | rafthttp: peer e81accfec2d09820 became active
2019-04-15 08:40:16.124162 I | rafthttp: established a TCP streaming connection with peer e81accfec2d09820 (stream MsgApp v2 writer)
2019-04-15 08:40:16.125778 I | rafthttp: established a TCP streaming connection with peer e81accfec2d09820 (stream Message reader)
2019-04-15 08:40:16.125857 I | rafthttp: established a TCP streaming connection with peer e81accfec2d09820 (stream MsgApp v2 reader)
2019-04-15 08:40:16.127395 I | rafthttp: established a TCP streaming connection with peer e81accfec2d09820 (stream Message writer)
2019-04-15 08:40:34.872408 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.068486619s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:40:34.872488 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.071905746s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:41:04.872576 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.066731429s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:41:04.872609 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.071900346s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:41:34.872677 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.065928798s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:41:34.872732 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.071717937s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:42:04.872827 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.071604205s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:42:04.872880 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.065188594s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:42:34.873003 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.071479701s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:42:34.873036 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.065424216s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:43:04.873157 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.071181654s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:43:04.873196 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.065360827s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:43:34.873327 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.071059639s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:43:34.873361 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.065111344s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:44:04.873469 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.064696469s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:44:04.873509 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.070874615s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:44:34.873708 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.064469707s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:44:34.873783 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.070780381s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:45:04.873966 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.070471381s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:45:04.874016 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.063941151s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:45:34.874173 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.070311869s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:45:34.874221 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.063936396s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:46:04.874347 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.070134859s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:46:04.874392 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.063524894s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:46:34.874525 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.069999707s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:46:34.874564 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.063618274s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:47:04.874728 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.069738076s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:47:04.874753 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.063165658s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:47:34.874866 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.069584105s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:47:34.874901 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.063204683s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:48:04.875076 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.069413769s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:48:04.875126 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.062971106s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:48:34.875278 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.068712209s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:48:34.875318 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.062869921s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:49:04.875442 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.068842277s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:49:04.875481 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.062606007s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:49:34.875680 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.068778791s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:49:34.875737 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.062149977s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:50:03.985989 I | mvcc: store.index: compact 3194942
2019-04-15 08:50:04.100873 I | mvcc: finished scheduled compaction at 3194942 (took 111.859245ms)
2019-04-15 08:50:04.875887 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.062473087s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:50:04.875911 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.068645725s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:50:34.876057 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.068467244s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:50:34.876100 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.061929695s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:51:04.876218 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.068299462s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:51:04.876258 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.061953622s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:51:34.876418 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.068109646s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:51:34.876455 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.061713216s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:52:04.876677 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.061597664s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:52:04.876740 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.068029616s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
2019-04-15 08:52:34.876886 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.061103244s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2019-04-15 08:52:34.876935 W | rafthttp: the clock difference against peer 25b94689aa475203 is too high [1.067724035s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
from dev-scripts.
russellb
commented on August 23, 2024
I logged into Karim's bare metal cluster and see this in logs:
Apr 15 15:30:18 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[27970]: E0415 15:30:18.793519 27970 kubelet.go:2273] node "kni1-c13u27.cloud.lab.eng.bos.redhat.com" not found
Apr 15 15:30:18 kni1-c13u27.cloud.lab.eng.bos.redhat.com hyperkube[27970]: E0415 15:30:18.852805 27970 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: nodes "kni1-c13u27.cloud.lab.eng.bos.redhat.com" is forbidden: User "system:anonymous" cannot list resource "nodes" in API group "" at the cluster scope
from dev-scripts.
russellb
commented on August 23, 2024
definitely need to figure out why time is out of sync on the 3 masters ...
$ for n in 11 12 13 ; do ssh [email protected].$n date ; done
Mon Apr 15 15:57:07 UTC 2019
Mon Apr 15 15:57:31 UTC 2019
Mon Apr 15 15:57:07 UTC 2019
from dev-scripts.
mcornea
commented on August 23, 2024
@russellb it's worth checking if the ntp server set in /etc/chrony.conf is reachable. I know that on the internal network NTP traffic is only allowed from specific hosts(the same query against clock.redhat.com works fine):
[centos@provisionhost-0 ~]$ sudo nmap -sU -p 123 --script ntp-info 2.rhel.pool.ntp.org
Starting Nmap 6.40 ( http://nmap.org ) at 2019-04-15 16:29 UTC
Nmap scan report for 2.rhel.pool.ntp.org (199.102.46.78)
Host is up (0.00036s latency).
Other addresses for 2.rhel.pool.ntp.org (not scanned): 107.155.79.108 192.111.144.114 198.50.238.156
rDNS record for 199.102.46.78: tock.usshc.com
PORT STATE SERVICE
123/udp open|filtered ntp
Nmap done: 1 IP address (1 host up) scanned in 5.72 seconds
from dev-scripts.
russellb
commented on August 23, 2024
Thanks @mcornea. This cluster is on the internal network ... sigh. We'll need to customize the ntp server on our deployed nodes.
from dev-scripts.
russellb
commented on August 23, 2024
I'll have a patch shortly to fix the NTP problem.
from dev-scripts.
gklein
commented on August 23, 2024
Example for chrony configuration using MCO
from dev-scripts.
karmab
commented on August 23, 2024
despite having ntp correctly set, i still got a notready node after several hours. here's relevant section of kubelet log
Apr 16 05:12:23 master-1 hyperkube[890]: E0416 05:12:23.087223 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://api.ostest.test.metalkube.org:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: connection refused
Apr 16 05:12:23 master-1 hyperkube[890]: E0416 05:12:23.089477 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api.ostest.test.metalkube.org:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: connection refused
Apr 16 05:12:23 master-1 hyperkube[890]: E0416 05:12:23.090543 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: connection refused
Apr 16 05:12:23 master-1 hyperkube[890]: W0416 05:12:23.258365 890 pod_container_deletor.go:75] Container "0cc98e2c027923871c8267cf8b23464b25f7677e13d6a33795482d072e3410f2" not found in pod's containers
Apr 16 05:12:23 master-1 hyperkube[890]: W0416 05:12:23.261207 890 pod_container_deletor.go:75] Container "68c51ccce953a8a4afd138fea01e3f7d5f5aca229dd6b6b547fae3bd245308dc" not found in pod's containers
Apr 16 05:12:23 master-1 hyperkube[890]: W0416 05:12:23.267178 890 status_manager.go:485] Failed to get status for pod "kube-apiserver-master-1_openshift-kube-apiserver(c6e984e30d079cf4d2c8794543a8a53f)": Get https://api.ostest.test.metalkube.org:6443/api/v1/namespaces/openshift-kube-apiserver/pods/kube-apiserver-master-1: dial tcp 192.168.111.5:6443: connect: connection refused
Apr 16 05:12:23 master-1 hyperkube[890]: E0416 05:12:23.269686 890 event.go:212] Unable to write event: 'Post https://api.ostest.test.metalkube.org:6443/api/v1/namespaces/openshift-kube-apiserver/events: dial tcp 192.168.111.5:6443: connect: connection refused' (may retry after sleeping)
Apr 16 05:12:23 master-1 hyperkube[890]: W0416 05:12:23.445595 890 kubelet_getters.go:284] Path "/var/lib/kubelet/pods/6f41c3805ace0aeedb0cffc142bbf6fd/volumes" does not exist
Apr 16 05:12:23 master-1 hyperkube[890]: E0416 05:12:23.513334 890 helpers.go:137] readString: Failed to read "/sys/fs/cgroup/cpu,cpuacct/libcontainer_34072_systemd_test_default.slice/cpu.shares": read /sys/fs/cgroup/cpu,cpuacct/libcontainer_34072_systemd_test_default.slice/cpu.shares: no such device
Apr 16 05:12:23 master-1 hyperkube[890]: W0416 05:12:23.515503 890 container.go:523] Failed to update stats for container "/libcontainer_34072_systemd_test_default.slice": open /sys/fs/cgroup/memory/libcontainer_34072_systemd_test_default.slice/memory.use_hierarchy: no such file or directory, continuing to push stats
Apr 16 05:12:23 master-1 hyperkube[890]: E0416 05:12:23.761777 890 event.go:212] Unable to write event: 'Post https://api.ostest.test.metalkube.org:6443/api/v1/namespaces/openshift-kube-apiserver/events: dial tcp 192.168.111.5:6443: connect: connection refused' (may retry after sleeping)
Apr 16 05:12:24 master-1 hyperkube[890]: E0416 05:12:24.088512 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://api.ostest.test.metalkube.org:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: connection refused
Apr 16 05:12:24 master-1 hyperkube[890]: E0416 05:12:24.093147 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: connection refused
Apr 16 05:12:24 master-1 hyperkube[890]: E0416 05:12:24.093147 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api.ostest.test.metalkube.org:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: connection refused
Apr 16 05:12:24 master-1 hyperkube[890]: W0416 05:12:24.274642 890 status_manager.go:485] Failed to get status for pod "kube-apiserver-master-1_openshift-kube-apiserver(c6e984e30d079cf4d2c8794543a8a53f)": Get https://api.ostest.test.metalkube.org:6443/api/v1/namespaces/openshift-kube-apiserver/pods/kube-apiserver-master-1: dial tcp 192.168.111.5:6443: connect: connection refused
Apr 16 05:12:26 master-1 hyperkube[890]: E0416 05:12:26.603905 890 kubelet_node_status.go:380] Error updating node status, will retry: error getting node "master-1": Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes/master-1?resourceVersion=0&timeout=10s: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:26 master-1 hyperkube[890]: E0416 05:12:26.611051 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0: read tcp 192.168.111.5:48720->192.168.111.5:6443: use of closed network connection
Apr 16 05:12:26 master-1 hyperkube[890]: E0416 05:12:26.611101 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api.ostest.test.metalkube.org:6443/api/v1/services?limit=500&resourceVersion=0: read tcp 192.168.111.5:48722->192.168.111.5:6443: use of closed network connection
Apr 16 05:12:26 master-1 hyperkube[890]: E0416 05:12:26.611139 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://api.ostest.test.metalkube.org:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster-1&limit=500&resourceVersion=0: read tcp 192.168.111.5:48714->192.168.111.5:6443: use of closed network connection
Apr 16 05:12:26 master-1 hyperkube[890]: W0416 05:12:26.611252 890 status_manager.go:485] Failed to get status for pod "kube-apiserver-master-1_openshift-kube-apiserver(c6e984e30d079cf4d2c8794543a8a53f)": Get https://api.ostest.test.metalkube.org:6443/api/v1/namespaces/openshift-kube-apiserver/pods/kube-apiserver-master-1: read tcp 192.168.111.5:48792->192.168.111.5:6443: use of closed network connection
Apr 16 05:12:26 master-1 hyperkube[890]: E0416 05:12:26.611493 890 kubelet_node_status.go:380] Error updating node status, will retry: error getting node "master-1": Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes/master-1?timeout=10s: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:26 master-1 hyperkube[890]: E0416 05:12:26.612276 890 kubelet_node_status.go:380] Error updating node status, will retry: error getting node "master-1": Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes/master-1?timeout=10s: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:26 master-1 hyperkube[890]: E0416 05:12:26.613475 890 kubelet_node_status.go:380] Error updating node status, will retry: error getting node "master-1": Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes/master-1?timeout=10s: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:26 master-1 hyperkube[890]: E0416 05:12:26.614950 890 kubelet_node_status.go:380] Error updating node status, will retry: error getting node "master-1": Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes/master-1?timeout=10s: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:26 master-1 hyperkube[890]: E0416 05:12:26.614976 890 kubelet_node_status.go:367] Unable to update node status: update node status exceeds retry count
Apr 16 05:12:27 master-1 hyperkube[890]: E0416 05:12:27.612557 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:27 master-1 hyperkube[890]: E0416 05:12:27.613045 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api.ostest.test.metalkube.org:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:27 master-1 hyperkube[890]: E0416 05:12:27.614106 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://api.ostest.test.metalkube.org:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:28 master-1 hyperkube[890]: E0416 05:12:28.613907 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:28 master-1 hyperkube[890]: E0416 05:12:28.614619 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api.ostest.test.metalkube.org:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:28 master-1 hyperkube[890]: E0416 05:12:28.615566 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://api.ostest.test.metalkube.org:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:29 master-1 hyperkube[890]: E0416 05:12:29.615623 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api.ostest.test.metalkube.org:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:29 master-1 hyperkube[890]: E0416 05:12:29.615671 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:30 master-1 hyperkube[890]: E0416 05:12:30.617198 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api.ostest.test.metalkube.org:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:30 master-1 hyperkube[890]: E0416 05:12:30.618089 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:30 master-1 hyperkube[890]: E0416 05:12:30.620218 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://api.ostest.test.metalkube.org:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:31 master-1 hyperkube[890]: W0416 05:12:31.397220 890 status_manager.go:485] Failed to get status for pod "kube-apiserver-master-1_openshift-kube-apiserver(c6e984e30d079cf4d2c8794543a8a53f)": Get https://api.ostest.test.metalkube.org:6443/api/v1/namespaces/openshift-kube-apiserver/pods/kube-apiserver-master-1: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:31 master-1 hyperkube[890]: E0416 05:12:31.618554 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api.ostest.test.metalkube.org:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:31 master-1 hyperkube[890]: E0416 05:12:31.619155 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:31 master-1 hyperkube[890]: E0416 05:12:31.621178 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://api.ostest.test.metalkube.org:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:32 master-1 hyperkube[890]: E0416 05:12:32.620594 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:32 master-1 hyperkube[890]: E0416 05:12:32.620733 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api.ostest.test.metalkube.org:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:32 master-1 hyperkube[890]: E0416 05:12:32.622065 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://api.ostest.test.metalkube.org:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:33 master-1 hyperkube[890]: E0416 05:12:33.621996 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:33 master-1 hyperkube[890]: E0416 05:12:33.622489 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api.ostest.test.metalkube.org:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:33 master-1 hyperkube[890]: E0416 05:12:33.623743 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://api.ostest.test.metalkube.org:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:33 master-1 hyperkube[890]: E0416 05:12:33.762989 890 event.go:212] Unable to write event: 'Post https://api.ostest.test.metalkube.org:6443/api/v1/namespaces/openshift-kube-apiserver/events: dial tcp 192.168.111.5:6443: connect: invalid argument' (may retry after sleeping)
Apr 16 05:12:34 master-1 hyperkube[890]: E0416 05:12:34.623306 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:34 master-1 hyperkube[890]: E0416 05:12:34.623913 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api.ostest.test.metalkube.org:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:34 master-1 hyperkube[890]: E0416 05:12:34.625007 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://api.ostest.test.metalkube.org:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:35 master-1 hyperkube[890]: E0416 05:12:35.624862 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:35 master-1 hyperkube[890]: E0416 05:12:35.624960 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api.ostest.test.metalkube.org:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:35 master-1 hyperkube[890]: E0416 05:12:35.626284 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://api.ostest.test.metalkube.org:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:36 master-1 hyperkube[890]: W0416 05:12:36.350276 890 status_manager.go:485] Failed to get status for pod "openshift-controller-manager-operator-69fdd65b49-cl48d_openshift-controller-manager-operator(cbca754c-5f66-11e9-ae76-ae49b64b2b03)": Get https://api.ostest.test.metalkube.org:6443/api/v1/namespaces/openshift-controller-manager-operator/pods/openshift-controller-manager-operator-69fdd65b49-cl48d: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:36 master-1 hyperkube[890]: E0416 05:12:36.616313 890 kubelet_node_status.go:380] Error updating node status, will retry: error getting node "master-1": Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes/master-1?resourceVersion=0&timeout=10s: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:36 master-1 hyperkube[890]: E0416 05:12:36.617058 890 kubelet_node_status.go:380] Error updating node status, will retry: error getting node "master-1": Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes/master-1?timeout=10s: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:36 master-1 hyperkube[890]: E0416 05:12:36.617806 890 kubelet_node_status.go:380] Error updating node status, will retry: error getting node "master-1": Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes/master-1?timeout=10s: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:36 master-1 hyperkube[890]: E0416 05:12:36.618695 890 kubelet_node_status.go:380] Error updating node status, will retry: error getting node "master-1": Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes/master-1?timeout=10s: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:36 master-1 hyperkube[890]: E0416 05:12:36.619503 890 kubelet_node_status.go:380] Error updating node status, will retry: error getting node "master-1": Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes/master-1?timeout=10s: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:36 master-1 hyperkube[890]: E0416 05:12:36.619522 890 kubelet_node_status.go:367] Unable to update node status: update node status exceeds retry count
Apr 16 05:12:36 master-1 hyperkube[890]: E0416 05:12:36.626034 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:36 master-1 hyperkube[890]: E0416 05:12:36.627489 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api.ostest.test.metalkube.org:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:36 master-1 hyperkube[890]: E0416 05:12:36.627559 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://api.ostest.test.metalkube.org:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:37 master-1 hyperkube[890]: E0416 05:12:37.628445 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:37 master-1 hyperkube[890]: E0416 05:12:37.628445 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api.ostest.test.metalkube.org:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:37 master-1 hyperkube[890]: E0416 05:12:37.630106 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://api.ostest.test.metalkube.org:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:38 master-1 hyperkube[890]: E0416 05:12:38.629781 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
Apr 16 05:12:38 master-1 hyperkube[890]: E0416 05:12:38.630397 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api.ostest.test.metalkube.org:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
from dev-scripts.
yprokule
commented on August 23, 2024
despite having ntp correctly set, i still got a notready node after several hours. here's relevant section of kubelet log
Apr 16 05:12:37 master-1 hyperkube[890]: E0416 05:12:37.630106 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://api.ostest.test.metalkube.org:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument Apr 16 05:12:38 master-1 hyperkube[890]: E0416 05:12:38.629781 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api.ostest.test.metalkube.org:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument Apr 16 05:12:38 master-1 hyperkube[890]: E0416 05:12:38.630397 890 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api.ostest.test.metalkube.org:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.111.5:6443: connect: invalid argument
That's what is investigated in #344
Hence - can we use #354 to track NTP related stuff and #344 for connect: invalid argument issue?
from dev-scripts.
karmab
commented on August 23, 2024
i suggest closing this as @russellb submitted a PR for using specific ntp servers
from dev-scripts.
hardys
commented on August 23, 2024
#364 merged, so can we close this now?
from dev-scripts.
gklein
commented on August 23, 2024
I didn't see this issue on my cluster for 35 hours since applying this PR. I'm fine with closing.
from dev-scripts.
bengland2
commented on August 23, 2024
I think this fix doesn't solve the problem for Ceph, because it uses clock.redhat.com. I've documented this in the attachment here. Ceph and NTP clock skew _ Mojo.pdf. I don't know of an automated solution, does anyone?
from dev-scripts.
e-minguez
commented on August 23, 2024
I think this fix doesn't solve the problem for Ceph, because it uses clock.redhat.com. I've documented this in the attachment here. Ceph and NTP clock skew _ Mojo.pdf. I don't know of an automated solution, does anyone?
I believe this will fix it #479 as it allows the user to customize the NTP servers.
from dev-scripts.
Related Issues (20)
- A cluster name that exceeds 14 characters causes an error during bridge creation HOT 5
- error TASK [libvirt : Create libvirt networks] HOT 5
- Deployment throws error with config OPENSHIFT_VERSION=4.5.0
- FATAL failed to fetch Master Machines: failed to load asset "Install Config": invalid "install-config.yaml" HOT 3
- Error when chown runs and user name differs from group name
- Use openshift-install coreos print-stream-json HOT 3
- Allow creating a custom number of extra disks HOT 2
- Align Ansible version to metal3-dev-env one HOT 2
- libvirt net ostestbm won't start after reboot HOT 2
- make agent fails with error AGENT_E2E_TEST_SCENARIO is missing
- lxml install needs devel packages for arm64
- The Cluster creation fails with Error: could not contact Ironic API: context deadline exceeded HOT 6
- openshift-sdn support is about to be removed from the installer
- Wrong command in ./01_install_requirements.sh script HOT 6
- Make sure NetworkManager-initscripts-updown is installed on RHEL9 host.
- Failure to create dualstack v6 primary cluster
- Error: failed to dial libvirt: dial unix /var/run/libvirt/libvirt-sock: connect: no such file or directory HOT 3
- 01_requirements.sh errors when `go` is pre-installed.
- Suspect yq doesn't always successfully install on 01_install_requirements.sh L100 HOT 3
- RHEL9: libvirt-sock file not found error during cluster bringup
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dev-scripts.