openshift / cluster-machine-approver Goto Github PK
View Code? Open in Web Editor NEWThe cluster-machine-approver validates and approves CSRs for nodes attempting to join the cluster.
License: Apache License 2.0
The cluster-machine-approver validates and approves CSRs for nodes attempting to join the cluster.
License: Apache License 2.0
The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.
release-4.17
release-4.18
For more information, see the branching documentation.
The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.
release-4.16
release-4.17
For more information, see the branching documentation.
The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.
release-4.5
release-4.4
Contact the Test Platform or Automated Release teams for more information.
When a node has extra IP on its interface (egressIP), a CSR generated include these extra IP(s) and CSR does not get approved automatically.
cluster-machine-approver/csr_check.go
Lines 314 to 319 in d326f0a
Looks like even after the openshift/cluster-api-provider-libvirt#122 merged I am still see the csr to pending state for worker and master nodes.
$ openshift-install version
openshift-install unreleased-master-556-g4b12088025778add89e95099eaeebb773c39436d
built from commit 4b12088025778add89e95099eaeebb773c39436d
$ oc adm release info --commits
Name: 4.0.0-0.alpha-2019-03-14-014544
Digest: sha256:8cc63b36d9c6c5a7d4aefac787e04cab06532695d082ed9dc78649de8910143e
Created: 2019-03-14 01:45:55 +0000 UTC
OS/Arch: linux/amd64
Manifests: 253
Release Metadata:
Version: 4.0.0-0.alpha-2019-03-14-014544
Upgrades: <none>
Component Versions:
Kubernetes 1.12.4
Images:
NAME REPO COMMIT
aws-machine-controllers https://github.com/openshift/cluster-api-provider-aws 7014df4dbf78e15550ee38eea1a0c71c826e6aff
cli https://github.com/openshift/origin c53f4620a46deacf86ba32126aaecb808929c3b5
cloud-credential-operator https://github.com/openshift/cloud-credential-operator 77b036138a618726f778a918ed4695acfea15716
cluster-authentication-operator https://github.com/openshift/cluster-authentication-operator 91c741c0583efef554fd70512cd6d03746887a0b
cluster-autoscaler https://github.com/openshift/kubernetes-autoscaler dc250254b5ec8ba23aab11cb452cfffcc04961e8
cluster-autoscaler-operator https://github.com/openshift/cluster-autoscaler-operator c440c867cadc9d9cd5f30e8ccf0d84eb67adc1b1
cluster-bootstrap https://github.com/openshift/cluster-bootstrap 90a38fd8d9dc0b0a61214f079fd4734b034bae0c
cluster-config-operator https://github.com/openshift/cluster-config-operator d41351b54f3207cd5767eac0e67d27dd1ef23572
cluster-dns-operator https://github.com/openshift/cluster-dns-operator c838447353c661941435f4d525c18ac6586a8bb4
cluster-image-registry-operator https://github.com/openshift/cluster-image-registry-operator 9d40b11148e524b9d02f01fbf32a475480bc2f23
cluster-ingress-operator https://github.com/openshift/cluster-ingress-operator 3d112dff06ec1920c7d28d7bf6a967e1e0c6ce34
cluster-kube-apiserver-operator https://github.com/openshift/cluster-kube-apiserver-operator e4c45564c712d08ff6534831dea91be65619a09a
cluster-kube-controller-manager-operator https://github.com/openshift/cluster-kube-controller-manager-operator e100a870e4517ce86fd05639cb627aa535fc80cd
cluster-kube-scheduler-operator https://github.com/openshift/cluster-kube-scheduler-operator 0fe1ac1edbdf69b7a3204effeaa54650146e5eed
cluster-machine-approver https://github.com/openshift/cluster-machine-approver ac4a1b248ef534cc2f80e3fb11a9da3bca42865f
cluster-monitoring-operator https://github.com/openshift/cluster-monitoring-operator 5977734d74f7e115da53f5b470602d13e167be68
cluster-network-operator https://github.com/openshift/cluster-network-operator bde231baefea6491df3070d57a9756cbbdd6fb1c
cluster-node-tuned https://github.com/openshift/openshift-tuned b580cb6f52a0e352aebbe0e368d5ec020230c532
cluster-node-tuning-operator https://github.com/openshift/cluster-node-tuning-operator 499b14908cbedb105324187e35b4386b296491ee
cluster-openshift-apiserver-operator https://github.com/openshift/cluster-openshift-apiserver-operator d1599258656276bee0914bf092fcff6587870c32
cluster-openshift-controller-manager-operator https://github.com/openshift/cluster-openshift-controller-manager-operator 69639ade927d81f62d3db21e66976cbe0f86d53b
cluster-samples-operator https://github.com/openshift/cluster-samples-operator dabaf7e6468e28c9e421fa8711b3ed94987597da
cluster-storage-operator https://github.com/openshift/cluster-storage-operator 4cdc1e782067eacd0eed79cc886b023868498194
cluster-svcat-apiserver-operator https://github.com/openshift/cluster-svcat-apiserver-operator 6345527484f64e8170f4985bd7143b964bf6f358
cluster-svcat-controller-manager-operator https://github.com/openshift/cluster-svcat-controller-manager-operator 83ec5b9fc15ba35db0cad5a395bc4dbe4e9b4b4e
cluster-version-operator https://github.com/openshift/cluster-version-operator a4d54ef3817b49d624d23dacdb96cc5369edbdc0
configmap-reloader https://github.com/openshift/configmap-reload 3c2f85724078cbf7ffab56886ff32d677c386afe
console https://github.com/openshift/console 5cf425d0069e27173bb32670cf7a523ad5c528e5
console-operator https://github.com/openshift/console-operator 5cf0cde3c2cf18650b8c09358de9c4988dbdbe93
container-networking-plugins-supported https://github.com/openshift/containernetworking-plugins 1fbaada6af2da68f0109b7faa6cab5ae9bb3831a
container-networking-plugins-unsupported https://github.com/openshift/containernetworking-plugins 1fbaada6af2da68f0109b7faa6cab5ae9bb3831a
coredns https://github.com/openshift/coredns fbcb8252a1bab3d32ecf2dd3307f798aacd0280e
deployer https://github.com/openshift/origin c53f4620a46deacf86ba32126aaecb808929c3b5
docker-builder https://github.com/openshift/builder e31f9be31233bbd8a9af31b0b1d5a4ad90a7b9a4
docker-registry https://github.com/openshift/image-registry 0cdfa08c32196e44b84acc056db7a869b6dcf121
etcd https://github.com/openshift/etcd a0e62b48f8db8572c129fa3d3507c7ce118ab650
grafana https://github.com/openshift/grafana 2ea5517e5d33531ee8b838c70666e484a79cd49d
haproxy-router https://github.com/openshift/router 80b8c3d8e67e7549c59957421db2a5d344d8796a
hyperkube https://github.com/openshift/origin c53f4620a46deacf86ba32126aaecb808929c3b5
hypershift https://github.com/openshift/origin c53f4620a46deacf86ba32126aaecb808929c3b5
installer https://github.com/openshift/installer 4b12088025778add89e95099eaeebb773c39436d
jenkins https://github.com/openshift/jenkins 84aaea2f2da2edd2a67e5f9c537c8806ed5c8013
jenkins-agent-maven https://github.com/openshift/jenkins 84aaea2f2da2edd2a67e5f9c537c8806ed5c8013
jenkins-agent-nodejs https://github.com/openshift/jenkins 84aaea2f2da2edd2a67e5f9c537c8806ed5c8013
k8s-prometheus-adapter https://github.com/openshift/k8s-prometheus-adapter 815fa76bdbccfd5ee6da8f9fa45d039c4342dcdb
kube-rbac-proxy https://github.com/openshift/kube-rbac-proxy 3f271e0951f18276ec54e8eac936725d6d68e073
kube-state-metrics https://github.com/openshift/kube-state-metrics 2ab51c9f341799107ffbf7f373ab55254dc044d0
libvirt-machine-controllers https://github.com/openshift/cluster-api-provider-libvirt 1ef5bd8bfb47dc4fe7310557eb2600ecbabe648a
machine-api-operator https://github.com/openshift/machine-api-operator 9650e16c98802a4b57b7551201b0973fcae2f738
machine-config-controller https://github.com/openshift/machine-config-operator facd5b36fa94aa1298743070663086545b158524
machine-config-daemon https://github.com/openshift/machine-config-operator facd5b36fa94aa1298743070663086545b158524
machine-config-operator https://github.com/openshift/machine-config-operator facd5b36fa94aa1298743070663086545b158524
machine-config-server https://github.com/openshift/machine-config-operator facd5b36fa94aa1298743070663086545b158524
machine-os-content
multus-cni https://github.com/openshift/multus-cni 7094c9675ff29d05b71857bb83f20aaa3441fd88
must-gather https://github.com/openshift/must-gather 8e0e2950101e55a8520df162d3fd35e06954eafc
node https://github.com/openshift/origin c53f4620a46deacf86ba32126aaecb808929c3b5
oauth-proxy https://github.com/openshift/oauth-proxy 40c12481bfdd3e87d133736351c907000d5759b2
openstack-machine-controllers https://github.com/openshift/cluster-api-provider-openstack 9e913e83ca639e7f6e10fdffa8445f504b101f3c
operator-lifecycle-manager https://github.com/operator-framework/operator-lifecycle-manager 840d806a3b20e5ebb7229631d0168864b1cfed12
operator-marketplace https://github.com/operator-framework/operator-marketplace afc2569981c56f8e184157df8788557cf0303570
operator-registry https://github.com/operator-framework/operator-registry bf04182378fe117b4c162a19f4c70b209e61f334
pod https://github.com/openshift/images 2f60da39a9d2e5cc00293b8ec7ad559fcd32446a
prom-label-proxy https://github.com/openshift/prom-label-proxy 46423f9d573c7d53f5727de1e2db095ae039da06
prometheus https://github.com/openshift/prometheus 9966508f5575af5907722d3bb258393d5b7b0b3d
prometheus-alertmanager https://github.com/openshift/prometheus-alertmanager 4617d5502332dc41c9c885cc12ecde5069191f73
prometheus-config-reloader https://github.com/openshift/prometheus-operator 9a3ca6e49d51786fe266cc57d6d8b45ce6e6b8d4
prometheus-node-exporter https://github.com/openshift/node_exporter f248b582878226c8a8cd650223cf981cc556eb44
prometheus-operator https://github.com/openshift/prometheus-operator 9a3ca6e49d51786fe266cc57d6d8b45ce6e6b8d4
service-ca-operator https://github.com/openshift/service-ca-operator 152a8ec56ec0a6d6be3356b56dc4ba76bd4fdafc
service-catalog https://github.com/openshift/service-catalog a958caedc282bda63a70806b0f0ea0d1f367a025
setup-etcd-environment https://github.com/openshift/machine-config-operator facd5b36fa94aa1298743070663086545b158524
telemeter https://github.com/openshift/telemeter 7d3d12728408a2bf73919c7c558d4b12e211949a
tests https://github.com/openshift/origin c53f4620a46deacf86ba32126aaecb808929c3b5
$ oc get csr
NAME AGE REQUESTOR CONDITION
csr-d9zwj 94s system:node:test1-4cvdr-worker-0-9qr9x Pending
csr-g6frt 13m system:node:test1-4cvdr-master-0 Pending
csr-gr8qt 99s system:node:test1-4cvdr-master-0 Pending
csr-rnm95 13m system:node:test1-4cvdr-worker-0-9qr9x Pending
csr-snz5f 41m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-v942z 25m system:node:test1-4cvdr-master-0 Pending
csr-vsjb6 25m system:node:test1-4cvdr-worker-0-9qr9x Pending
csr-x67j4 34m system:node:test1-4cvdr-master-0 Pending
csr-xctrl 38m system:node:test1-4cvdr-worker-0-9qr9x Pending
csr-xx5bd 38m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-z2j96 41m system:node:test1-4cvdr-master-0 Pending
csr-zwhnv 33m system:node:test1-4cvdr-worker-0-9qr9x Pending
$ oc describe machine test1-4cvdr-worker-0-9qr9x
Name: test1-4cvdr-worker-0-9qr9x
Namespace: openshift-machine-api
Labels: machine.openshift.io/cluster-api-cluster=test1-4cvdr
machine.openshift.io/cluster-api-machine-role=worker
machine.openshift.io/cluster-api-machine-type=worker
machine.openshift.io/cluster-api-machineset=test1-4cvdr-worker-0
Annotations: <none>
API Version: machine.openshift.io/v1beta1
Kind: Machine
Metadata:
Creation Timestamp: 2019-03-14T06:08:34Z
Finalizers:
machine.machine.openshift.io
Generate Name: test1-4cvdr-worker-0-
Generation: 1
Owner References:
API Version: machine.openshift.io/v1beta1
Block Owner Deletion: true
Controller: true
Kind: MachineSet
Name: test1-4cvdr-worker-0
UID: 841d7007-461f-11e9-b814-52fdfc072182
Resource Version: 23615
Self Link: /apis/machine.openshift.io/v1beta1/namespaces/openshift-machine-api/machines/test1-4cvdr-worker-0-9qr9x
UID: 999d7d3f-461f-11e9-b814-52fdfc072182
Spec:
Metadata:
Creation Timestamp: <nil>
Provider Spec:
Value:
API Version: libvirtproviderconfig.k8s.io/v1alpha1
Autostart: false
Cloud Init: <nil>
Domain Memory: 4096
Domain Vcpu: 2
Ign Key:
Ignition:
User Data Secret: worker-user-data
Kind: LibvirtMachineProviderConfig
Network Interface Address: 192.168.126.0/24
Network Interface Hostname:
Network Interface Name: test1-4cvdr
Network UUID:
Uri: qemu+tcp://192.168.122.1/system
Volume:
Base Volume ID: /var/lib/libvirt/images/test1-4cvdr-base
Pool Name: default
Volume Name:
Versions:
Kubelet:
Status:
Addresses:
Address: 192.168.126.51
Type: InternalIP
Last Updated: 2019-03-14T06:47:29Z
Node Ref:
Kind: Node
Name: test1-4cvdr-worker-0-9qr9x
UID: d7dadbfa-461f-11e9-b814-52fdfc072182
Provider Status:
API Version: libvirtproviderconfig.openshift.io/v1beta1
Conditions: <nil>
Instance ID: c48c8280-2cd9-4d95-8b12-ef4b68f74a47
Instance State: Running
Kind: LibvirtMachineProviderStatus
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Created 44m libvirt-controller Created Machine test1-4cvdr-worker-0-9qr9x
$ oc describe machine test1-4cvdr-master-0
Name: test1-4cvdr-master-0
Namespace: openshift-machine-api
Labels: machine.openshift.io/cluster-api-cluster=test1-4cvdr
machine.openshift.io/cluster-api-machine-role=master
machine.openshift.io/cluster-api-machine-type=master
Annotations: <none>
API Version: machine.openshift.io/v1beta1
Kind: Machine
Metadata:
Creation Timestamp: 2019-03-14T06:07:57Z
Finalizers:
machine.machine.openshift.io
Generation: 1
Resource Version: 25564
Self Link: /apis/machine.openshift.io/v1beta1/namespaces/openshift-machine-api/machines/test1-4cvdr-master-0
UID: 83be4ef4-461f-11e9-b814-52fdfc072182
Spec:
Metadata:
Creation Timestamp: <nil>
Provider Spec:
Value:
API Version: libvirtproviderconfig.k8s.io/v1alpha1
Autostart: false
Cloud Init: <nil>
Domain Memory: 4096
Domain Vcpu: 2
Ign Key:
Ignition:
User Data Secret: master-user-data
Kind: LibvirtMachineProviderConfig
Network Interface Address: 192.168.126.0/24
Network Interface Hostname:
Network Interface Name: test1-4cvdr
Network UUID:
Uri: qemu+tcp://192.168.122.1/system
Volume:
Base Volume ID: /var/lib/libvirt/images/test1-4cvdr-base
Pool Name: default
Volume Name:
Versions:
Kubelet:
Status:
Addresses:
Address: 192.168.126.11
Type: InternalIP
Last Updated: 2019-03-14T06:53:23Z
Node Ref:
Kind: Node
Name: test1-4cvdr-master-0
UID: 59eaeb56-461f-11e9-b814-52fdfc072182
Provider Status:
API Version: libvirtproviderconfig.openshift.io/v1beta1
Conditions: <nil>
Instance ID: 7a24914a-abab-4e75-ab93-3e75f62a936d
Instance State: Running
Kind: LibvirtMachineProviderStatus
Events: <none>
// Need to manually approve to get the logs otherwise not even able to get the logs from the pod.
$ oc adm certificate approve $(oc get csr | egrep ^csr | awk '{ print $1 }')
certificatesigningrequest.certificates.k8s.io/csr-52mvs approved
certificatesigningrequest.certificates.k8s.io/csr-5892l approved
certificatesigningrequest.certificates.k8s.io/csr-79jtk approved
certificatesigningrequest.certificates.k8s.io/csr-c2hl5 approved
certificatesigningrequest.certificates.k8s.io/csr-d7v4j approved
certificatesigningrequest.certificates.k8s.io/csr-d9zwj approved
certificatesigningrequest.certificates.k8s.io/csr-fdcpl approved
certificatesigningrequest.certificates.k8s.io/csr-fhmqt approved
certificatesigningrequest.certificates.k8s.io/csr-fl5ck approved
certificatesigningrequest.certificates.k8s.io/csr-g6frt approved
certificatesigningrequest.certificates.k8s.io/csr-gdpmv approved
certificatesigningrequest.certificates.k8s.io/csr-gr8qt approved
certificatesigningrequest.certificates.k8s.io/csr-kfnh2 approved
certificatesigningrequest.certificates.k8s.io/csr-nck8r approved
certificatesigningrequest.certificates.k8s.io/csr-rnm95 approved
certificatesigningrequest.certificates.k8s.io/csr-sh79t approved
certificatesigningrequest.certificates.k8s.io/csr-snz5f approved
certificatesigningrequest.certificates.k8s.io/csr-v942z approved
certificatesigningrequest.certificates.k8s.io/csr-vsjb6 approved
certificatesigningrequest.certificates.k8s.io/csr-x67j4 approved
certificatesigningrequest.certificates.k8s.io/csr-xctrl approved
certificatesigningrequest.certificates.k8s.io/csr-xx5bd approved
certificatesigningrequest.certificates.k8s.io/csr-z2j96 approved
certificatesigningrequest.certificates.k8s.io/csr-zwhnv approved
$ oc logs machine-approver-794dbdf959-fg9sk -n openshift-cluster-machine-approver
W0314 06:06:55.120048 1 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0314 06:06:55.123766 1 main.go:185] Starting Machine Approver
I0314 06:06:55.124695 1 reflector.go:202] Starting reflector *v1beta1.CertificateSigningRequest (0s) from github.com/openshift/cluster-machine-appover/main.go:187
[...]
I0314 06:06:55.124761 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
t *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 06:10:08.356476 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 06:10:08.378310 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 06:10:09.379025 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
I0314 06:10:19.430096 1 main.go:97] CSR csr-snz5f added
I0314 06:10:19.433810 1 main.go:107] CSR csr-snz5f is already approved
I0314 06:10:19.459804 1 main.go:97] CSR csr-xx5bd added
I0314 06:10:19.460034 1 main.go:107] CSR csr-xx5bd is already approved
I0314 06:10:19.460159 1 main.go:97] CSR csr-z2j96 added
I0314 06:10:19.646952 1 main.go:123] CSR csr-z2j96 not authorized: DNS name 'test1-4cvdr-master-0' not in machine names:
I0314 06:10:19.647022 1 main.go:97] CSR csr-xctrl added
I0314 06:10:19.692526 1 main.go:123] CSR csr-xctrl not authorized: No target machine
E0314 06:12:02.137016 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=7, ErrCode=NO_ERROR, debug=""
I0314 06:13:21.952599 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 06:13:21.957317 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 06:13:22.998092 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
I0314 06:13:34.175385 1 main.go:97] CSR csr-x67j4 added
I0314 06:13:34.220734 1 main.go:123] CSR csr-x67j4 not authorized: DNS name 'test1-4cvdr-master-0' not in machine names:
E0314 06:14:42.640278 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=NO_ERROR, debug=""
E0314 06:14:42.718667 1 reflector.go:322] github.com/openshift/cluster-machine-appover/main.go:187: Failed to watch *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?resourceVersion=7626&timeoutSeconds=506&watch=true: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 06:14:43.719185 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 06:14:43.721781 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 06:14:44.726563 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
I0314 06:14:45.734155 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
I0314 06:15:00.141271 1 main.go:97] CSR csr-zwhnv added
I0314 06:15:00.247120 1 main.go:123] CSR csr-zwhnv not authorized: DNS name 'test1-4cvdr-worker-0-9qr9x' not in machine names:
E0314 06:16:38.801481 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=NO_ERROR, debug=""
E0314 06:16:39.235203 1 reflector.go:322] github.com/openshift/cluster-machine-appover/main.go:187: Failed to watch *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?resourceVersion=8709&timeoutSeconds=346&watch=true: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 06:16:40.242618 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 06:16:40.252673 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 06:22:56.462083 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
I0314 06:23:06.316077 1 main.go:97] CSR csr-v942z added
I0314 06:23:06.914276 1 main.go:123] CSR csr-v942z not authorized: DNS name 'test1-4cvdr-master-0' not in machine names:
I0314 06:23:08.794254 1 main.go:97] CSR csr-vsjb6 added
I0314 06:23:08.816293 1 main.go:123] CSR csr-vsjb6 not authorized: DNS name 'test1-4cvdr-worker-0-9qr9x' not in machine names:
E0314 06:28:59.524805 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=7, ErrCode=NO_ERROR, debug=""
E0314 06:28:59.745565 1 reflector.go:322] github.com/openshift/cluster-machine-appover/main.go:187: Failed to watch *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?resourceVersion=15861&timeoutSeconds=414&watch=true: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 06:29:00.764640 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 06:29:00.766272 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 06:29:12.662008 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
I0314 06:34:25.601010 1 main.go:97] CSR csr-rnm95 added
I0314 06:34:25.624114 1 main.go:123] CSR csr-rnm95 not authorized: DNS name 'test1-4cvdr-worker-0-9qr9x' not in machine names:
I0314 06:34:26.007757 1 main.go:97] CSR csr-g6frt added
I0314 06:34:26.017777 1 main.go:123] CSR csr-g6frt not authorized: DNS name 'test1-4cvdr-master-0' not in machine names:
E0314 06:34:57.130570 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=9, ErrCode=NO_ERROR, debug=""
W0314 06:35:17.928596 1 reflector.go:341] github.com/openshift/cluster-machine-appover/main.go:187: watch of *v1beta1.CertificateSigningRequest ended with: too old resource version: 18939 (19525)
I0314 06:35:18.936018 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 06:41:00.656540 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=NO_ERROR, debug=""
E0314 06:41:00.668793 1 reflector.go:322] github.com/openshift/cluster-machine-appover/main.go:187: Failed to watch *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?resourceVersion=19525&timeoutSeconds=387&watch=true: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 06:41:03.729372 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
I0314 06:46:40.340407 1 main.go:97] CSR csr-gr8qt added
I0314 06:46:40.381704 1 main.go:123] CSR csr-gr8qt not authorized: DNS name 'test1-4cvdr-master-0' not in machine names:
I0314 06:46:45.436348 1 main.go:97] CSR csr-d9zwj added
I0314 06:46:45.449629 1 main.go:123] CSR csr-d9zwj not authorized: DNS name 'test1-4cvdr-worker-0-9qr9x' not in machine names:
E0314 06:47:03.408702 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=7, ErrCode=NO_ERROR, debug=""
E0314 06:47:03.449794 1 reflector.go:322] github.com/openshift/cluster-machine-appover/main.go:187: Failed to watch *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?resourceVersion=23254&timeoutSeconds=365&watch=true: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 06:47:04.450512 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 06:47:04.453498 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 06:47:08.457412 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 06:47:08.465135 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 06:47:09.465395 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 06:53:04.562790 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3, ErrCode=NO_ERROR, debug=""
E0314 06:53:04.615390 1 reflector.go:322] github.com/openshift/cluster-machine-appover/main.go:187: Failed to watch *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?resourceVersion=23575&timeoutSeconds=408&watch=true: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 06:53:05.617095 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 06:53:05.621995 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 06:53:06.634340 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 06:53:06.738199 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 06:53:07.738706 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
I0314 06:58:24.388602 1 main.go:97] CSR csr-79jtk added
I0314 06:58:24.415346 1 main.go:123] CSR csr-79jtk not authorized: DNS name 'test1-4cvdr-master-0' not in machine names:
I0314 06:58:29.384471 1 main.go:97] CSR csr-fl5ck added
I0314 06:58:29.390655 1 main.go:123] CSR csr-fl5ck not authorized: DNS name 'test1-4cvdr-worker-0-9qr9x' not in machine names:
E0314 06:59:01.680006 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=7, ErrCode=NO_ERROR, debug=""
E0314 06:59:01.686782 1 reflector.go:322] github.com/openshift/cluster-machine-appover/main.go:187: Failed to watch *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?resourceVersion=26897&timeoutSeconds=558&watch=true: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 06:59:02.687321 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 06:59:02.689258 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 06:59:03.707043 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 06:59:13.871021 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa" cannot list resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
I0314 06:59:14.871272 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 07:05:12.387729 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=NO_ERROR, debug=""
E0314 07:05:12.444768 1 reflector.go:322] github.com/openshift/cluster-machine-appover/main.go:187: Failed to watch *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?resourceVersion=27405&timeoutSeconds=389&watch=true: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 07:05:13.445389 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 07:05:13.446618 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 07:05:14.448180 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 07:05:14.450451 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 07:05:15.451019 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
I0314 07:10:40.328038 1 main.go:97] CSR csr-fhmqt added
I0314 07:10:40.344356 1 main.go:123] CSR csr-fhmqt not authorized: DNS name 'test1-4cvdr-master-0' not in machine names:
I0314 07:10:45.388961 1 main.go:97] CSR csr-c2hl5 added
I0314 07:10:45.401358 1 main.go:123] CSR csr-c2hl5 not authorized: DNS name 'test1-4cvdr-worker-0-9qr9x' not in machine names:
E0314 07:11:25.522508 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=7, ErrCode=NO_ERROR, debug=""
E0314 07:11:25.612774 1 reflector.go:322] github.com/openshift/cluster-machine-appover/main.go:187: Failed to watch *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?resourceVersion=30926&timeoutSeconds=361&watch=true: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 07:11:26.616219 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 07:11:26.646738 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 07:11:27.652708 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 07:17:37.986396 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3, ErrCode=NO_ERROR, debug=""
E0314 07:17:37.991234 1 reflector.go:322] github.com/openshift/cluster-machine-appover/main.go:187: Failed to watch *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?resourceVersion=31702&timeoutSeconds=509&watch=true: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 07:22:56.412726 1 main.go:97] CSR csr-nck8r added
I0314 07:22:56.491801 1 main.go:123] CSR csr-nck8r not authorized: DNS name 'test1-4cvdr-master-0' not in machine names:
I0314 07:23:01.388462 1 main.go:97] CSR csr-52mvs added
I0314 07:23:01.419196 1 main.go:123] CSR csr-52mvs not authorized: DNS name 'test1-4cvdr-worker-0-9qr9x' not in machine names:
E0314 07:23:48.094877 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=7, ErrCode=NO_ERROR, debug=""
E0314 07:23:48.138728 1 reflector.go:322] github.com/openshift/cluster-machine-appover/main.go:187: Failed to watch *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?resourceVersion=34299&timeoutSeconds=308&watch=true: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 07:29:58.451163 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 07:29:58.467661 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 07:29:59.469180 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
I0314 07:35:12.326795 1 main.go:97] CSR csr-kfnh2 added
I0314 07:35:12.372763 1 main.go:123] CSR csr-kfnh2 not authorized: DNS name 'test1-4cvdr-master-0' not in machine names:
I0314 07:35:17.379595 1 main.go:97] CSR csr-d7v4j added
I0314 07:35:17.385114 1 main.go:123] CSR csr-d7v4j not authorized: DNS name 'test1-4cvdr-worker-0-9qr9x' not in machine names:
E0314 07:36:14.767221 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=7, ErrCode=NO_ERROR, debug=""
E0314 07:36:14.889972 1 reflector.go:322] github.com/openshift/cluster-machine-appover/main.go:187: Failed to watch *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?resourceVersion=37203&timeoutSeconds=323&watch=true: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 07:36:15.891381 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 07:36:15.894326 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 07:36:16.897561 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 07:36:16.913293 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 07:36:17.914077 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 07:42:24.725431 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3, ErrCode=NO_ERROR, debug=""
W0314 07:42:41.459745 1 reflector.go:341] github.com/openshift/cluster-machine-appover/main.go:187: watch of *v1beta1.CertificateSigningRequest ended with: too old resource version: 38149 (39544)
I0314 07:42:42.476689 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
I0314 07:48:00.324119 1 main.go:97] CSR csr-5892l added
I0314 07:48:00.388408 1 main.go:123] CSR csr-5892l not authorized: DNS name 'test1-4cvdr-master-0' not in machine names:
I0314 07:48:05.378133 1 main.go:97] CSR csr-gdpmv added
I0314 07:48:05.388994 1 main.go:123] CSR csr-gdpmv not authorized: DNS name 'test1-4cvdr-worker-0-9qr9x' not in machine names:
E0314 07:48:28.703772 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=9, ErrCode=NO_ERROR, debug=""
E0314 07:48:28.833398 1 reflector.go:322] github.com/openshift/cluster-machine-appover/main.go:187: Failed to watch *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?resourceVersion=40931&timeoutSeconds=390&watch=true: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 07:48:29.849885 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 07:54:45.455421 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=NO_ERROR, debug=""
E0314 07:54:45.486627 1 reflector.go:322] github.com/openshift/cluster-machine-appover/main.go:187: Failed to watch *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?resourceVersion=41379&timeoutSeconds=463&watch=true: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 07:54:46.505545 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 07:54:46.508790 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 07:54:47.521790 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
E0314 07:54:47.540889 1 reflector.go:205] github.com/openshift/cluster-machine-appover/main.go:187: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
I0314 07:54:48.542511 1 reflector.go:240] Listing and watching *v1beta1.CertificateSigningRequest from github.com/openshift/cluster-machine-appover/main.go:187
I0314 08:00:16.324881 1 main.go:97] CSR csr-fdcpl added
I0314 08:00:16.358588 1 main.go:123] CSR csr-fdcpl not authorized: DNS name 'test1-4cvdr-master-0' not in machine names:
I0314 08:00:21.379383 1 main.go:97] CSR csr-sh79t added
I0314 08:00:21.387497 1 main.go:123] CSR csr-sh79t not authorized: DNS name 'test1-4cvdr-worker-0-9qr9x' not in machine names:
E0314 08:01:00.159548 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=7, ErrCode=NO_ERROR, debug=""
E0314 08:01:00.203593 1 reflector.go:322] github.com/openshift/cluster-machine-appover/main.go:187: Failed to watch *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?resourceVersion=44401&timeoutSeconds=426&watch=true: dial tcp 127.0.0.1:6443: connect: connection refused
Please update the OWNERS file to point to the BZ component for this repository. This is important for people that are triaging bugs and for ART.
Example usage https://github.com/openshift/cluster-kube-apiserver-operator/blob/c70661b/OWNERS#L11
Thank you.
This is something we occasionally see in OpenStack deployments -- the installer succeeds, all nodes are Ready
, but some of the system:node:<worker>
CSRs are stuck in Pending
.
The CMA log shows this:
I0828 10:43:14.365504 1 main.go:107] CSR csr-29xv9 added
I0828 10:43:14.380314 1 main.go:132] CSR csr-29xv9 not authorized: No target machine
I0828 10:43:14.380364 1 main.go:164] Error syncing csr csr-29xv9: No target machine
<snip>
I0828 10:43:14.610334 1 main.go:107] CSR csr-29xv9 added
I0828 10:43:14.634045 1 main.go:132] CSR csr-29xv9 not authorized: No target machine
E0828 10:43:14.634206 1 main.go:174] No target machine
I0828 10:43:14.634260 1 main.go:175] Dropping CSR "csr-29xv9" out of the queue: No target machine
After that the CSR is dropped from the queue (seemingly) for ever.
As far as I can tell, this is because of a race between cluster-api-provider-openstack
setting the status.nodeRef
in the worker machine
and machine-config-operator
trying to approve the CSR. I haven't had a verification of this (it happens once every ten deployments or so and it's hard to "catch it in the act"), but it fits my understanding of the code and systems.
When this happens, deleting the cluster-machine-approver
pod and having it recreated approves those CSRs immediately (by that point nodeRef
exists so the machine check succeeds right away).
The error in the logs is coming out of here:
cluster-machine-approver/main.go
Lines 163 to 164 in a3fe0bb
Here's a run where I've increased the number of retries drastically (and the CSRs would have timed out otherwise):
I0829 10:35:24.052997 1 main.go:107] CSR csr-6fxl4 added
I0829 10:35:24.518607 1 main.go:132] CSR csr-6fxl4 not authorized: No target machine
I0829 10:35:24.518678 1 main.go:168] Error syncing csr csr-6fxl4: No target machine
I0829 10:35:24.867562 1 main.go:107] CSR csr-6fxl4 added
I0829 10:35:24.924957 1 main.go:132] CSR csr-6fxl4 not authorized: No target machine
I0829 10:35:24.924989 1 main.go:168] Error syncing csr csr-6fxl4: No target machine
I0829 10:35:25.063741 1 main.go:107] CSR csr-6fxl4 added
I0829 10:35:25.097110 1 main.go:132] CSR csr-6fxl4 not authorized: No target machine
I0829 10:35:25.097603 1 main.go:168] Error syncing csr csr-6fxl4: No target machine
I0829 10:35:25.158316 1 main.go:107] CSR csr-6fxl4 added
I0829 10:35:25.168623 1 main.go:132] CSR csr-6fxl4 not authorized: No target machine
I0829 10:35:25.168664 1 main.go:168] Error syncing csr csr-6fxl4: No target machine
I0829 10:35:25.209530 1 main.go:107] CSR csr-6fxl4 added
I0829 10:35:25.219819 1 main.go:132] CSR csr-6fxl4 not authorized: No target machine
I0829 10:35:25.219885 1 main.go:168] Error syncing csr csr-6fxl4: No target machine
I0829 10:35:25.321667 1 main.go:107] CSR csr-6fxl4 added
I0829 10:35:25.343725 1 main.go:132] CSR csr-6fxl4 not authorized: No target machine
I0829 10:35:25.343759 1 main.go:168] Error syncing csr csr-6fxl4: No target machine
I0829 10:35:25.504062 1 main.go:107] CSR csr-6fxl4 added
I0829 10:35:25.514563 1 main.go:132] CSR csr-6fxl4 not authorized: No target machine
I0829 10:35:25.514619 1 main.go:168] Error syncing csr csr-6fxl4: No target machine
I0829 10:35:25.834951 1 main.go:107] CSR csr-6fxl4 added
I0829 10:35:25.875817 1 main.go:132] CSR csr-6fxl4 not authorized: No target machine
I0829 10:35:25.875861 1 main.go:168] Error syncing csr csr-6fxl4: No target machine
I0829 10:35:26.516110 1 main.go:107] CSR csr-6fxl4 added
I0829 10:35:26.525090 1 main.go:132] CSR csr-6fxl4 not authorized: No target machine
I0829 10:35:26.525123 1 main.go:168] Error syncing csr csr-6fxl4: No target machine
I0829 10:35:27.805403 1 main.go:107] CSR csr-6fxl4 added
I0829 10:35:28.089184 1 main.go:147] CSR csr-6fxl4 approved
This took about 4 seconds and 10 retries.
Would it be possible to increase the threshold for failure? Despite the supposed exponential backoff, the delays seem to be rising very slowly and the time out currently happens in under 300 milliseconds. That seems like a really low threshold to me.
It seems to me that something like 20-30 retries should comfortably cover everything we've seen in the wild so far.
I'll be happy to write a pull request, but I'm not sure what should the right approach here. I can think of any of:
handleErr
status.nodeRef
fieldRelevant BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1741829
I'm sure there are things I'm missing here, I'd appreciate any comments or suggestions.
Ever since d7e76c4, the CSR on not-so-fast hosts (e.g nested-virt) don't get auto-approved any more and people have to manually approve them for cluster to initialize. After an enlightening discussion with the commit author @enj, I'm not sure what can be done in the short term for this but we have to fix this sooner or later somehow.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.