Operator fails with 'Generation: expected: 3, actual: 5' about cluster-openshift-apiserver-operator HOT 11 CLOSED

@vrutkovs i believe this is transient error. The operator is reporting available==true, the problem here is that we just not cleaning up the failing status.

/cc @deads2k
/cc @sanchezl

from cluster-openshift-apiserver-operator.

Right, this happens on ~10% of installs and doesn't seem to be reporting a real issue.

These conditions are used in playbooks to wait for operator to complete setup though

from cluster-openshift-apiserver-operator.

This now happens fairly often and blocks ingress operator install: operator gets stuck on 'Generation: expected: 3, actual: 2' (numbers may vary, but this is most frequent combination) and ingress is not deployed

from cluster-openshift-apiserver-operator.

This causes CVO to stumble and blocks other operators - e.g. machine approver, so some CI runs don't get CSR approved and return empty logs during artifacts stage

from cluster-openshift-apiserver-operator.

@abhinavdahiya how long the CVO wait to settle? Also, any idea why the CI isn't pulling pod logs?

from cluster-openshift-apiserver-operator.

how long the CVO wait to settle?

it waits as long it takes. CVO keeps retrying always...

Also, any idea why the CI isn't pulling pod logs?

hmm, do you have any particular one?

from cluster-openshift-apiserver-operator.

hmm, do you have any particular one?

the one linked from the description. https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_openshift-ansible/10695/pull-ci-openshift-openshift-ansible-master-e2e-gcp/1078

from cluster-openshift-apiserver-operator.

hmm, do you have any particular one?

the one linked from the description. https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_openshift-ansible/10695/pull-ci-openshift-openshift-ansible-master-e2e-gcp/1078

This is ansible ci job, i have no clue how it s setup. 😇

from cluster-openshift-apiserver-operator.

Also, any idea why the CI isn't pulling pod logs?

Investigated this with @eparis in Slack.
CVO starts operators.
apiserver gets stuck on Generation: expected 3, actual: 5 or similar
CVO is paused and never calls machine-approver operator
node csrs are stuck in Pending and any oc log call returns tls error. As a result we can get events, pods list and nodes - but not the logs for pods

from cluster-openshift-apiserver-operator.

$ oc get clusteroperator -o yaml openshift-cluster-kube-apiserver-operator
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: 2018-12-12T15:18:13Z
  generation: 1
  name: openshift-cluster-kube-apiserver-operator
  resourceVersion: "97258"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/openshift-cluster-kube-apiserver-operator
  uid: 2451d21c-fe21-11e8-a512-021a424c72ca
spec: {}
status:
  conditions:
  - lastTransitionTime: 2018-12-12T19:01:09Z
    status: "False"
    type: Failing
  - lastTransitionTime: 2018-12-12T19:01:09Z
    message: 3 of 3 nodes are at revision 1
    status: "True"
    type: Available
  - lastTransitionTime: 2018-12-12T19:01:09Z
    message: 0 of 3 nodes are at revision 1
    reason: AllNodesAtLatestRevision
    status: "False"
    type: Progressing
  extension: null
  version: ""

Somehow both 0/3 and 3/3 nodes were at revision 1 at the same time. Hiesenburg!!!

from cluster-openshift-apiserver-operator.

I waited at least 4 hours for it to settle (and this is how it settled)

from cluster-openshift-apiserver-operator.