[Flaking Test] gce-master-scale-correctness (Unexpected CSI call) about kubernetes HOT 10 OPEN

/help

We need to rewrite csi-driver-hostpath from glog to klog and check klog has a better limit. The flaky test should then start working.

from kubernetes.

/reopen
we still need to tag a new csi-driver-hostpath release + maybe update the image version in couple of places

from kubernetes.

When the test was working few days ago (job result, external-provisioner log), a single CreateVolume call in the test had ~ 9KB line in the provisioner log.

Now (job result, external-provisioner log), such a CreateVolume call has ~17KB, because provisioner sends more preferred and requisite segments in the call (had 161 segments, now has 303). I am not sure where we have any limit, it could be 16KB per log line hardcoded somewhere.

from kubernetes.

I can see the mock CSI driver logs are cut in the middle, search for CreateVolume and GRPC request JSON on the next log line is cut in the middle:

https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/1802386080972935168/artifacts/_sig-storage_CSI_Mock_volume_storage_capacity/unlimited/storage_capacity/unlimited/csi-mockplugin-0-mock.log

from kubernetes.

This line seems to cut long lines: https://github.com/kubernetes-csi/csi-driver-host-path/blob/586ed79ac6ae87be0186bd4a4fe9589054d73199/pkg/hostpath/server.go#L108

Maybe it's glog (sic!), maybe protosanitizer.

from kubernetes.

glog seems to have 1500 character limit on log lines: https://github.com/golang/glog/blob/861d094757551d2f65d1003f0ec93bc92b254a5a/internal/logsink/logsink.go#L33

So... I would not say it's a blocker, the test was rock stable for months.

Someone needs to rewrite the CSI driver from glog to klog and check klog has a better limit.

@carlory not sure if you have time

from kubernetes.

@jsafrane:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

from kubernetes.

@jsafrane: Reopened this issue.

from kubernetes.

/triage accepted

from kubernetes.

#124842 will update test-manfiests

from kubernetes.