All PVC going in pending state during peer add operations about gluster-csi-driver HOT 17 CLOSED

from gluster-csi-driver.

If newly added glusterd2 pod does not have devices attached to it, CSI driver will not be able to create replica 3 volumes.

from gluster-csi-driver.

New gd2 pod has devices attached. I will shortly attach the logs.

from gluster-csi-driver.

csi.log
glusterd2.log

from gluster-csi-driver.

@Madhu-1 I see log flooding in glusterd2.log . Is it expected or some weird behaviour happening?

from gluster-csi-driver.

This is based on the retry going to done by CSI driver, I think this is expected, the request to the CSI driver will be sent by the sidecar containers.

from gluster-csi-driver.

@ksandha need logs from gluster-provisioner container in provisioner pod.

from gluster-csi-driver.

I don't think this should be expected. Once the new pod is in cluster of gd2 all the new PVC should come in bound state.

from gluster-csi-driver.

@ksandha retry happens where CSI driver not able to create the volume in the glusterd2 cluster, you may be seeing more HTTP logs in the glusterd2.

please provide logs from gluster-provisioner container in provisioner pod. to confirm the failure, whether there is an issue with CSI driver or the glusterd2 cluster,

from gluster-csi-driver.

@ksandha any update on this issue?

from gluster-csi-driver.

@Madhu-1 i will reproduce this issue again as i accidentally deleted the set-up , couldn't collect the logs of the specific container from the cluster.

from gluster-csi-driver.

[vagrant@kube1 ~]$ kubectl -n gcs logs csi-provisioner-glusterfsplugin-0 csi-provisioner
I1128 05:51:19.662256 1 csi-provisioner.go:82] Version: v0.4.0-rc.1-5-gf2ffed45
I1128 05:51:19.662702 1 csi-provisioner.go:96] Building kube configs for running in cluster...
I1128 05:52:44.593217 1 controller.go:572] Starting provisioner controller org.gluster.glusterfs_csi-provisioner-glusterfsplugin-0_d389fe28-f2d1-11e8-839a-0a580ae94009!
I1128 05:52:44.693603 1 controller.go:621] Started provisioner controller org.gluster.glusterfs_csi-provisioner-glusterfsplugin-0_d389fe28-f2d1-11e8-839a-0a580ae94009!
I1128 06:40:24.403631 1 controller.go:927] provision "default/gcs-pvc1" class "glusterfs-csi": started
I1128 06:40:24.437181 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc1", UID:"7c167cab-f2d8-11e8-bee7-5254002fa4e8", APIVersion:"v1", ResourceVersion:"8103", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/gcs-pvc1"
I1128 06:40:26.358678 1 controller.go:551] successfully created PV {GCEPersistentDisk:nil AWSElasticBlockStore:nil HostPath:nil Glusterfs:nil NFS:nil RBD:nil ISCSI:nil Cinder:nil CephFS:nil FC:nil Flocker:nil FlexVolume:nil AzureFile:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil PortworxVolume:nil ScaleIO:nil Local:nil StorageOS:nil CSI:&CSIPersistentVolumeSource{Driver:org.gluster.glusterfs,VolumeHandle:pvc-7c167cab-f2d8-11e8-bee7-5254002fa4e8,ReadOnly:false,FSType:ext4,VolumeAttributes:map[string]string{glusterbkpservers: gluster-kube1-0.glusterd2.gcs:gluster-kube2-0.glusterd2.gcs,glusterserver: gluster-kube3-0.glusterd2.gcs,glustervol: pvc-7c167cab-f2d8-11e8-bee7-5254002fa4e8,storage.kubernetes.io/csiProvisionerIdentity: 1543384279692-8081-org.gluster.glusterfs,},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,}}
I1128 06:40:26.360801 1 controller.go:1027] provision "default/gcs-pvc1" class "glusterfs-csi": volume "pvc-7c167cab-f2d8-11e8-bee7-5254002fa4e8" provisioned
I1128 06:40:26.360842 1 controller.go:1041] provision "default/gcs-pvc1" class "glusterfs-csi": trying to save persistentvolume "pvc-7c167cab-f2d8-11e8-bee7-5254002fa4e8"
I1128 06:40:26.389881 1 controller.go:1048] provision "default/gcs-pvc1" class "glusterfs-csi": persistentvolume "pvc-7c167cab-f2d8-11e8-bee7-5254002fa4e8" saved
I1128 06:40:26.389937 1 controller.go:1089] provision "default/gcs-pvc1" class "glusterfs-csi": succeeded
I1128 06:40:26.390022 1 controller.go:927] provision "default/gcs-pvc1" class "glusterfs-csi": started
I1128 06:40:26.392183 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc1", UID:"7c167cab-f2d8-11e8-bee7-5254002fa4e8", APIVersion:"v1", ResourceVersion:"8103", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-7c167cab-f2d8-11e8-bee7-5254002fa4e8
I1128 06:40:26.400877 1 controller.go:936] provision "default/gcs-pvc1" class "glusterfs-csi": persistentvolume "pvc-7c167cab-f2d8-11e8-bee7-5254002fa4e8" already exists, skipping
I1128 08:05:26.467424 1 controller.go:1098] delete "pvc-7c167cab-f2d8-11e8-bee7-5254002fa4e8": started
I1128 08:05:27.214262 1 controller.go:1126] delete "pvc-7c167cab-f2d8-11e8-bee7-5254002fa4e8": volume deleted
I1128 08:05:27.253018 1 controller.go:1136] delete "pvc-7c167cab-f2d8-11e8-bee7-5254002fa4e8": persistentvolume deleted
I1128 08:05:27.253051 1 controller.go:1138] delete "pvc-7c167cab-f2d8-11e8-bee7-5254002fa4e8": succeeded
I1128 09:40:28.723323 1 controller.go:927] provision "default/gcs-pvc1" class "glusterfs-csi": started
I1128 09:40:28.776159 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc1", UID:"a3f71a18-f2f1-11e8-bee7-5254002fa4e8", APIVersion:"v1", ResourceVersion:"29693", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/gcs-pvc1"
W1128 09:40:28.800870 1 controller.go:686] Retrying syncing claim "default/gcs-pvc1" because failures 0 < threshold 15
E1128 09:40:28.800937 1 controller.go:701] error syncing claim "default/gcs-pvc1": failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: no space available or all the devices are not registered
I1128 09:40:28.800979 1 controller.go:927] provision "default/gcs-pvc1" class "glusterfs-csi": started
I1128 09:40:28.801221 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc1", UID:"a3f71a18-f2f1-11e8-bee7-5254002fa4e8", APIVersion:"v1", ResourceVersion:"29693", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: no space available or all the devices are not registered
I1128 09:40:28.814655 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc1", UID:"a3f71a18-f2f1-11e8-bee7-5254002fa4e8", APIVersion:"v1", ResourceVersion:"29694", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/gcs-pvc1"
W1128 09:40:28.834904 1 controller.go:686] Retrying syncing claim "default/gcs-pvc1" because failures 1 < threshold 15
E1128 09:40:28.834940 1 controller.go:701] error syncing claim "default/gcs-pvc1": failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: no space available or all the devices are not registered
I1128 09:40:28.834967 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc1", UID:"a3f71a18-f2f1-11e8-bee7-5254002fa4e8", APIVersion:"v1", ResourceVersion:"29694", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: no space available or all the devices are not registered
I1128 09:40:43.801281 1 controller.go:927] provision "default/gcs-pvc1" class "glusterfs-csi": started
I1128 09:40:43.817843 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc1", UID:"a3f71a18-f2f1-11e8-bee7-5254002fa4e8", APIVersion:"v1", ResourceVersion:"29694", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/gcs-pvc1"
W1128 09:40:43.838011 1 controller.go:686] Retrying syncing claim "default/gcs-pvc1" because failures 2 < threshold 15
E1128 09:40:43.838060 1 controller.go:701] error syncing claim "default/gcs-pvc1": failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: no space available or all the devices are not registered
I1128 09:40:43.838108 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc1", UID:"a3f71a18-f2f1-11e8-bee7-5254002fa4e8", APIVersion:"v1", ResourceVersion:"29694", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: no space available or all the devices are not registered
I1128 09:41:43.838414 1 controller.go:927] provision "default/gcs-pvc1" class "glusterfs-csi": started
I1128 09:41:43.852861 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc1", UID:"a3f71a18-f2f1-11e8-bee7-5254002fa4e8", APIVersion:"v1", ResourceVersion:"29694", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/gcs-pvc1"
W1128 09:41:43.876377 1 controller.go:686] Retrying syncing claim "default/gcs-pvc1" because failures 3 < threshold 15
E1128 09:41:43.876422 1 controller.go:701] error syncing claim "default/gcs-pvc1": failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: no space available or all the devices are not registered
I1128 09:41:43.876904 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc1", UID:"a3f71a18-f2f1-11e8-bee7-5254002fa4e8", APIVersion:"v1", ResourceVersion:"29694", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: no space available or all the devices are not registered
I1128 09:43:43.876844 1 controller.go:927] provision "default/gcs-pvc1" class "glusterfs-csi": started
I1128 09:43:43.891557 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc1", UID:"a3f71a18-f2f1-11e8-bee7-5254002fa4e8", APIVersion:"v1", ResourceVersion:"29694", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/gcs-pvc1"
W1128 09:43:43.911766 1 controller.go:686] Retrying syncing claim "default/gcs-pvc1" because failures 4 < threshold 15
E1128 09:43:43.911820 1 controller.go:701] error syncing claim "default/gcs-pvc1": failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: no space available or all the devices are not registered
I1128 09:43:43.911878 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc1", UID:"a3f71a18-f2f1-11e8-bee7-5254002fa4e8", APIVersion:"v1", ResourceVersion:"29694", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: no space available or all the devices are not registered
I1128 09:47:43.912225 1 controller.go:927] provision "default/gcs-pvc1" class "glusterfs-csi": started
I1128 09:47:43.935184 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc1", UID:"a3f71a18-f2f1-11e8-bee7-5254002fa4e8", APIVersion:"v1", ResourceVersion:"29694", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/gcs-pvc1"
W1128 09:47:43.957162 1 controller.go:686] Retrying syncing claim "default/gcs-pvc1" because failures 5 < threshold 15
E1128 09:47:43.957206 1 controller.go:701] error syncing claim "default/gcs-pvc1": failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: no space available or all the devices are not registered
I1128 09:47:43.957712 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc1", UID:"a3f71a18-f2f1-11e8-bee7-5254002fa4e8", APIVersion:"v1", ResourceVersion:"29694", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: no space available or all the devices are not registered
[vagrant@kube1 ~]$

from gluster-csi-driver.

failed to create volume: no space available or all the devices are not registered

this is the issue with the gd2 , not with the CSI driver

from gluster-csi-driver.

@ksandha we need to move this issue to gd2 instead of CSI driver

from gluster-csi-driver.

Okay replicating it there. @Madhu-1

from gluster-csi-driver.

@ksandha please close this issue once you move it to gd2.

from gluster-csi-driver.

Cloned the issue hence closing it in CSI branch.

from gluster-csi-driver.