// ACCESSIBILITY_CONSTRAINTS indicates that the volumes for this
      // plugin may not be equally accessible by all nodes in the
      // cluster. The CO MUST use the topology information returned by
      // CreateVolumeRequest along with the topology information
      // returned by NodeGetInfo to ensure that a given volume is
      // accessible from a given node when scheduling workloads.
      ACCESSIBILITY_CONSTRAINTS = 2;

/assign

For some operations the operation poll doesn't actually return an error but there is an error on the field on the op itself.

"You can see that a 200 OK was returned and the operation has status=DONE, but there's an "error" field on the operation you should be looking at.

So your code should probably look at pollOp.error in addition to err.

There are additional fields "httpErrorStatusCode" and "httpErrorMessage". Perhaps it should be 403 instead of 400, but that's a separate issue."

Kubernetes recently solved by:

if op.Error != nil && len(op.Error.Errors) > 0 && op.Error.Errors[0] != nil {
		e := op.Error.Errors[0]
		o.err = &GCEOperationError{HTTPStatusCode: op.HTTPStatusCode, Code: e.Code, Message: e.Message}
}

Support changing filesystem type instead of just always defaulting to ext4

Right now CI will just create instance in project dyzz-test for testing, would be better to "loan" a temporary tenant project using Boskos when the tests are being run on CI.
When the loaned project is released there is an automatic job that cleans up so we don't accidentally leave test artifacts around.

/cc @krzyzacy

Need to add unit tests for controller calls

Add a flag --node to turn on the node server, --controller to turn on the controller server. They should both default to false.

Currently every driver runs all the node, controller, and identity servers by default.

go/src/sigs.k8s.io/gcp-compute-persistent-disk-csi-driver is the correct directory for naming, not go/src/github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver

Needs to be changed.

/cc @krzyzacy

Depends on #46 #47 #48

We need to integrate with CI for Github Pull Requests.

Should run:

• Sanity Tests
• E2E Tests
• Linter
• ???

Currently the Daemonset constantly attempts to create the pod on master and fails because of resource limits. We should stop scheduling on master so that it does not continually loop attempted creation.

This is high priority as we have seen the daemonset pod kick off the api server from the master putting the whole cluster in a bad state.

Codebase is currently littered with TODO's, some attributed to nobody/nothing and some to dyzz.

We should clean up the TODOs and create a github issue for each then link the Github issue in the TODO for tracking purposes

Support RePD for GCE PD.

Will likely depend on topology work as well

Everywhere in the project. package, file, directory, type names etc. GCE is not descriptive enough, in most of these places it is specifically gce-pd

Originally tried "compute-admin" and "compute-storage-admin" scopes but they seemed to not contain enough permissions for attach.

This is a tracking bug to revisit tightening the scopes required to deploy. If "Attach" is not currently supported in the "compute-admin" scope a bug should be opened against GCE Permissions because it definitely should be.

• Make GCP service account token file user configuratable
• Figure out actual minimal permissions set required for GCP service account
• Make namespace configurable

Hello,
Trying to deploy gcp-pd-csi to a GKE cluster. The deployment works, but trying to attach a disk I get the following error from the attacher:

GRPC error: rpc error: code = Unknown desc = googleapi: Error 403: Required 'compute.disks.get' permission for 'projects/XPROJECT/zones/europe-west3-a/disks/YDISK

(This is directly from the attacher log but the same error propagates up to the create pod error.

I'm using a service account in the deploy which is set-up as: Kubernetes Engine Admin, Service Account User, Storage Admin.

I'm trying to debug if the problem is in the deployment or is the service account missing certain rights.

• Add Known Issues
• Add Project Status
• Add Plugin Features (supported parameters)
• Add Future Features
• Add User Guide
  • Improve setup scripts to work every time, with reasonable error messages when variables not defined
• Add Kubernetes Development
• Add Dependency Management Information

Also enable Kubelet registrar functionality

Currently NodeGetInfoResponse returns the default of 0 for MaxVolumesPerNode so the CO will decide how many volumes can be published on a node.

For GCE we need to return a different number based on node type as the Max Attachable Volumes depends on the number of vCPUs the instance has.

For the actual limits see:
See: https://cloud.google.com/compute/docs/disks/
"persistent disk limits" section

You should be able to GET the instance from the cloud and pull the number of vCPUs from that.
Bonus: We seem to need information from the node object a lot, caching the relevant information somewhere would be nice. Maybe in the GCENodeServer object

k8s/k8s has some pretty well tested and thorough mount utilities. It would be nice to consume them directly instead of copy-pasting the mount utilities into this driver. There is an active issue tracking factoring these utilities out of k8s for easier consumption: kubernetes/kubernetes#64953

We should ensure that CSI Probe call is implemented and integrate https://github.com/kubernetes-csi/livenessprobe to automatically manage health of the CSI driver (e.g. restart driver if unhealthy).

We want to support raw block device in the driver

/assign @verult

There is a new process to push to gcr.io/google-containers. First push to staging-k8s.gcr.io then there are some other specific steps after it has been reviewed for the image to get moved from staging to prod.

/assign

Currently while deploying the driver, the Service Account environment variable needs to be in the format */cloud-sa.json. Ideally should be configurable based on the variable set by the user.

The E2E Test should be as close as production env as possible. This would require it to use a Service Account with the same roles as we provide in the setup-project script.

However, we probably do not want to just run the setup-project script again without thinking because this could be highly disruptive of production service accounts. It would be good to create a special service account with the same permissions just for the test, and make sure that it is deleted afterwards.

/cc @msau42

Currently container is using fedora:26 base image. This is not ideal.

We might want to change it to something more lightweight/secure like alpine or scratch.

/cc @msau42 might have some good ideas

It may be cleaner to encapsulate the WaitForOp in the cloud provider CreateDisk method, instead of having the driver do it. This is more in line with how the in-tree driver is implemented.

Currently E2E tests just copy the binary onto an instance to run the tests. They should run the tests with the containerized version of the driver because that is what the end user will likely be consuming, it will also detect errors in the containerization process.

These are really generic names:
$ export SA_FILE=~/.../cloud-sa.json
$ export GCEPD_SA_NAME=sample-service-account

It may be better go prefix them with "gce-pd-csi" or something like that.

Also as a side note, because the SA_FILE filename must be cloud-sa.json, it may be better to make the directory configurable, rather than the filename configurable.

There exists some manual udevadm triggering in the mount manager code due to a preexisting issue of devices not showing up properly. We need to verify that this udevadm trigger behavior is still called and works properly in this driver

Depends on: kubernetes/kubernetes#62561

We want to set up a config to run the Kubernetes E2E test for GCE PD here:
https://github.com/kubernetes/kubernetes/blob/master/test/e2e/storage/csi_volumes.go

Things like:
go vet
go fmt
etc.

Maybe just take some of the ones from test-infra:
https://github.com/kubernetes/test-infra/tree/master/hack

/assign

Currently points to my personal staging image which is changing all the time

Right now all CI Tests are running under one job pull-gcp-compute-persistent-disk-csi-driver-test.

We should split them up based on test type. Ex.

pull-gcp-compute-persistent-disk-csi-driver-sanity
pull-gcp-compute-persistent-disk-csi-driver-e2e
pull-gcp-compute-persistent-disk-csi-driver-kubernetes

Each of these should invoke a go test or equivalent directly so that test results are streamed through Gubernator propertly.

From Leonid:

So, apparently, deploy_driver.sh example under  gcp-compute-persistent-disk-csi-driver assumes that the creator is cluster-admin, which doesn't looks to be the case for GKE-gcloud-connect.
Manually creating relevant binding:
kubectl create clusterrolebinding cluster-admin-binding   --clusterrole cluster-admin   --user $(gcloud config get-value account)

seems to be enough to make the deployment work.

I verified his recommendation got me past this issue.

The E2E test on CI is really flaky. Needs investigation.

GCE PD in-tree driver supports partitioning

    gcePersistentDisk:
      pdName: partitioned-disk
      fsType: ext4
      partition: 2

According to security team this would be best.

I see the following when deploying to GKE:

2018-07-16 07:35:05.000 PDT
Unknown user "system:serviceaccount:default:csi-controller-sa"

But kubectl shows that the account exists:

$ kubectl get sa
NAME                SECRETS   AGE
csi-controller-sa   1         8m
csi-node-sa         1         8m
default             1         1h

Also see this error:

E  github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:serviceaccount:default:csi-controller-sa" cannot list persistentvolumeclaims at the cluster scope: [clusterrole.rbac.authorization.k8s.io "system:csi-external-attacher" not found, clusterrole.rbac.authorization.k8s.io "system:csi-external-provisioner" not found] 
  undefined
E  Unknown user "system:serviceaccount:default:csi-controller-sa"
 
  undefined

But kubectl shows the clusterrolebinding exists

$ kubectl describe clusterrolebindings csi-controller-attacher-binding
Name:         csi-controller-attacher-binding
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"csi-controller-attacher-binding","namespa...
Role:
  Kind:  ClusterRole
  Name:  system:csi-external-attacher
Subjects:
  Kind            Name               Namespace
  ----            ----               ---------
  ServiceAccount  csi-controller-sa  default

$ kubectl describe clusterrolebindings csi-controller-provisioner-binding
Name:         csi-controller-provisioner-binding
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"csi-controller-provisioner-binding","name...
Role:
  Kind:  ClusterRole
  Name:  system:csi-external-provisioner
Subjects:
  Kind            Name               Namespace
  ----            ----               ---------
  ServiceAccount  csi-controller-sa  default

E2E Framework should actually run locally, we can create instances (possibly multizonal) that run the driver binary that actually exposes their endpoint as an SSL secured TCP endpoint:
https://grpc.io/docs/guides/auth.html#examples

This way the e2e tests can actually spin these instances up, and spin up a grpc client that we use to call these instances for the tests.

This makes the multi-zonal tests easier to create and makes it so that we dont have to stream test commands and results back and forth between the instances

Makefile defines:

STAGINGIMAGE=${GCE_PD_CSI_STAGING_IMAGE}

But provides no default, so if that environment variable is not set, make build-container fails.

Refactor out volume capability validation code, call it in various controller functions to validate that we support the volume capabilities before doing any controller calls

After the attach call, and waiting for the attach op to complete. Sometimes the attach will not error but the disk will not be attached.

We should add an additional verification to make sure that the disk has been attached. We can see this on the disk object:

// Users: [Output Only] Links to the users of the disk (attached
// instances) in form: project/zones/zone/instances/instance
Users []string `json:"users,omitempty"`

OR on the instance there is AttachedDisks.

https://github.com/container-storage-interface/spec/releases/tag/v0.3.0

/assign

Also would be nice if it was more platform agnostic

Maybe Python...?