Installed csi according docs.
Started.
So far so good, added storage class 'vsphere-gold' and created pvc.
time="2019-01-21T15:19:45Z" level=debug msg="enabled context injector"
time="2019-01-21T15:19:45Z" level=debug msg="init req & rep validation" withSpec=false
time="2019-01-21T15:19:45Z" level=debug msg="init implicit rep validation" withSpecRep=false
time="2019-01-21T15:19:45Z" level=debug msg="init req validation" withSpecReq=false
time="2019-01-21T15:19:45Z" level=debug msg="enabled request ID injector"
time="2019-01-21T15:19:45Z" level=debug msg="enabled request logging"
time="2019-01-21T15:19:45Z" level=debug msg="enabled response logging"
time="2019-01-21T15:19:45Z" level=debug msg="enabled serial volume access"
time="2019-01-21T15:19:45Z" level=info msg="configured: io.k8s.cloud-provider-vsphere.vsphere" api=FCD mode=controller
time="2019-01-21T15:19:45Z" level=info msg="identity service registered"
time="2019-01-21T15:19:45Z" level=info msg="controller service registered"
time="2019-01-21T15:19:45Z" level=info msg=serving endpoint="unix:///var/lib/csi/sockets/pluginproxy/csi.sock"
time="2019-01-21T15:19:46Z" level=debug msg="/csi.v1.Identity/Probe: REQ 0001: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:19:46Z" level=debug msg="/csi.v1.Identity/Probe: REP 0001: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:19:46Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REQ 0002: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:19:46Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REP 0002: Name=io.k8s.cloud-provider-vsphere.vsphere, VendorVersion=v0.1.1, XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:19:46Z" level=debug msg="/csi.v1.Identity/GetPluginCapabilities: REQ 0003: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:19:46Z" level=debug msg="/csi.v1.Identity/GetPluginCapabilities: REP 0003: Capabilities=[service:<type:CONTROLLER_SERVICE > ], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:19:46Z" level=debug msg="/csi.v1.Controller/ControllerGetCapabilities: REQ 0004: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:19:46Z" level=debug msg="/csi.v1.Controller/ControllerGetCapabilities: REP 0004: Capabilities=[rpc:<type:LIST_VOLUMES > rpc:<type:CREATE_DELETE_VOLUME > rpc:<type:PUBLISH_UNPUBLISH_VOLUME > ], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:28:35Z" level=debug msg="/csi.v1.Identity/GetPluginCapabilities: REQ 0005: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:28:35Z" level=debug msg="/csi.v1.Identity/GetPluginCapabilities: REP 0005: Capabilities=[service:<type:CONTROLLER_SERVICE > ], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:28:35Z" level=debug msg="/csi.v1.Controller/ControllerGetCapabilities: REQ 0006: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:28:35Z" level=debug msg="/csi.v1.Controller/ControllerGetCapabilities: REP 0006: Capabilities=[rpc:<type:LIST_VOLUMES > rpc:<type:CREATE_DELETE_VOLUME > rpc:<type:PUBLISH_UNPUBLISH_VOLUME > ], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:28:35Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REQ 0007: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:28:35Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REP 0007: Name=io.k8s.cloud-provider-vsphere.vsphere, VendorVersion=v0.1.1, XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:28:35Z" level=debug msg="/csi.v1.Controller/CreateVolume: REQ 0008: Name=pvc-34370d70-1d91-11e9-8953-00505697cc54, CapacityRange=required_bytes:1073741824 , VolumeCapabilities=[mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > ], Parameters=map[parent_type:Datastore parent_name:ESXi-CL1-VV1 (RAID 50)], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
ERROR: logging before flag.Parse: E0121 15:28:40.988002 1 connection.go:63] Failed to create govmomi client. err: ServerFaultCode: Cannot complete login due to an incorrect user name or password.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x15a6d36]
According 'Cannot complete login due to an incorrect user name or password' i decided there is authentication issue.
Recreated cluster.
Added server, username and password to vsphere.conf
Installed ccm and csi.
Created pvc and got:
kubectl --kubeconfig admin.conf -n kube-system logs -f vsphere-csi-controller-0 vsphere-csi-controller
time="2019-01-21T15:54:15Z" level=debug msg="enabled context injector"
time="2019-01-21T15:54:15Z" level=debug msg="init req & rep validation" withSpec=false
time="2019-01-21T15:54:15Z" level=debug msg="init implicit rep validation" withSpecRep=false
time="2019-01-21T15:54:15Z" level=debug msg="init req validation" withSpecReq=false
time="2019-01-21T15:54:15Z" level=debug msg="enabled request ID injector"
time="2019-01-21T15:54:15Z" level=debug msg="enabled request logging"
time="2019-01-21T15:54:15Z" level=debug msg="enabled response logging"
time="2019-01-21T15:54:15Z" level=debug msg="enabled serial volume access"
time="2019-01-21T15:54:15Z" level=info msg="configured: io.k8s.cloud-provider-vsphere.vsphere" api=FCD mode=controller
time="2019-01-21T15:54:15Z" level=info msg="identity service registered"
time="2019-01-21T15:54:15Z" level=info msg="controller service registered"
time="2019-01-21T15:54:15Z" level=info msg=serving endpoint="unix:///var/lib/csi/sockets/pluginproxy/csi.sock"
time="2019-01-21T15:54:16Z" level=debug msg="/csi.v1.Identity/Probe: REQ 0001: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:54:16Z" level=debug msg="/csi.v1.Identity/Probe: REP 0001: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:54:16Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REQ 0002: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:54:16Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REP 0002: Name=io.k8s.cloud-provider-vsphere.vsphere, VendorVersion=v0.1.1, XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:54:16Z" level=debug msg="/csi.v1.Identity/GetPluginCapabilities: REQ 0003: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:54:16Z" level=debug msg="/csi.v1.Identity/GetPluginCapabilities: REP 0003: Capabilities=[service:<type:CONTROLLER_SERVICE > ], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:54:16Z" level=debug msg="/csi.v1.Controller/ControllerGetCapabilities: REQ 0004: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:54:16Z" level=debug msg="/csi.v1.Controller/ControllerGetCapabilities: REP 0004: Capabilities=[rpc:<type:LIST_VOLUMES > rpc:<type:CREATE_DELETE_VOLUME > rpc:<type:PUBLISH_UNPUBLISH_VOLUME > ], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:58:19Z" level=debug msg="/csi.v1.Identity/GetPluginCapabilities: REQ 0005: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:58:19Z" level=debug msg="/csi.v1.Identity/GetPluginCapabilities: REP 0005: Capabilities=[service:<type:CONTROLLER_SERVICE > ], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:58:19Z" level=debug msg="/csi.v1.Controller/ControllerGetCapabilities: REQ 0006: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:58:19Z" level=debug msg="/csi.v1.Controller/ControllerGetCapabilities: REP 0006: Capabilities=[rpc:<type:LIST_VOLUMES > rpc:<type:CREATE_DELETE_VOLUME > rpc:<type:PUBLISH_UNPUBLISH_VOLUME > ], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:58:19Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REQ 0007: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:58:19Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REP 0007: Name=io.k8s.cloud-provider-vsphere.vsphere, VendorVersion=v0.1.1, XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2019-01-21T15:58:19Z" level=debug msg="/csi.v1.Controller/CreateVolume: REQ 0008: Name=pvc-5ee40328-1d95-11e9-8858-0050569742c5, CapacityRange=required_bytes:1073741824 , VolumeCapabilities=[mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > ], Parameters=map[parent_name:ESXi-CL1-VV1 (RAID 50) parent_type:Datastore], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x15a6d36]
goroutine 76 [running]:
k8s.io/cloud-provider-vsphere/vendor/github.com/vmware/govmomi/vslm.NewObjectManager(...)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/vmware/govmomi/vslm/object_manager.go:41
k8s.io/cloud-provider-vsphere/pkg/common/vclib.(*DatastoreInfo).GetFirstClassDiskInfo(0xc00072e270, 0x1e3c700, 0xc0006071d0, 0xc0006453b0, 0x28, 0x1, 0x0, 0x0, 0x41199c)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/pkg/common/vclib/datastore.go:197 +0x86
k8s.io/cloud-provider-vsphere/pkg/common/vclib.(*Datacenter).GetFirstClassDisk(0xc0000a6228, 0x1e3c700, 0xc0006071d0, 0xc0000a5280, 0x16, 0x1bd6372, 0x9, 0xc0006453b0, 0x28, 0x1, ...)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/pkg/common/vclib/datacenter.go:518 +0x1ab
k8s.io/cloud-provider-vsphere/pkg/csi/service/fcd.(*controller).CreateVolume(0xc00000ab80, 0x1e3c700, 0xc0006071d0, 0xc0005b83f0, 0xc00000ab80, 0xc000620e01, 0xc000185390)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/pkg/csi/service/fcd/controller.go:132 +0x4a0
k8s.io/cloud-provider-vsphere/vendor/github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1(0x1e3c700, 0xc0006071d0, 0x1b46320, 0xc0005b83f0, 0x0, 0x1e2a300, 0xc000620e40, 0x0)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:4579 +0x86
k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume(0xc0000b0d80, 0x1e3c700, 0xc0006071d0, 0xc0005b83f0, 0xc000620c60, 0xc000620c80, 0x0, 0x0, 0x0, 0x0)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware/serialvolume/serial_volume_locker.go:162 +0x161
k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle(0xc0000b0d80, 0x1e3c700, 0xc0006071d0, 0x1b46320, 0xc0005b83f0, 0xc000620c60, 0xc000620c80, 0x180, 0x14d, 0x0, ...)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware/serialvolume/serial_volume_locker.go:90 +0x37d
k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle-fm(0x1e3c700, 0xc0006071d0, 0x1b46320, 0xc0005b83f0, 0xc000620c60, 0xc000620c80, 0xc00036a0c0, 0xc000185558, 0x4c7ab7, 0xc00036a0c0)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware/serialvolume/serial_volume_locker.go:71 +0x73
k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1(0x1e3c700, 0xc0006071d0, 0x1b46320, 0xc0005b83f0, 0x14d, 0x0, 0xc00036a0c0, 0x0)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/utils/utils_middleware.go:99 +0x63
k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware/logging.(*interceptor).handleServer.func1(0x1e18160, 0xc0000b0c80, 0xc000185670, 0x1)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware/logging/logging_interceptor.go:84 +0x49
k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware/logging.(*interceptor).handle(0xc0000b0ce0, 0x1e3c700, 0xc0006071d0, 0x1c0b16f, 0x1f, 0x1b46320, 0xc0005b83f0, 0xc000185720, 0xc000000064, 0x412da5, ...)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware/logging/logging_interceptor.go:130 +0xc6
k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware/logging.(*interceptor).handleServer(0xc0000b0ce0, 0x1e3c700, 0xc0006071d0, 0x1b46320, 0xc0005b83f0, 0xc000620c60, 0xc000620ca0, 0xc0003f57c0, 0x0, 0xc00023a000, ...)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware/logging/logging_interceptor.go:83 +0xe0
k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware/logging.(*interceptor).handleServer-fm(0x1e3c700, 0xc0006071d0, 0x1b46320, 0xc0005b83f0, 0xc000620c60, 0xc000620ca0, 0x10, 0x17ddfe0, 0x316da01, 0xffffffffffffffff)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware/logging/logging_interceptor.go:58 +0x73
k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1(0x1e3c700, 0xc0006071d0, 0x1b46320, 0xc0005b83f0, 0x8, 0x0, 0x0, 0x30)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/utils/utils_middleware.go:99 +0x63
k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware/requestid.(*interceptor).handleServer(0xc000045860, 0x1e3c700, 0xc0006071d0, 0x1b46320, 0xc0005b83f0, 0xc000620c60, 0xc000620cc0, 0xc000333918, 0x4d7b4b, 0x1ab6f40, ...)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware/requestid/request_id_injector.go:83 +0x28e
k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware/requestid.(*interceptor).handleServer-fm(0x1e3c700, 0xc0006071d0, 0x1b46320, 0xc0005b83f0, 0xc000620c60, 0xc000620cc0, 0x1880ba0, 0xc0003f57b0, 0x1e3c700, 0xc0006071d0)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware/requestid/request_id_injector.go:24 +0x73
k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1(0x1e3c700, 0xc0006071d0, 0x1b46320, 0xc0005b83f0, 0xc0006071d0, 0x20, 0x20, 0x1a6cc00)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/utils/utils_middleware.go:99 +0x63
k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi.(*StoragePlugin).injectContext(0xc000416000, 0x1e3c700, 0xc000606ea0, 0x1b46320, 0xc0005b83f0, 0xc000620c60, 0xc000620ce0, 0x0, 0x0, 0xc00023a000, ...)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware.go:226 +0xa7
k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi.(*StoragePlugin).injectContext-fm(0x1e3c700, 0xc000606ea0, 0x1b46320, 0xc0005b83f0, 0xc000620c60, 0xc000620ce0, 0x854efa, 0x1a6cc00, 0xc000620d00, 0xc000620c60)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/middleware.go:22 +0x73
k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1(0x1e3c700, 0xc000606ea0, 0x1b46320, 0xc0005b83f0, 0x31ed738, 0xc000333ae8, 0x40c1d8, 0x20)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/utils/utils_middleware.go:99 +0x63
k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/utils.ChainUnaryServer.func2(0x1e3c700, 0xc000606ea0, 0x1b46320, 0xc0005b83f0, 0xc000620c60, 0xc000620c80, 0x1882e80, 0x31ed738, 0x1b7d5e0, 0xc00002c900)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/rexray/gocsi/utils/utils_middleware.go:106 +0xf9
k8s.io/cloud-provider-vsphere/vendor/github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler(0x1b0d460, 0xc00000ab80, 0x1e3c700, 0xc000606ea0, 0xc0005b8380, 0xc00000b340, 0x0, 0x0, 0x30, 0xc0003a6660)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:4581 +0x158
k8s.io/cloud-provider-vsphere/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc000163180, 0x1e492e0, 0xc00066d000, 0xc00002c900, 0xc0003e90e0, 0x31bf9a0, 0x0, 0x0, 0x0)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/google.golang.org/grpc/server.go:1026 +0x4cd
k8s.io/cloud-provider-vsphere/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000163180, 0x1e492e0, 0xc00066d000, 0xc00002c900, 0x0)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/google.golang.org/grpc/server.go:1252 +0x1308
k8s.io/cloud-provider-vsphere/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00060c4f0, 0xc000163180, 0x1e492e0, 0xc00066d000, 0xc00002c900)
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/google.golang.org/grpc/server.go:699 +0x9f
created by k8s.io/cloud-provider-vsphere/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
/Users/trhoden/go/src/k8s.io/cloud-provider-vsphere/vendor/google.golang.org/grpc/server.go:697 +0xa1