vmware-tanzu / velero-plugin-for-vsphere Goto Github PK

View Code? Open in Web Editor NEW

59.0 15.0 50.0 8.96 MB

Plugin to support Velero on vSphere

License: Other

Makefile 0.98% Go 98.29% Shell 0.73%

velero-plugin velero vsphere

velero-plugin-for-vsphere's Introduction

Velero Plugin for vSphere

Overview

This repository contains the Velero Plugin for vSphere. This plugin provides crash-consistent snapshots of vSphere block volumes and backup of volume data into S3 compatible storage.

Releases

For releases, please refer to the releases page.

Kubernetes distributions

Velero Plugin for vSphere supports the following Kubernetes distributions:

For more information on vSphere with Tanzu, (formerly known as vSphere with Kubernetes and Project Pacific), and especially for information about the role of the Supervisor Cluster, please see vSphere with Tanzu Configuration and Management.

For more information on TKGS, the Tanzu Kubernetes Grid Service, please see Provisioning and Managing Tanzu Kubernetes Clusters Using the Tanzu Kubernetes Grid Service.

Important note on vSphere with Tanzu networking requirements

vSphere with Tanzu supports two distinct networking deployments with the release of vSphere 7.0 Update 1 (U1). The first deployment leverages NSX-T to provide load balancing services as well as overlay networking for Pod VM to Pod VM communication in the Supervisor Cluster. The second networking configuration supported by vSphere with Tanzu uses native vSphere network distributed switches and a HA Proxy to provide load balancing services. Native vSphere networking does not provide any overlay networking capabilities, and thus this deployment does not currently support Pod VMs in the Supervisor. Since Pod VMs are a requirement when wish to use the Velero Plugin for vSphere in vSphere with Tanzu, it currently requires a networking deployment that uses NSX-T. vSphere with Tanzu deployments that use native vSphere distributed switch networking and the HA Proxy for load balancing cannot currently be backed up using the Velero Plugin for vSphere.

TKGS is the Tanzu Kubernetes Grid Service, a service available in vSphere with Tanzu to enable the deployment of TKG (guest) clusters in vSphere with Tanzu namespaces. Whilst TKGS is available in both deployments types of vSphere with Tanzu (NSX-T and native vSphere networking), the ability to backup TKGS guest clusters also requires the Supervisor Cluster to have the Velero Plugin for vSphere to be installed in the Supervisor Cluster. Since this is not possible with vSphere with Tanzu which uses native vSphere distributed switch networking, the Velero Plugin for vSphere is currently unable to backup and restore TKG guest clustes in these deployments.

Architecture

Velero handles Kubernetes metadata during backup and retore. Velero relies on its plugin to backup and retore PVCs. Velero Plugin for vSphere includes the following components:

Velero vSphere Operator - a Supervisor Service that helps users install Velero and its vSphere plugin on the Supervisor Cluster; must be enabled through vSphere UI to support backup and restore on Supervisor and Guest Clusters
vSphere Plugin - deployed with Velero; called by Velero to backup and restore a PVC
Backupdriver - handles the backup and restore of PVCs; relies on the Data Mover to upload or download data
Data Mover - handles upload of snapshot data to object store and download of snapshot data from object store
- Note: The Data Mover is only functionally tested and it is not meant to work at scale and does not promise any performance expectations. It is not meant to backup business critical applications in production.

Backup and restore workflows are described in details here.

Velero Plugin for vSphere Installation and Configuration details

For details on how to use Velero Plugin for vSphere for each Kubernetes flavor, refer to the following documents:

Note: Velero needs to be installed in each TKG workload cluster.

Known issues

Known issues are documented here.

Troubleshooting

If you encounter issues, review the troubleshooting docs or file an issue.

Developer Guide

Developer Guide is available at here.

velero-plugin-for-vsphere's People

Contributors

Stargazers

Watchers

velero-plugin-for-vsphere's Issues

Failed to pull image while using internal registry

I'm trying to run the plugin for the backup of my pks with an internal registry but it seems that the pod fails to run due to unsuccessful image pull of "dpcpinternal/data-manager-for-plugin:master-e607c74-13.Apr.2020.00.06.16". Is there any way to get this image pull working with internal registry/air-gapped environment?

$kubectl get pods  
NAMESPACE     NAME                               READY   STATUS             RESTARTS   AGE
velero        datamgr-for-vsphere-plugin-5zz6s   0/1     ImagePullBackOff   0          3m25s
velero        datamgr-for-vsphere-plugin-swmvr   0/1     ImagePullBackOff   0          3m25s

$kubectl describe pod datamgr-for-vsphere-plugin-5zz6s -n velero
Events:
  Type     Reason     Age                             From                                           Message
  ----     ------     ----                            ----                                           -------
  Normal   Scheduled  <invalid>                       default-scheduler                              Successfully assigned velero/datamgr-for-vsphere-plugin-5zz6s to 2ab529d0-803d-457d-a0a0-456218197cfa
  Warning  Failed     <invalid>                       kubelet, 2ab529d0-803d-457d-a0a0-456218197cfa  Failed to pull image "dpcpinternal/data-manager-for-plugin:master-e607c74-13.Apr.2020.00.06.16": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.x.xx.x:53: read udp 10.xx.xx.x:34205->10.xx.xx.xx:53: i/o timeout

steps to reproduce

with a minio s3 up in my cluster, I proceeded with velero installation:

$velero install --bucket velero --provider aws --secret-file credentials-velero --use-volume-snapshots=false --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=$S3_URL,publicUrl=$S3_IP --image $INTERNALREPO/velero/velero:v1.4.2 --plugins $INTERNALREPO/velero/velero-plugin-for-aws:v1.4.2

Next, i added the velero plugin for vsphere as follows:
$velero plugin add $INTERALREPO/vsphereveleroplugin/velero-plugin-for-vsphere:1.0.0

it seems like the velero plugin for vsphere is not added successfully.

$ velero get plugin
NAME                                 KIND
velero.io/crd-remap-version          BackupItemAction
velero.io/pod                        BackupItemAction
velero.io/pv                         BackupItemAction
velero.io/service-account            BackupItemAction
velero.io/aws                        ObjectStore
velero.io/add-pv-from-pvc            RestoreItemAction
velero.io/add-pvc-from-pod           RestoreItemAction
velero.io/change-pvc-node-selector   RestoreItemAction
velero.io/change-storage-class       RestoreItemAction
velero.io/cluster-role-bindings      RestoreItemAction
velero.io/crd-preserve-fields        RestoreItemAction
velero.io/job                        RestoreItemAction
velero.io/pod                        RestoreItemAction
velero.io/restic                     RestoreItemAction
velero.io/role-bindings              RestoreItemAction
velero.io/service                    RestoreItemAction
velero.io/service-account            RestoreItemAction
velero.io/aws                        VolumeSnapshotter

Support insecureSkipTLSVerify for s3 upload

It would be great to support insecureSkipTLSVerify as the aws plugin for uploading to s3 using vsphere-astrolabe-repo.

error code: 14009 for PV restore

Describe the problem/challenge you have

Hello, I can't restore my persistent volumes on my Kubernetes cluster on vSphere 6.7 U3 plateform

I see this error on datamgr-for-vsphere-plugin pod

time="2020-07-21T11:25:05Z" level=error msg="Failed to copy from remote repository." Remote PEID="ivd:9bd1229e-b184-439d-a37d-350bd73d6a1e:a2e0594f-01bb-4b5a-a965-286ff37b4aca" error="copyInt failed: Open virtual disk file failed. The error code is 14009. with error code: 14009" error.file="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity.go:109" error.function=github.com/vmware-tanzu/astrolabe/pkg/ivd.IVDProtectedEntity.getDataWriter logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/dataMover/data_mover.go:118"

Environment:

Velero version : v1.4.2
Kubernetes : v1.18.6
vSphere : 6.7.0, 16075168

I try to restart my plateform : no success.

Daemonset datamgr-for-vsphere-plugin fails when there is no default backup storage location

Is there a hard requirement for velero-plugin-for-vsphere to have a default backup storage location configured?
Environment:

OCP 4.5
vCenter 6.7U3

velero install \
     --plugins velero/velero-plugin-for-aws:v1.0.0 \
     --secret-file ./credentials-velero \
     --use-volume-snapshots=false \
     --no-default-backup-location

All the pods in daemon set datamgr-for-vsphere-plugin fail with below error

time="2020-08-19T02:19:17Z" level=info msg="Starting data manager server  (-)" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/server/server.go:111"
time="2020-08-19T02:19:17Z" level=info msg="data manager server is started" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/server/server.go:174"
time="2020-08-19T02:19:17Z" level=info msg="Retrieved k8s secret, vsphere-config-secret" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:63"
time="2020-08-19T02:19:17Z" level=info msg="DataMover: vSphere VC credential is retrieved" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/dataMover/data_mover.go:42"
time="2020-08-19T02:19:17Z" level=info msg="RetrieveVSLFromVeleroBSLs: Failed to get Velero default backup storage location" error="backupstoragelocations.velero.io \"default\" not found" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:131"
Failed to fire hook: object logged as error does not satisfy error interface
time="2020-08-19T02:19:17Z" level=error msg="RetrieveVSLFromVeleroBSLs: Failed to list Velero default backup storage location" error="<nil>" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:134"
time="2020-08-19T02:19:17Z" level=info msg="DataMover: Velero Backup Storage Location is retrieved, region=<nil>, bucket=<nil>" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/dataMover/data_mover.go:49"
time="2020-08-19T02:19:17Z" level=error msg="No such key region in params map" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:234"
time="2020-08-19T02:19:17Z" level=error msg="Failed to get s3PETM from params map, region=<nil>, bucket=<nil>" error="Missing region param, cannot initialize S3 PETM" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:183" error.function=github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils.GetS3PETMFromParamsMap logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/dataMover/data_mover.go:54"
An error occurred: Missing region param, cannot initialize S3 PETM

When I install velero with a default storage location and then add velero-plugin-for-vsphere plugin, then it works fine.

Persistentvolume not being restored

this is what I do:

velero backup create -w backup-$NOW --snapshot-volumes --volume-snapshot-locations vsl-vsphere --include-namespaces=$NS --include-resources=persistentvolumeclaim,persistentvolume

velero restore create -w --from-backup backup-$NOW --restore-volumes

version:

$ velero version
Client:
        Version: v1.4.2
        Git commit: 56a08a4d695d893f0863f697c2f926e27d70c0c5
Server:
        Version: v1.4.2

I can see Upload object created and in Complete state.

apiVersion: veleroplugin.io/v1
kind: Upload
metadata:
  creationTimestamp: '2020-08-20T13:42:58Z'
  generation: 3
  name: upload-dea042f5-b466-4c0e-9d5e-d13ef702315f
  namespace: velero
  resourceVersion: '67242501'
  selfLink: >-
    /apis/veleroplugin.io/v1/namespaces/velero/uploads/upload-dea042f5-b466-4c0e-9d5e-d13ef702315f
  uid: 1aac86f6-2ac1-461a-8c1e-ca1d3eacf7d9
spec:
  backupTimestamp: '2020-08-20T13:42:58Z'
  snapshotID: >-
    ivd:2579dbe8-6df5-4f21-ae1f-4d10718767bf:dea042f5-b466-4c0e-9d5e-d13ef702315f
status:
  completionTimestamp: '2020-08-20T13:43:19Z'
  message: Upload completed
  nextRetryTimestamp: '2020-08-20T13:42:58Z'
  phase: Completed
  processingNode: worker1
  progress: {}
  startTimestamp: '2020-08-20T13:42:58Z'

This is the log from restore:

$ velero restore logs backup-1597930951-20200820162241
time="2020-08-20T14:22:41Z" level=info msg="starting restore" logSource="pkg/controller/restore_controller.go:453" restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:22:41Z" level=info msg="Starting restore of backup velero/backup-1597930951" logSource="pkg/restore/restore.go:345" restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:22:41Z" level=info msg="Restoring cluster level resource 'persistentvolumes'" logSource="pkg/restore/restore.go:704" restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:22:41Z" level=info msg="Getting client for /v1, Kind=PersistentVolume" logSource="pkg/restore/restore.go:746" restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:32:41Z" level=info msg="Restoring resource 'persistentvolumeclaims' into namespace 'kucera'" logSource="pkg/restore/restore.go:702" restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:32:41Z" level=info msg="Getting client for /v1, Kind=PersistentVolumeClaim" logSource="pkg/restore/restore.go:746" restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:32:41Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:964" restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:32:41Z" level=info msg="Executing AddPVFromPVCAction" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:44" pluginName=velero restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:32:41Z" level=info msg="Adding PV pvc-7bfc65fe-cf6d-4c7b-b988-66bc0e8925d0 as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:66" pluginName=velero restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:32:41Z" level=info msg="Skipping persistentvolumes/pvc-7bfc65fe-cf6d-4c7b-b988-66bc0e8925d0 because it's already been restored." logSource="pkg/restore/restore.go:844" restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:32:41Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:964" restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:32:41Z" level=info msg="Executing ChangePVCNodeSelectorAction" cmd=/velero logSource="pkg/restore/change_pvc_node_selector.go:63" pluginName=velero restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:32:41Z" level=info msg="Done executing ChangePVCNodeSelectorAction" cmd=/velero logSource="pkg/restore/change_pvc_node_selector.go:91" pluginName=velero restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:32:41Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:964" restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:32:41Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:32:41Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:32:41Z" level=info msg="Attempting to restore PersistentVolumeClaim: skusam-csi" logSource="pkg/restore/restore.go:1070" restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:32:42Z" level=info msg="Waiting for all restic restores to complete" logSource="pkg/restore/restore.go:470" restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:32:42Z" level=info msg="Done waiting for all restic restores to complete" logSource="pkg/restore/restore.go:486" restore=velero/backup-1597930951-20200820162241
time="2020-08-20T14:32:42Z" level=info msg="restore completed" logSource="pkg/controller/restore_controller.go:468" restore=velero/backup-1597930951-20200820162241

The problem is that there is no Download object created. And I don't know why restic is mentioned.

Am I doing something wrong?

Wrong X.509 Certificate format

Hello, we are using self-signed certs. but datamgr doesn't seem to be handling them well:

time="2020-08-17T07:15:58Z" level=info msg="Enqueueing upload" controller=upload generation=8 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/upload_controller.go:145" name=upload-5a6806f2-6ecb-48c7-ae18-4d42355bd59f namespace=velero phase=InProgress
2020-08-17T07:15:59.610Z warning -[00017] [Originator@6876 sub=vimaccess] cannot get thumbprint: SSL error code '151441516', exception: 'Wrong X.509 Certificate format'
2020-08-17T07:15:59.611Z warning -[00036] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
time="2020-08-17T07:15:59Z" level=error msg="Failed at copying to remote repository" Local PEID="ivd:2579dbe8-6df5-4f21-ae1f-4d10718767bf:5a6806f2-6ecb-48c7-ae18-4d42355bd59f" error="Prepare for access failed. The error code is 3014. with error code: 3014" error.file="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity.go:70" error.function=github.com/vmware-tanzu/astrolabe/pkg/ivd.IVDProtectedEntity.GetDataReader logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/dataMover/data_mover.go:96"

Any idea how to fix this?

MinIO - x509: certificate signed by unknown authority

When using a local MinIO instance, the vSphere plugin for velero is unable to upload any of the snapshots taken. This is not an issue with the AWS plugin which uploads the rest of the data as it was provided the certificate authority. Does this plugin not support local certificate authorities or self-signed certificates when uploading to MinIO?

apiVersion: datamover.cnsdp.vmware.com/v1alpha1
kind: Upload
metadata:
  creationTimestamp: "2021-01-24T18:31:09Z"
  generation: 5
  labels:
    velero.io/exclude-from-backup: "true"
  managedFields:
  - apiVersion: datamover.cnsdp.vmware.com/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:velero.io/exclude-from-backup: {}
      f:spec:
        .: {}
        f:backupRepository: {}
        f:backupTimestamp: {}
        f:snapshotID: {}
        f:snapshotReference: {}
      f:status:
        .: {}
        f:progress: {}
    manager: backup-driver
    operation: Update
    time: "2021-01-24T18:31:09Z"
  - apiVersion: datamover.cnsdp.vmware.com/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:completionTimestamp: {}
        f:currentBackOff: {}
        f:message: {}
        f:nextRetryTimestamp: {}
        f:phase: {}
        f:processingNode: {}
        f:retryCount: {}
        f:startTimestamp: {}
    manager: datamgr-server
    operation: Update
    time: "2021-01-24T18:31:12Z"
  name: upload-fc6d45e1-6f41-4cf1-9b05-3db91d073cc7
  namespace: velero
  resourceVersion: "116037417"
  selfLink: /apis/datamover.cnsdp.vmware.com/v1alpha1/namespaces/velero/uploads/upload-fc6d45e1-6f41-4cf1-9b05-3db91d073cc7
  uid: 684f703e-3dac-4255-b5d6-c4e05b0a0f56
spec:
  backupRepository: br-2d05c49e-a4ad-4bfb-a128-ea690095a4b1
  backupTimestamp: "2021-01-24T18:31:13Z"
  snapshotID: ivd:c2faeab4-eb9f-4e17-9a3e-da417b09c617:fc6d45e1-6f41-4cf1-9b05-3db91d073cc7
  snapshotReference: harbor-system/snap-c0c44fac-f1a6-4bf3-af6f-5b06932c76f7
status:
  completionTimestamp: "2021-01-24T18:31:22Z"
  currentBackOff: 2
  message: |-
    Failed to upload snapshot, ivd:c2faeab4-eb9f-4e17-9a3e-da417b09c617:fc6d45e1-6f41-4cf1-9b05-3db91d073cc7, to durable object storage. Failed to delete peinfo from bucket "velero": Unable to delete object "plugins/vsphere-astrolabe-repo/ivd/peinfo/ivd:c2faeab4-eb9f-4e17-9a3e-da417b09c617:fc6d45e1-6f41-4cf1-9b05-3db91d073cc7" from bucket "velero": RequestError: send request failed
    caused by: Get https://minio.example.com:9000/velero?prefix=plugins%2Fvsphere-astrolabe-repo%2Fivd%2Fpeinfo%2Fivd%3Ac2faeab4-eb9f-4e17-9a3e-da417b09c617%3Afc6d45e1-6f41-4cf1-9b05-3db91d073cc7: x509: certificate signed by unknown authority
  nextRetryTimestamp: "2021-01-24T18:33:22Z"
  phase: UploadError
  processingNode: kube-node02.example.com
  progress: {}
  retryCount: 2
  startTimestamp: "2021-01-24T18:31:13Z"

Problems restoring stateful sets using velero plugin on vSphere with Tanzu

Describe the bug

I followed the instructions from Cormac Hogan (https://cormachogan.com/2021/03/03/vsphere-with-tanzu-stateful-application-backup-restore-using-velero-vsphere-operator/) to backup and restore a stateful application.
But the restore stucks and never gets finished.

From what I can see in vCenter, the PVC is showing up in the namespace folder, but is never created in container volumes.

To Reproduce

see the attached Readme, YAML and Log files

backup-and-restore-cassandra.zip

cannot use custom certificate in air gapped environment

[A clear and concise description of what the bug is]

after creating the backup with :
velero snapshot-location create vsl-vsphere --provider velero.io/vsphere
the backup completes as expected and the i can see the backup and snapshot files int he bucket.

when trying to restore from backup the restore is stuck on "in progress" and the backup-driver logs:

"send request failed caused by : get https://mys3storage/velero/plugins/vsphere-astrikabe-repo/ivd x509 certificate signed by unknown authority "

[Steps to reproduce the behavior]
create a custom pem encoded ca bundle that includes the s3 storage

I installed velero with :
velero install \ --provider aws \ --plugins velero/velero-plugin-for-aws:v1.1.0 \ --bucket velero2 \ --secret-file ./credentials-velero \ --use-volume-snapshots=true \ --backup-location-config region=default,s3ForcePathStyle="true",s3Url=http://minio.10.x.y.z.nip.io \ --image velero/velero:v1.5.3 \ --snapshot-location-config region="default"

and the plugin with :
velero plugin add vsphereveleroplugin/velero-plugin-for-vsphere:1.1.0

then create the snapshot location :

velero snapshot-location create vsl-vsphere --provider velero.io/vsphere

[A clear and concise description of what you expected to happen]
the restore to work as expected with the custom ca bundle

[Miscellaneous information that will assist in solving the issue]
how do i get the plugin to trust my custom ca bundle

Restore stuck in phase "InProgress" for Vanilla cluster

Describe the bug

Restore stuck in phase "InProgress" for Vanilla cluster.

To Reproduce

succeeds to create backup with snapshot by command velero -n wangc backup create bk --snapshot-volumes=true --include-namespaces nginx-example --include-cluster-resources=false --storage-location=bsl-aws --volume-snapshot-locations=vsl-vsphere
fails to restore the backup with snapshot by command velero -n wangc restore create --from-backup bk

❯ velero -n wangc restore get
NAME                BACKUP   STATUS       STARTED                         COMPLETED                       ERRORS   WARNINGS   CREATED                         SELECTOR
bk-20210615202001   bk       InProgress   2021-06-15 20:20:04 +0800 CST   <nil>                           0        0          2021-06-15 20:20:05 +0800 CST   <none>

Expected behavior
Restore should success if creating backup works.

Troubleshooting Information

[Please refer to the Troubleshooting page and collect the required information]

Velero version: 1.5.3

Velero features (use velero client config get features): features:

velero-plugin-for-vsphere version: v1.1.1

velero/velero-plugin-for-aws: v1.1.0

Kubernetes cluster flavor: Vanilla

vSphere CSI driver version: harbor-repo.vmware.com/velero/backup-driver:1.1.0-rc6
all images in vsphere-csi-controller deployment and vsphere-csi-node daemonset: harbor-repo.vmware.com/velero/data-manager-for-plugin:1.1.0-rc6

Kubernetes version (use kubectl version)
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:18:45Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:13:49Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

vCenter version: 7.0.1.10000
ESXi version: 7.0.1

kubectl -n wangc logs deploy/velero
❯ velero -n wangc backup describe bk
Name:         bk
Namespace:    wangc
Labels:       velero.io/storage-location=bsl-aws
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.16.3
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=16

Phase:  Completed

Errors:    0
Warnings:  0

Namespaces:
  Included:  nginx-example
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  excluded

Label selector:  <none>

Storage Location:  bsl-aws

Velero-Native Snapshot PVs:  true

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2021-06-04 18:39:16 +0800 CST
Completed:  2021-06-04 18:39:30 +0800 CST

Expiration:  2021-07-04 18:39:16 +0800 CST

Total items to be backed up:  9
Items backed up:              9

Velero-Native Snapshots: <none included>

❯ velero -n wangc restore describe bk-20210615202001
Name:         bk-20210615202001
Namespace:    wangc
Labels:       <none>
Annotations:  <none>

Phase:  InProgress

Started:    2021-06-15 20:20:04 +0800 CST
Completed:  <n/a>

Backup:  bk

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

❯ kubectl get crd -A
NAME                                                   CREATED AT
backuprepositories.backupdriver.cnsdp.vmware.com       2021-06-04T07:23:24Z
backuprepositoryclaims.backupdriver.cnsdp.vmware.com   2021-06-04T07:23:24Z
backups.velero.io                                      2021-06-07T01:21:22Z
backupstoragelocations.velero.io                       2021-06-07T01:21:22Z
clonefromsnapshots.backupdriver.cnsdp.vmware.com       2021-06-04T07:23:24Z
deletebackuprequests.velero.io                         2021-06-07T01:21:22Z
deletesnapshots.backupdriver.cnsdp.vmware.com          2021-06-04T07:23:24Z
downloadrequests.velero.io                             2021-06-07T01:21:23Z
downloads.datamover.cnsdp.vmware.com                   2021-06-04T07:23:24Z
podvolumebackups.velero.io                             2021-06-07T01:21:23Z
podvolumerestores.velero.io                            2021-06-07T01:21:23Z
resticrepositories.velero.io                           2021-06-07T01:21:23Z
restores.velero.io                                     2021-06-07T01:21:24Z
schedules.velero.io                                    2021-06-07T01:21:24Z
serverstatusrequests.velero.io                         2021-06-07T01:21:24Z
snapshots.backupdriver.cnsdp.vmware.com                2021-06-04T07:23:24Z
uploads.datamover.cnsdp.vmware.com                     2021-06-04T07:23:25Z
volumesnapshotlocations.velero.io                      2021-06-07T01:21:24Z

Screenshots

[If applicable, add screenshots to help explain your problem.]

Anything else you would like to add:
backup-driver.log.zip
data-mgt.log.zip
velero.log.zip

[Miscellaneous information that will assist in solving the issue]

Airgapped environment setup is not possible

Hey folks i am using velero in an air gapped environment.
I have pull and push all required images in harbor registry and all are working as expected except this vspere plugin.

The problem is

velero plugin add harbor.mylab.local/library/velero-plugin-for-vsphere:1.0.1
data-manager-for-plugin:1.0.1
Failed to pull image "harbor.mylab.local/data-manager-for-plugin:1.0.1": rpc error: code = Unknown desc = failed to pull and unpack image "harbor.mylab.local/data-manager-for-plugin:1.0.1": failed to resolve reference "harbor.mylab.local/data-manager-for-plugin:1.0.1": failed to authorize: failed to fetch anonymous token: unexpected status: 500 Internal Server Error

You see in deployment it is not picking " /Library" part

plugin requires permissions on Vcenter level

Describe the problem/challenge you have
cannot use plugin if permissions are not granted on VCenter level.
since were on shared infrastructure inside our company ESX admins are not able to grant us these permissions

Describe the solution you'd like
a workaround for this issue

Data upload progress hangs up in New phase

I try using this plugin and data upload progress hangs up in New phase without any errors.

My current installation:

Velero 1.4 installed via helm chart

velero	backup-system	2       	2020-06-10 16:51:58.676120722 +0300 MSK	deployed	velero-2.12.0	1.4.0

vSphere version 7.0.0
vSphere CSI driver 1.0.2
Kubenetes v1.17.5

How to reproduce:

I try to backup these persitent volumes:

kubectl get pvc -n artifacts 
NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-harbor-harbor-redis-0               Bound    pvc-7ad38f5e-b249-4036-946e-71ef993e2989   1Gi        RWO            vsphere-fast   18h
data-harbor-harbor-trivy-0               Bound    pvc-74ed3888-6059-4c50-8ea1-42eca02aea4e   5Gi        RWO            vsphere-fast   18h
database-data-harbor-harbor-database-0   Bound    pvc-0501e0bb-b131-4c78-954e-862bb37768fb   2Gi        RWO            vsphere-fast   18h
harbor-harbor-chartmuseum                Bound    pvc-58177c93-e63a-4c6a-87d8-d45047afded2   5Gi        RWO            vsphere-fast   18h
harbor-harbor-jobservice                 Bound    pvc-04d9e434-d51b-47ef-8831-87b5a200eaf3   1Gi        RWO            vsphere-fast   18h
harbor-harbor-registry                   Bound    pvc-9e4a68c0-6f1a-4ca5-957b-bc85a7ac429a   5Gi        RWO            vsphere-fast   18h

Check that a velero-plugin-for-vsphere plugin enabled:

export VELERO_NAMESPACE=backup-system

velero plugin add vsphereveleroplugin/velero-plugin-for-vsphere:1.0.0
An error occurred: Deployment.apps "velero" is invalid: spec.template.spec.initContainers[2].name: Duplicate value: "velero-plugin-for-vsphere"

velero plugin get                                                    
NAME                                 KIND
velero.io/crd-remap-version          BackupItemAction
velero.io/pod                        BackupItemAction
velero.io/pv                         BackupItemAction
velero.io/service-account            BackupItemAction
velero.io/aws                        ObjectStore
velero.io/add-pv-from-pvc            RestoreItemAction
velero.io/add-pvc-from-pod           RestoreItemAction
velero.io/change-pvc-node-selector   RestoreItemAction
velero.io/change-storage-class       RestoreItemAction
velero.io/cluster-role-bindings      RestoreItemAction
velero.io/crd-preserve-fields        RestoreItemAction
velero.io/job                        RestoreItemAction
velero.io/pod                        RestoreItemAction
velero.io/restic                     RestoreItemAction
velero.io/role-bindings              RestoreItemAction
velero.io/service                    RestoreItemAction
velero.io/service-account            RestoreItemAction
velero.io/aws                        VolumeSnapshotter
velero.io/vsphere                    VolumeSnapshotter

Check that snapshot-location enabled as described in README:

velero snapshot-location get 
NAME          PROVIDER
default       aws
vsl-vsphere   velero.io/vsphere

Create backup:

velero backup create my-backup1 --include-namespaces=artifacts --snapshot-volumes --volume-snapshot-locations vsl-vsphere -n backup-system

After that, all upload tasks hang in New phase

kubectl get -n backup-system uploads.veleroplugin.io -o yaml 
apiVersion: v1
items:
- apiVersion: veleroplugin.io/v1
  kind: Upload
  metadata:
    creationTimestamp: "2020-06-11T06:00:39Z"
    generation: 1
    name: upload-22ca9d7b-bc07-4fb2-a89b-f095d3372da4
    namespace: backup-system
    resourceVersion: "6537241"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/uploads/upload-22ca9d7b-bc07-4fb2-a89b-f095d3372da4
    uid: 478e90e7-544f-40dd-b6a9-d5c8734647a1
  spec:
    backupTimestamp: "2020-06-11T06:00:39Z"
    snapshotID: ivd:6b26e118-4442-4e2b-8158-4543e1d894a0:22ca9d7b-bc07-4fb2-a89b-f095d3372da4
  status:
    nextRetryTimestamp: "2020-06-11T06:00:39Z"
    phase: New
    progress: {}
- apiVersion: veleroplugin.io/v1
  kind: Upload
  metadata:
    creationTimestamp: "2020-06-11T06:00:44Z"
    generation: 1
    name: upload-8c3f2e48-090b-4f06-90f4-e11e01a141f6
    namespace: backup-system
    resourceVersion: "6537266"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/uploads/upload-8c3f2e48-090b-4f06-90f4-e11e01a141f6
    uid: 4ae49f43-55bd-4b6b-8ec3-0e8f18104667
  spec:
    backupTimestamp: "2020-06-11T06:00:45Z"
    snapshotID: ivd:1fb69632-cb72-4248-9d53-25cf9b4c6660:8c3f2e48-090b-4f06-90f4-e11e01a141f6
  status:
    nextRetryTimestamp: "2020-06-11T06:00:45Z"
    phase: New
    progress: {}
- apiVersion: veleroplugin.io/v1
  kind: Upload
  metadata:
    creationTimestamp: "2020-06-11T06:00:32Z"
    generation: 1
    name: upload-a14136f2-9843-44a3-b39e-652f710dcc0f
    namespace: backup-system
    resourceVersion: "6537211"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/uploads/upload-a14136f2-9843-44a3-b39e-652f710dcc0f
    uid: f8b616cf-f531-4b96-a14f-4e375c881de6
  spec:
    backupTimestamp: "2020-06-11T06:00:33Z"
    snapshotID: ivd:09ddf908-1f18-4d77-910d-e1958a649052:a14136f2-9843-44a3-b39e-652f710dcc0f
  status:
    nextRetryTimestamp: "2020-06-11T06:00:33Z"
    phase: New
    progress: {}
- apiVersion: veleroplugin.io/v1
  kind: Upload
  metadata:
    creationTimestamp: "2020-06-11T06:00:21Z"
    generation: 1
    name: upload-b04dcafe-e257-4d36-8f05-d92ae754b0d3
    namespace: backup-system
    resourceVersion: "6537165"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/uploads/upload-b04dcafe-e257-4d36-8f05-d92ae754b0d3
    uid: 8183c12d-ce0f-40e9-a43a-0fcfe3c9d332
  spec:
    backupTimestamp: "2020-06-11T06:00:22Z"
    snapshotID: ivd:a4f617bb-8446-491e-8352-386d4408c02b:b04dcafe-e257-4d36-8f05-d92ae754b0d3
  status:
    nextRetryTimestamp: "2020-06-11T06:00:22Z"
    phase: New
    progress: {}
- apiVersion: veleroplugin.io/v1
  kind: Upload
  metadata:
    creationTimestamp: "2020-06-11T06:00:49Z"
    generation: 1
    name: upload-b063d387-fea2-4548-9054-e7b4226609b9
    namespace: backup-system
    resourceVersion: "6537287"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/uploads/upload-b063d387-fea2-4548-9054-e7b4226609b9
    uid: 7dd002ef-3887-4a57-898d-6ff2d20c1e8e
  spec:
    backupTimestamp: "2020-06-11T06:00:50Z"
    snapshotID: ivd:9e6e1200-b9d2-432c-b6e1-8c0016ef2181:b063d387-fea2-4548-9054-e7b4226609b9
  status:
    nextRetryTimestamp: "2020-06-11T06:00:50Z"
    phase: New
    progress: {}
- apiVersion: veleroplugin.io/v1
  kind: Upload
  metadata:
    creationTimestamp: "2020-06-11T06:00:26Z"
    generation: 1
    name: upload-ec2eec23-5ace-4f8d-9985-3b7bf01bf5fb
    namespace: backup-system
    resourceVersion: "6537191"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/uploads/upload-ec2eec23-5ace-4f8d-9985-3b7bf01bf5fb
    uid: 07153f79-4d37-4f8b-b9ef-863aac71ca1e
  spec:
    backupTimestamp: "2020-06-11T06:00:27Z"
    snapshotID: ivd:cfec9a46-ac8a-437c-bbe9-91ba8c715e4c:ec2eec23-5ace-4f8d-9985-3b7bf01bf5fb
  status:
    nextRetryTimestamp: "2020-06-11T06:00:27Z"
    phase: New
    progress: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

I can't find any errors in the velero pod logs

velero.log

Thin-provisioned VMDKs inflated during upload process

Overview

It appears that thin-provisioned VMDKs on the datastore are being inflated during upload process to S3-compatible storage.

Is there any roadmap to being able to upload the thin VMDKs or otherwise save space, or have I otherwise misunderstood?

You can see in the logs and the minio snapshot below that 3G has been uploaded whereas in the vSphere snapshot, the different disk components don't sum up to nearly that much.

Additional information

`velero` deployment images

image: velero/velero:v1.5.2
image: velero/velero-plugin-for-aws:v1.1.0
image: velero/velero-plugin-for-microsoft-azure:v1.1.0
image: vsphereveleroplugin/velero-plugin-for-vsphere:1.1.0
image: velero/velero-plugin-for-csi:v0.1.2

Persistent volume

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    example-vsphere-csi-backup: "yes"
  name: example-vsphere-csi-v2-volume
  namespace: default
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi
  storageClassName: vsphere-csi-v2

Backup

apiVersion: velero.io/v1
kind: Backup
metadata:
  labels:
    velero.io/storage-location: minio-gateway
  name: example-vsphere-csi-v2-backup-2021-02-02t15-48-08-00
  namespace: backup
spec:
  defaultVolumesToRestic: true
  includedNamespaces:
  - default
  labelSelector:
    matchExpressions:
    - key: example-vsphere-csi-backup
      operator: Exists
  storageLocation: minio-gateway
  ttl: 720h0m0s
  volumeSnapshotLocations:
  - vsl-vsphere

`minio` pod and service

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: minio-gateway
  name: minio-gateway
  namespace: backup
spec:
  containers:
  - args:
    - gateway
    - azure
    envFrom:
    - secretRef:
        name: minio-azure-credentials
    image: minio/minio
    imagePullPolicy: Always
    name: minio-gateway
---
apiVersion: v1
kind: Service
metadata:
  name: minio-gateway
  namespace: backup
spec:
  selector:
    run: minio-gateway
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9000

BackupStorageLocation

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: minio-gateway
  namespace: backup
spec:
  backupSyncPeriod: 2m0s
  provider: aws
  objectStorage:
    bucket: velero
    prefix: example-prefix
  config:
    region: minio
    s3ForcePathStyle: "true"
    s3Url: http://minio-gateway.backup.svc

    # AWS profile within the credentials file to use for the backup storage location.
    # This is mounted within the velero pod
    profile: default

velero does not access the vCenter

My env:

Kubernetes and Network Add-on
k8s: v1.18.2
calico: v3.13.3

cpi/csi vSphere
cpi: v1.1.0
csi-provisioner: v1.4.0
csi-driver: v2.0.0-rc.1
csi-attacher: v2.0.0

valero
valero server: v1.3

When I run a backup, valero cannot access the vCenter API. It tries to resolve the name on Kubernetes' internal DNS. I think that in the state valero can't access a resource external to the Kubernetes cluster.

level=error msg="Error at creating new IVD PETM from vcUrl: {https myuser%40coolcorp.priv:mypassword vcenter.coolcorp.priv:443 /sdk false }, s3URLBase: VOID_URL" backup=velero/confluence-backup cmd=/plugins/velero-plugin-for-vsphere error="Post https://vcenter.coolcorp.priv:443/sdk: dial tcp: lookup vcenter.coolcorp.priv on 10.96.0.10:53: server misbehaving" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:178" pluginName=velero-plugin-for-vsphere

Can't restore backup due to download snapshot operation failed with "Failed to download snapshot object storage. copyInt failed: ServerFaultCode: InvalidArgument: is incorrectly formatted." error.

I can't restore backup due to download snapshot operation failed with Failed to download snapshot, ivd:298725a8-07af-4bba-a27c-6f67ef017409:cd1131c2-f438-4059-adbb-1a45a4ef2c00, from durable object storage. copyInt failed: ServerFaultCode: InvalidArgument: is incorrectly formatted. error.

Velero and vsphere configuration

Vsphere version is 7.0.0
VMware ESXi, 6.7.0

Virtual machines placed in DevOps/Kubernetesfolder:

govc ls '/Nord 4/vm/DevOps/Kubernetes'
/Nord 4/vm/DevOps/Kubernetes/k8s-dev1-master1
/Nord 4/vm/DevOps/Kubernetes/k8s-dev1-worker1
/Nord 4/vm/DevOps/Kubernetes/k8s-dev1-worker3
/Nord 4/vm/DevOps/Kubernetes/k8s-dev1-master2
/Nord 4/vm/DevOps/Kubernetes/k8s-dev1-worker2
/Nord 4/vm/DevOps/Kubernetes/k8s-dev1-master3

CNS volumes placed in /Nord 4/datastore/Kubernetes/vol-nord4kub:

govc ls '/Nord 4/datastore/Kubernetes'
/Nord 4/datastore/Kubernetes/vol-nord4kub

Velero and velero plugin for vsphere configuration:

kubectl get deployments.apps velero -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
    deployment.kubernetes.io/revision: "2"
    meta.helm.sh/release-name: velero
    meta.helm.sh/release-namespace: backup-system
creationTimestamp: "2020-06-25T12:45:25Z"
generation: 2
labels:
    app.kubernetes.io/instance: velero
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: velero
    helm.sh/chart: velero-2.12.0
name: velero
namespace: backup-system
resourceVersion: "11633743"
selfLink: /apis/apps/v1/namespaces/backup-system/deployments/velero
uid: 43e0d7ef-60f9-45a0-90b9-e42fc5ef3899
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
    matchLabels:
    app.kubernetes.io/instance: velero
    app.kubernetes.io/name: velero
strategy:
    rollingUpdate:
    maxSurge: 25%
    maxUnavailable: 25%
    type: RollingUpdate
template:
    metadata:
    annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: "8085"
        prometheus.io/scrape: "true"
    creationTimestamp: null
    labels:
        app.kubernetes.io/instance: velero
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: velero
        helm.sh/chart: velero-2.12.0
        name: velero
    spec:
    containers:
    - args:
        - server
        command:
        - /velero
        env:
        - name: VELERO_SCRATCH_DIR
        value: /scratch
        - name: VELERO_NAMESPACE
        valueFrom:
            fieldRef:
            apiVersion: v1
            fieldPath: metadata.namespace
        - name: LD_LIBRARY_PATH
        value: /plugins
        - name: AWS_SHARED_CREDENTIALS_FILE
        value: /credentials/cloud
        image: velero/velero:v1.4.0
        imagePullPolicy: IfNotPresent
        name: velero
        ports:
        - containerPort: 8085
        name: monitoring
        protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /plugins
        name: plugins
        - mountPath: /credentials
        name: cloud-credentials
        - mountPath: /scratch
        name: scratch
    dnsPolicy: ClusterFirst
    initContainers:
    - image: velero/velero-plugin-for-aws:v1.0.1
        imagePullPolicy: IfNotPresent
        name: velero-plugin-for-aws
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /target
        name: plugins
    - image: vsphereveleroplugin/velero-plugin-for-vsphere:1.0.1
        imagePullPolicy: IfNotPresent
        name: velero-plugin-for-vsphere
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /target
        name: plugins
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext: {}
    serviceAccount: velero
    serviceAccountName: velero
    terminationGracePeriodSeconds: 30
    volumes:
    - name: cloud-credentials
        secret:
        defaultMode: 420
        secretName: velero
    - emptyDir: {}
        name: plugins
    - emptyDir: {}
        name: scratch
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2020-06-25T12:45:29Z"
    lastUpdateTime: "2020-06-25T12:45:29Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
- lastTransitionTime: "2020-06-25T12:45:25Z"
    lastUpdateTime: "2020-06-25T12:45:44Z"
    message: ReplicaSet "velero-bcf988bd5" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
observedGeneration: 2
readyReplicas: 1
replicas: 1
updatedReplicas: 1

kubectl get daemonsets.apps datamgr-for-vsphere-plugin -o yaml  
apiVersion: apps/v1
kind: DaemonSet
metadata:
annotations:
    deprecated.daemonset.template.generation: "1"
creationTimestamp: "2020-06-25T12:45:23Z"
generation: 1
labels:
    component: velero
name: datamgr-for-vsphere-plugin
namespace: backup-system
resourceVersion: "11633689"
selfLink: /apis/apps/v1/namespaces/backup-system/daemonsets/datamgr-for-vsphere-plugin
uid: 12b587f7-5710-4f70-86b0-bbecd5424d9e
spec:
revisionHistoryLimit: 10
selector:
    matchLabels:
    name: datamgr-for-vsphere-plugin
template:
    metadata:
    creationTimestamp: null
    labels:
        component: velero
        name: datamgr-for-vsphere-plugin
    spec:
    containers:
    - args:
        - server
        command:
        - /datamgr
        env:
        - name: NODE_NAME
        valueFrom:
            fieldRef:
            apiVersion: v1
            fieldPath: spec.nodeName
        - name: VELERO_NAMESPACE
        valueFrom:
            fieldRef:
            apiVersion: v1
            fieldPath: metadata.namespace
        - name: VELERO_SCRATCH_DIR
        value: /scratch
        - name: LD_LIBRARY_PATH
        value: /vddkLibs
        - name: GOOGLE_APPLICATION_CREDENTIALS
        value: /credentials/cloud
        - name: AWS_SHARED_CREDENTIALS_FILE
        value: /credentials/cloud
        - name: AZURE_CREDENTIALS_FILE
        value: /credentials/cloud
        image: vsphereveleroplugin/data-manager-for-plugin:1.0.1
        imagePullPolicy: IfNotPresent
        name: datamgr-for-vsphere-plugin
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /plugins
        name: plugins
        - mountPath: /scratch
        name: scratch
        - mountPath: /credentials
        name: cloud-credentials
    dnsPolicy: ClusterFirst
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext:
        runAsUser: 0
    serviceAccount: velero
    serviceAccountName: velero
    terminationGracePeriodSeconds: 30
    volumes:
    - emptyDir: {}
        name: plugins
    - emptyDir: {}
        name: scratch
    - name: cloud-credentials
        secret:
        defaultMode: 420
        secretName: cloud-credentials
updateStrategy:
    rollingUpdate:
    maxUnavailable: 1
    type: RollingUpdate
status:
currentNumberScheduled: 6
desiredNumberScheduled: 6
numberAvailable: 6
numberMisscheduled: 0
numberReady: 6
observedGeneration: 1
updatedNumberScheduled: 6

How to reproduce

I using harbor as a test application, which I trying to backup:

kubectl get pods -n artifacts
NAME                                         READY   STATUS    RESTARTS   AGE
harbor-harbor-chartmuseum-5c44474cc4-rwsnr   1/1     Running   0          5m53s
harbor-harbor-clair-795c78f4b7-wtlkg         2/2     Running   3          5m53s
harbor-harbor-core-584577c87c-vwf9f          1/1     Running   0          5m53s
harbor-harbor-database-0                     1/1     Running   0          5m53s
harbor-harbor-jobservice-56ff4cff7-4kwzk     1/1     Running   0          5m53s
harbor-harbor-portal-6f7f6d5469-mnddx        1/1     Running   0          5m53s
harbor-harbor-redis-0                        1/1     Running   0          5m53s
harbor-harbor-registry-785dff9499-rmz8r      2/2     Running   0          5m53s
harbor-harbor-trivy-0                        1/1     Running   0          5m53s

kubectl get pvc -n artifacts
NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-harbor-harbor-redis-0               Bound    pvc-a8b6e6ef-890c-4bc9-8fa6-1fe2edff81f5   1Gi        RWO            vsphere-fast   6m32s
data-harbor-harbor-trivy-0               Bound    pvc-d1ffaa34-5e2b-4422-bee6-730a103e6069   5Gi        RWO            vsphere-fast   6m32s
database-data-harbor-harbor-database-0   Bound    pvc-26b28bee-eb03-4280-8847-b8bcb8a94881   2Gi        RWO            vsphere-fast   6m32s
harbor-harbor-chartmuseum                Bound    pvc-2bd01051-c511-40e4-9d8e-80408fe67a8e   5Gi        RWO            vsphere-fast   6m33s
harbor-harbor-jobservice                 Bound    pvc-63b78814-96ad-4c82-9014-67b488b9ca62   1Gi        RWO            vsphere-fast   6m33s
harbor-harbor-registry                   Bound    pvc-3a863f4b-2a1d-4d3c-9f6a-480530a7065a   5Gi        RWO            vsphere-fast   6m33s

kubectl get pv | grep artifacts
pvc-26b28bee-eb03-4280-8847-b8bcb8a94881   2Gi        RWO            Delete           Bound    artifacts/database-data-harbor-harbor-database-0   vsphere-fast            6m36s
pvc-2bd01051-c511-40e4-9d8e-80408fe67a8e   5Gi        RWO            Delete           Bound    artifacts/harbor-harbor-chartmuseum                vsphere-fast            6m39s
pvc-3a863f4b-2a1d-4d3c-9f6a-480530a7065a   5Gi        RWO            Delete           Bound    artifacts/harbor-harbor-registry                   vsphere-fast            6m45s
pvc-63b78814-96ad-4c82-9014-67b488b9ca62   1Gi        RWO            Delete           Bound    artifacts/harbor-harbor-jobservice                 vsphere-fast            6m43s
pvc-a8b6e6ef-890c-4bc9-8fa6-1fe2edff81f5   1Gi        RWO            Delete           Bound    artifacts/data-harbor-harbor-redis-0               vsphere-fast            6m39s
pvc-d1ffaa34-5e2b-4422-bee6-730a103e6069   5Gi        RWO            Delete           Bound    artifacts/data-harbor-harbor-trivy-0               vsphere-fast            6m40s

Create a backup:

export VELERO_NAMESPACE=backup-system

velero backup create artifacts-4 --include-namespaces=artifacts --snapshot-volumes --volume-snapshot-locations vsl-vsphere -n backup-system

Check that the backup status is ok:

velero backup describe artifacts-4          
Name:         artifacts-4
Namespace:    backup-system
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.17.5
            velero.io/source-cluster-k8s-major-version=1
            velero.io/source-cluster-k8s-minor-version=17

Phase:  Completed

Namespaces:
Included:  artifacts
Excluded:  <none>

Resources:
Included:        *
Excluded:        <none>
Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Snapshot PVs:  true

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2020-06-25 15:53:55 +0300 MSK
Completed:  2020-06-25 15:55:09 +0300 MSK

Expiration:  2020-07-25 15:53:55 +0300 MSK

Persistent Volumes:  6 of 6 snapshots completed successfully (specify --details for more information)

kubectl -n backup-system get uploads.veleroplugin.io -o yaml
apiVersion: v1
items:
- apiVersion: veleroplugin.io/v1
kind: Upload
metadata:
    creationTimestamp: "2020-06-25T12:54:27Z"
    generation: 3
    name: upload-198e2f5a-11dd-4631-9f3f-39fe567d27a9
    namespace: backup-system
    resourceVersion: "11637089"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/uploads/upload-198e2f5a-11dd-4631-9f3f-39fe567d27a9
    uid: a445993c-acd1-4784-8a12-3a069c1e1432
spec:
    backupTimestamp: "2020-06-25T12:54:30Z"
    snapshotID: ivd:0cb56d79-1234-4eec-b002-f8a7a9334c19:198e2f5a-11dd-4631-9f3f-39fe567d27a9
status:
    completionTimestamp: "2020-06-25T12:57:48Z"
    message: Upload completed
    nextRetryTimestamp: "2020-06-25T12:54:30Z"
    phase: Completed
    processingNode: k8s-dev1-worker1
    progress: {}
    startTimestamp: "2020-06-25T12:57:06Z"
- apiVersion: veleroplugin.io/v1
kind: Upload
metadata:
    creationTimestamp: "2020-06-23T15:36:02Z"
    generation: 3
    name: upload-1c178c63-5785-4be5-8b12-7bbd75895a18
    namespace: backup-system
    resourceVersion: "10953393"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/uploads/upload-1c178c63-5785-4be5-8b12-7bbd75895a18
    uid: a2544d42-d35f-4a4e-9421-a26f6e8acd4a
spec:
    backupTimestamp: "2020-06-23T15:36:10Z"
    snapshotID: ivd:b0748c62-ade1-470e-857a-cb910c669285:1c178c63-5785-4be5-8b12-7bbd75895a18
status:
    completionTimestamp: "2020-06-23T15:38:28Z"
    message: Upload completed
    nextRetryTimestamp: "2020-06-23T15:36:10Z"
    phase: Completed
    processingNode: k8s-dev1-worker1
    progress: {}
    startTimestamp: "2020-06-23T15:36:17Z"
- apiVersion: veleroplugin.io/v1
kind: Upload
metadata:
    creationTimestamp: "2020-06-23T15:36:09Z"
    generation: 3
    name: upload-2093a61c-5bd2-4e02-9e64-1dd7405a8b3a
    namespace: backup-system
    resourceVersion: "10953419"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/uploads/upload-2093a61c-5bd2-4e02-9e64-1dd7405a8b3a
    uid: 700d7255-82d9-4624-a5a1-b89b6abb24be
spec:
    backupTimestamp: "2020-06-23T15:36:16Z"
    snapshotID: ivd:6b0c8d8e-f69b-460c-82ae-5f82f3d10e64:2093a61c-5bd2-4e02-9e64-1dd7405a8b3a
status:
    completionTimestamp: "2020-06-23T15:38:29Z"
    message: Upload completed
    nextRetryTimestamp: "2020-06-23T15:36:16Z"
    phase: Completed
    processingNode: k8s-dev1-worker3
    progress: {}
    startTimestamp: "2020-06-23T15:36:20Z"
- apiVersion: veleroplugin.io/v1
kind: Upload
metadata:
    creationTimestamp: "2020-06-25T12:54:34Z"
    generation: 3
    name: upload-32e81218-d487-4f43-bf6c-11471b113b7d
    namespace: backup-system
    resourceVersion: "11637391"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/uploads/upload-32e81218-d487-4f43-bf6c-11471b113b7d
    uid: eb8d02b3-ba1c-4f08-a539-cc5a5c5d98f5
spec:
    backupTimestamp: "2020-06-25T12:54:37Z"
    snapshotID: ivd:1b1edb6b-4d8b-4548-b3f6-8fc8948b8b1b:32e81218-d487-4f43-bf6c-11471b113b7d
status:
    completionTimestamp: "2020-06-25T12:58:48Z"
    message: Upload completed
    nextRetryTimestamp: "2020-06-25T12:54:37Z"
    phase: Completed
    processingNode: k8s-dev1-worker2
    progress: {}
    startTimestamp: "2020-06-25T12:55:58Z"
- apiVersion: veleroplugin.io/v1
kind: Upload
metadata:
    creationTimestamp: "2020-06-23T15:36:31Z"
    generation: 3
    name: upload-341bc779-4d03-4ebb-9ff1-0d9dd2a91be6
    namespace: backup-system
    resourceVersion: "10954357"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/uploads/upload-341bc779-4d03-4ebb-9ff1-0d9dd2a91be6
    uid: b8aef8e7-c9ac-47ec-99b0-a4845914e482
spec:
    backupTimestamp: "2020-06-23T15:36:39Z"
    snapshotID: ivd:6b8fc1bd-67da-4f61-a085-a0a97c821f43:341bc779-4d03-4ebb-9ff1-0d9dd2a91be6
status:
    completionTimestamp: "2020-06-23T15:41:53Z"
    message: Upload completed
    nextRetryTimestamp: "2020-06-23T15:36:39Z"
    phase: Completed
    processingNode: k8s-dev1-worker2
    progress: {}
    startTimestamp: "2020-06-23T15:39:24Z"
- apiVersion: veleroplugin.io/v1
kind: Upload
metadata:
    creationTimestamp: "2020-06-25T12:54:18Z"
    generation: 3
    name: upload-3f58e523-dbcd-490e-a101-784c28a8e0a5
    namespace: backup-system
    resourceVersion: "11636600"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/uploads/upload-3f58e523-dbcd-490e-a101-784c28a8e0a5
    uid: a4482072-874c-4481-a4c3-15685fc18dd6
spec:
    backupTimestamp: "2020-06-25T12:54:21Z"
    snapshotID: ivd:844aeeb6-e066-4fc3-b9ff-47c0bf276e66:3f58e523-dbcd-490e-a101-784c28a8e0a5
status:
    completionTimestamp: "2020-06-25T12:55:58Z"
    message: Upload completed
    nextRetryTimestamp: "2020-06-25T12:54:21Z"
    phase: Completed
    processingNode: k8s-dev1-worker2
    progress: {}
    startTimestamp: "2020-06-25T12:54:34Z"
- apiVersion: veleroplugin.io/v1
kind: Upload
metadata:
    creationTimestamp: "2020-06-23T15:36:20Z"
    generation: 3
    name: upload-5dd26825-7010-4693-8ce1-5787683472a5
    namespace: backup-system
    resourceVersion: "10953862"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/uploads/upload-5dd26825-7010-4693-8ce1-5787683472a5
    uid: 48d6c41c-f15e-4f5b-8913-f3fad6ef8e8d
spec:
    backupTimestamp: "2020-06-23T15:36:28Z"
    snapshotID: ivd:2f70d796-be4a-4c33-bf08-4cacf7e3cddd:5dd26825-7010-4693-8ce1-5787683472a5
status:
    completionTimestamp: "2020-06-23T15:40:09Z"
    message: Upload completed
    nextRetryTimestamp: "2020-06-23T15:36:28Z"
    phase: Completed
    processingNode: k8s-dev1-worker1
    progress: {}
    startTimestamp: "2020-06-23T15:39:18Z"
- apiVersion: veleroplugin.io/v1
kind: Upload
metadata:
    creationTimestamp: "2020-06-23T15:36:26Z"
    generation: 3
    name: upload-949b2476-3800-4990-b0d8-70ec9680f0cc
    namespace: backup-system
    resourceVersion: "10953667"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/uploads/upload-949b2476-3800-4990-b0d8-70ec9680f0cc
    uid: bd32cd69-1d6f-4715-80dd-bbe2d06c4121
spec:
    backupTimestamp: "2020-06-23T15:36:34Z"
    snapshotID: ivd:f5920044-a3ff-4ce8-a349-f5f63211894d:949b2476-3800-4990-b0d8-70ec9680f0cc
status:
    completionTimestamp: "2020-06-23T15:39:19Z"
    message: Upload completed
    nextRetryTimestamp: "2020-06-23T15:36:34Z"
    phase: Completed
    processingNode: k8s-dev1-worker2
    progress: {}
    startTimestamp: "2020-06-23T15:36:34Z"
- apiVersion: veleroplugin.io/v1
kind: Upload
metadata:
    creationTimestamp: "2020-06-25T12:54:02Z"
    generation: 3
    name: upload-a3c94241-cbda-49b1-85c7-ae3f30eeb466
    namespace: backup-system
    resourceVersion: "11636627"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/uploads/upload-a3c94241-cbda-49b1-85c7-ae3f30eeb466
    uid: b98cf42a-a663-43b3-8a1f-b9d2f5e28707
spec:
    backupTimestamp: "2020-06-25T12:54:05Z"
    snapshotID: ivd:50f27094-d730-46f9-80be-944ee75e85b6:a3c94241-cbda-49b1-85c7-ae3f30eeb466
status:
    completionTimestamp: "2020-06-25T12:56:11Z"
    message: Upload completed
    nextRetryTimestamp: "2020-06-25T12:54:05Z"
    phase: Completed
    processingNode: k8s-dev1-worker1
    progress: {}
    startTimestamp: "2020-06-25T12:54:05Z"
- apiVersion: veleroplugin.io/v1
kind: Upload
metadata:
    creationTimestamp: "2020-06-23T15:36:14Z"
    generation: 3
    name: upload-ba158e6f-7c61-4f18-9af5-47d51a86326a
    namespace: backup-system
    resourceVersion: "10953628"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/uploads/upload-ba158e6f-7c61-4f18-9af5-47d51a86326a
    uid: 63820c82-0532-4bf1-ace8-df2faca319e6
spec:
    backupTimestamp: "2020-06-23T15:36:22Z"
    snapshotID: ivd:7ce568e0-8481-47ec-9191-a6c4e2e23f2d:ba158e6f-7c61-4f18-9af5-47d51a86326a
status:
    completionTimestamp: "2020-06-23T15:39:18Z"
    message: Upload completed
    nextRetryTimestamp: "2020-06-23T15:36:22Z"
    phase: Completed
    processingNode: k8s-dev1-worker1
    progress: {}
    startTimestamp: "2020-06-23T15:38:28Z"
- apiVersion: veleroplugin.io/v1
kind: Upload
metadata:
    creationTimestamp: "2020-06-25T12:54:09Z"
    generation: 3
    name: upload-cd1131c2-f438-4059-adbb-1a45a4ef2c00
    namespace: backup-system
    resourceVersion: "11636893"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/uploads/upload-cd1131c2-f438-4059-adbb-1a45a4ef2c00
    uid: a7c6b59f-aa85-4681-b0cf-81fd978bc32e
spec:
    backupTimestamp: "2020-06-25T12:54:12Z"
    snapshotID: ivd:298725a8-07af-4bba-a27c-6f67ef017409:cd1131c2-f438-4059-adbb-1a45a4ef2c00
status:
    completionTimestamp: "2020-06-25T12:57:06Z"
    message: Upload completed
    nextRetryTimestamp: "2020-06-25T12:54:12Z"
    phase: Completed
    processingNode: k8s-dev1-worker1
    progress: {}
    startTimestamp: "2020-06-25T12:56:11Z"
- apiVersion: veleroplugin.io/v1
kind: Upload
metadata:
    creationTimestamp: "2020-06-25T12:55:05Z"
    generation: 3
    name: upload-e456e8d6-506b-4aaf-a8ee-c77a31494bca
    namespace: backup-system
    resourceVersion: "11637248"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/uploads/upload-e456e8d6-506b-4aaf-a8ee-c77a31494bca
    uid: 1475517a-0dbc-4960-b4ff-8927e6413cce
spec:
    backupTimestamp: "2020-06-25T12:55:08Z"
    snapshotID: ivd:4493b048-f5b6-455d-aacc-eb34828173c1:e456e8d6-506b-4aaf-a8ee-c77a31494bca
status:
    completionTimestamp: "2020-06-25T12:58:18Z"
    message: Upload completed
    nextRetryTimestamp: "2020-06-25T12:55:08Z"
    phase: Completed
    processingNode: k8s-dev1-worker3
    progress: {}
    startTimestamp: "2020-06-25T12:55:08Z"
kind: List
metadata:
resourceVersion: ""
selfLink: ""

Simulate a disaster:

kubectl delete ns artifacts
namespace "artifacts" deleted

Try to restore from the backup:

velero restore create --from-backup artifacts-4
Restore request "artifacts-4-20200625161007" submitted successfully.
Run `velero restore describe artifacts-4-20200625161007` or `velero restore logs artifacts-4-20200625161007` for more details.

Check restore status and volume snapshot download status:

velero restore get
NAME                         BACKUP        STATUS            WARNINGS   ERRORS   CREATED                         SELECTOR
artifacts-4-20200625161007   artifacts-4   PartiallyFailed   0          6        2020-06-25 16:10:12 +0300 MSK   <none>

kubectl get downloads.veleroplugin.io  -o yaml 
apiVersion: v1
items:
- apiVersion: veleroplugin.io/v1
kind: Download
metadata:
    creationTimestamp: "2020-06-25T14:20:23Z"
    generation: 15
    name: download-198e2f5a-11dd-4631-9f3f-39fe567d27a9-91faad80-73f7-476c-945f-6da5b9563f9b
    namespace: backup-system
    resourceVersion: "11663107"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/downloads/download-198e2f5a-11dd-4631-9f3f-39fe567d27a9-91faad80-73f7-476c-945f-6da5b9563f9b
    uid: 8c9c7bfe-8ac0-4ff0-bdd5-8ea3d1f03c67
spec:
    restoreTimestamp: "2020-06-25T14:20:40Z"
    snapshotID: ivd:0cb56d79-1234-4eec-b002-f8a7a9334c19:198e2f5a-11dd-4631-9f3f-39fe567d27a9
status:
    completionTimestamp: "2020-06-25T14:40:45Z"
    message: 'Failed to download snapshot, ivd:0cb56d79-1234-4eec-b002-f8a7a9334c19:198e2f5a-11dd-4631-9f3f-39fe567d27a9,
    from durable object storage. copyInt failed: ServerFaultCode: InvalidArgument:  is
    incorrectly formatted.'
    nextRetryTimestamp: "2020-06-25T14:40:44Z"
    phase: Failed
    processingNode: k8s-dev1-worker1
    progress: {}
    retryCount: 6
    startTimestamp: "2020-06-25T14:20:40Z"
- apiVersion: veleroplugin.io/v1
kind: Download
metadata:
    creationTimestamp: "2020-06-25T13:40:11Z"
    generation: 15
    name: download-32e81218-d487-4f43-bf6c-11471b113b7d-7ce9e5d0-d284-411a-9933-aa1680db122a
    namespace: backup-system
    resourceVersion: "11653099"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/downloads/download-32e81218-d487-4f43-bf6c-11471b113b7d-7ce9e5d0-d284-411a-9933-aa1680db122a
    uid: 7563382d-1c10-4518-b7aa-68fe6e1ea49d
spec:
    restoreTimestamp: "2020-06-25T13:40:28Z"
    snapshotID: ivd:1b1edb6b-4d8b-4548-b3f6-8fc8948b8b1b:32e81218-d487-4f43-bf6c-11471b113b7d
status:
    completionTimestamp: "2020-06-25T14:00:33Z"
    message: 'Failed to download snapshot, ivd:1b1edb6b-4d8b-4548-b3f6-8fc8948b8b1b:32e81218-d487-4f43-bf6c-11471b113b7d,
    from durable object storage. copyInt failed: ServerFaultCode: InvalidArgument:  is
    incorrectly formatted.'
    nextRetryTimestamp: "2020-06-25T14:00:32Z"
    phase: Failed
    processingNode: k8s-dev1-worker1
    progress: {}
    retryCount: 6
    startTimestamp: "2020-06-25T13:40:28Z"
- apiVersion: veleroplugin.io/v1
kind: Download
metadata:
    creationTimestamp: "2020-06-25T12:40:56Z"
    generation: 15
    name: download-341bc779-4d03-4ebb-9ff1-0d9dd2a91be6-814b31ba-ec71-42ce-8da0-d9e51dab280d
    namespace: backup-system
    resourceVersion: "11635429"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/downloads/download-341bc779-4d03-4ebb-9ff1-0d9dd2a91be6-814b31ba-ec71-42ce-8da0-d9e51dab280d
    uid: 81f18386-443b-49f2-b20f-d27e59819392
spec:
    restoreTimestamp: "2020-06-25T12:41:10Z"
    snapshotID: ivd:6b8fc1bd-67da-4f61-a085-a0a97c821f43:341bc779-4d03-4ebb-9ff1-0d9dd2a91be6
status:
    completionTimestamp: "2020-06-25T12:51:37Z"
    message: "Failed to download snapshot, ivd:6b8fc1bd-67da-4f61-a085-a0a97c821f43:341bc779-4d03-4ebb-9ff1-0d9dd2a91be6,
    from durable object storage. GetObject failed for bucket velero, key plugins/vsphere-astrolabe-repo/ivd/peinfo/ivd:6b8fc1bd-67da-4f61-a085-a0a97c821f43:341bc779-4d03-4ebb-9ff1-0d9dd2a91be6:
    NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id:
    161BCA752D4850A3, host id: "
    nextRetryTimestamp: "2020-06-25T12:56:37Z"
    phase: Failed
    processingNode: k8s-dev1-worker3
    progress: {}
    retryCount: 6
    startTimestamp: "2020-06-25T12:41:10Z"
- apiVersion: veleroplugin.io/v1
kind: Download
metadata:
    creationTimestamp: "2020-06-25T14:00:17Z"
    generation: 15
    name: download-3f58e523-dbcd-490e-a101-784c28a8e0a5-bfb60b4f-4e28-481b-ae85-c371f508336d
    namespace: backup-system
    resourceVersion: "11658098"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/downloads/download-3f58e523-dbcd-490e-a101-784c28a8e0a5-bfb60b4f-4e28-481b-ae85-c371f508336d
    uid: 1f55e145-971b-4d67-a6a1-ee4f689b9e62
spec:
    restoreTimestamp: "2020-06-25T14:00:34Z"
    snapshotID: ivd:844aeeb6-e066-4fc3-b9ff-47c0bf276e66:3f58e523-dbcd-490e-a101-784c28a8e0a5
status:
    completionTimestamp: "2020-06-25T14:20:39Z"
    message: 'Failed to download snapshot, ivd:844aeeb6-e066-4fc3-b9ff-47c0bf276e66:3f58e523-dbcd-490e-a101-784c28a8e0a5,
    from durable object storage. copyInt failed: ServerFaultCode: InvalidArgument:  is
    incorrectly formatted.'
    nextRetryTimestamp: "2020-06-25T14:20:38Z"
    phase: Failed
    processingNode: k8s-dev1-worker1
    progress: {}
    retryCount: 6
    startTimestamp: "2020-06-25T14:00:34Z"
- apiVersion: veleroplugin.io/v1
kind: Download
metadata:
    creationTimestamp: "2020-06-25T13:25:05Z"
    generation: 15
    name: download-a3c94241-cbda-49b1-85c7-ae3f30eeb466-1bd36330-292f-4b51-b66f-1cda295e9575
    namespace: backup-system
    resourceVersion: "11648082"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/downloads/download-a3c94241-cbda-49b1-85c7-ae3f30eeb466-1bd36330-292f-4b51-b66f-1cda295e9575
    uid: 75656b23-2e6e-4709-8480-cee8cdb16c4c
spec:
    restoreTimestamp: "2020-06-25T13:25:22Z"
    snapshotID: ivd:50f27094-d730-46f9-80be-944ee75e85b6:a3c94241-cbda-49b1-85c7-ae3f30eeb466
status:
    completionTimestamp: "2020-06-25T13:40:27Z"
    message: 'Failed to download snapshot, ivd:50f27094-d730-46f9-80be-944ee75e85b6:a3c94241-cbda-49b1-85c7-ae3f30eeb466,
    from durable object storage. copyInt failed: ServerFaultCode: InvalidArgument:  is
    incorrectly formatted.'
    nextRetryTimestamp: "2020-06-25T13:45:26Z"
    phase: Failed
    processingNode: k8s-dev1-worker1
    progress: {}
    retryCount: 6
    startTimestamp: "2020-06-25T13:25:22Z"
- apiVersion: veleroplugin.io/v1
kind: Download
metadata:
    creationTimestamp: "2020-06-25T13:09:59Z"
    generation: 15
    name: download-cd1131c2-f438-4059-adbb-1a45a4ef2c00-f4db3609-4546-4da4-9edf-0f9e90eea1b0
    namespace: backup-system
    resourceVersion: "11644318"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/downloads/download-cd1131c2-f438-4059-adbb-1a45a4ef2c00-f4db3609-4546-4da4-9edf-0f9e90eea1b0
    uid: 6a9cc449-2dbb-4d75-a405-23e18c5048aa
spec:
    restoreTimestamp: "2020-06-25T13:10:15Z"
    snapshotID: ivd:298725a8-07af-4bba-a27c-6f67ef017409:cd1131c2-f438-4059-adbb-1a45a4ef2c00
status:
    completionTimestamp: "2020-06-25T13:25:21Z"
    message: 'Failed to download snapshot, ivd:298725a8-07af-4bba-a27c-6f67ef017409:cd1131c2-f438-4059-adbb-1a45a4ef2c00,
    from durable object storage. copyInt failed: ServerFaultCode: InvalidArgument:  is
    incorrectly formatted.'
    nextRetryTimestamp: "2020-06-25T13:30:20Z"
    phase: Failed
    processingNode: k8s-dev1-worker1
    progress: {}
    retryCount: 6
    startTimestamp: "2020-06-25T13:10:15Z"
- apiVersion: veleroplugin.io/v1
kind: Download
metadata:
    creationTimestamp: "2020-06-25T14:40:29Z"
    generation: 15
    name: download-e456e8d6-506b-4aaf-a8ee-c77a31494bca-ddc3880f-416e-4ab8-bb08-29c8aa1daccc
    namespace: backup-system
    resourceVersion: "11668113"
    selfLink: /apis/veleroplugin.io/v1/namespaces/backup-system/downloads/download-e456e8d6-506b-4aaf-a8ee-c77a31494bca-ddc3880f-416e-4ab8-bb08-29c8aa1daccc
    uid: d09ebbc3-3484-431c-be6b-402164076521
spec:
    restoreTimestamp: "2020-06-25T14:40:46Z"
    snapshotID: ivd:4493b048-f5b6-455d-aacc-eb34828173c1:e456e8d6-506b-4aaf-a8ee-c77a31494bca
status:
    completionTimestamp: "2020-06-25T15:00:52Z"
    message: 'Failed to download snapshot, ivd:4493b048-f5b6-455d-aacc-eb34828173c1:e456e8d6-506b-4aaf-a8ee-c77a31494bca,
    from durable object storage. copyInt failed: ServerFaultCode: InvalidArgument:  is
    incorrectly formatted.'
    nextRetryTimestamp: "2020-06-25T15:00:50Z"
    phase: Failed
    processingNode: k8s-dev1-worker1
    progress: {}
    retryCount: 6
    startTimestamp: "2020-06-25T14:40:46Z"
kind: List
metadata:
resourceVersion: ""
selfLink: ""

Full backup and restore logs:

velero.log
datamgr-for-vsphere-plugin.log

Datamgr-for-vsphere-plugin PODS are crashing (see attached logs)

I have installed Velero 1.4 AWS plugin 1.1.0 with 'restic' plugin and MinIO for local object store into VMware PKS (TGKI cluster).
Followed instructions in the very helpful blog: https://beyondelastic.com/2020/04/30/backup-and-migrate-tkgi-pks-to-tkg-with-velero/

I have configured vSphere snapshot plugin in the cluster that has CSI driver enabled:

velero plugin add vsphereveleroplugin/velero-plugin-for-vsphere:1.0.0

and ran successful backup creation using command:

velero backup create my-ghost-backup --include-namespaces=ghost --snapshot-volumes --volume-snapshot-locations

Backup were created successfully AFAIK:

velero backup describe my-ghost-backup-new
Name: my-ghost-backup-new
Namespace: velero
Labels: velero.io/storage-location=default
Annotations: velero.io/source-cluster-k8s-gitversion=v1.16.7+vmware.1
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=16

Phase: Completed

Namespaces:
Included: ghost
Excluded:

Resources:
Included: *
Excluded:
Cluster-scoped: auto

Label selector:

Storage Location: default

Velero-Native Snapshot PVs: true

TTL: 720h0m0s

Hooks:

Backup Format Version: 1

Started: 2020-06-15 16:47:41 +0000 UTC
Completed: 2020-06-15 16:47:57 +0000 UTC

Expiration: 2020-07-15 16:47:41 +0000 UTC

Total items to be backed up: 10
Items backed up: 10

Velero-Native Snapshots: 1 of 1 snapshots completed successfully (specify --details for more information)

However, deleting app names[ace and trying to restore from that backup consistently were unsuccessful. Upon examination, I have found out that 'datamgr-for-vsphere-plugin' pods are consistently crashing (see attached log files
datamgr-for-vsphere-plugin-crashed.txt
datamgr-for-vsphere-plugin-p9rxj-crashed.txt

VELERO_INSTALL_LOCAL_MINIO.txt

)

kubectl get po -n velero
NAME READY STATUS RESTARTS AGE
datamgr-for-vsphere-plugin-7zzfk 0/1 CrashLoopBackOff 3563 14d
datamgr-for-vsphere-plugin-kn5mw 0/1 CrashLoopBackOff 3567 14d
datamgr-for-vsphere-plugin-p9rxj 0/1 CrashLoopBackOff 3555 14d
datamgr-for-vsphere-plugin-zfp44 0/1 CrashLoopBackOff 3560 14d
minio-845c8c775b-cskd4 1/1 Running 0 16d
restic-4qsbf 1/1 Running 0 14d
restic-8jj6g 1/1 Running 0 16d
restic-cxplm 1/1 Running 0 16d
restic-nqkhh 1/1 Running 0 14d
velero-85c4f4547-msgm5 1/1 Running 0 14d

Datamgr daemonset on 1.0.2 upgrade

We've noticed that the datamgr daemonset isn't updated when we upgrade the plugin to 1.0.2. Is this by design?

We've confirmed that a fresh install using velero-plugin-for-vsphere:1.0.2 does in fact spin up the correct datamgr daemonset (#157).

Thanks.

no mention of a need for a vsphere-config-secret

In the installation instructions, there is no mention for the need to have a secret in kube-system named vsphere-config-secret. It may be assumed that this secret exists already as the vSphere CPI should be configured but it is not always named the same and may not always have the necessary paraeters in the desired format.

For example, in TKG, the secret is named csi-vsphere-config and it's contents look similar to the following:

`[Global]
insecure-flag = true
cluster-id = default/vsan-cluster

[VirtualCenter "vcsa-01a.corp.local"]
user = [email protected]
password = VMware1!
datacenters = RegionA01

[Network]
public-network = DSwitch-Management`

What I found is expected with the Velero vsphere plugin is a secret in kube-system named vsphere-config-secret with contents looking similar to the following:

`[Global]
insecure-flag = true
cluster-id = default/vsan-cluster

[VirtualCenter "vcsa-01a.corp.local"]
user = [email protected]
password = VMware1!
datacenters = RegionA01

[Network]
public-network = DSwitch-Management

ubuntu@cli-vm:~/velero$ cat csi-vsphere.conf
[Global]
insecure-flag = "true"
cluster-id = "default/vsan-cluster"

[VirtualCenter "vcsa-01a.corp.local"]
port = "443"
user = "[email protected]"
password = "VMware1!"
datacenters = "RegionA01"

[Network]
public-network = "DSwitch-Management"`

The main things that I found were problematic were the lack of quotes and the missing port = line.

Can we get this clearly documented?

Misleading Docs since Velero Supervisor Services Not Visible

Describe the bug

I'm trying to do some testing on backup and restore of a Supervisor Cluster.

Looking at the documentation for Installing Velero on Supervisor cluster, it shows there should a service to enable. I just installed a fresh vSphere with Tanzu U2 environment and do not have any services available to install. I'm not using NSX-T nor VCF. This is a simple installation using the Distributed Switch and NSX-ALB.

If this behavior is expected, the documents should be updated to reflect the platform or criteria required to have these services enabled.

To Reproduce

vSphere with Tanzu using the Virtual Distributed Switch with NSX-ALB. Do not use NSX-T or VCF.

Expected behavior

The documentation should reflect the requirements to have Supervisor Services. If these Services are not available, where is the documentation to do a backup and restoration of a Supervisor Cluster when these services cannot be enabled?

Screenshots

verero.io/vsphere image doesn't exist

I'm seeing the following when walking through the steps to get the vsphere plugin working:

Warning Failed 25s (x3 over 61s) kubelet, vsan-cluster-md-0-68498d7f4f-2l9d2 Failed to pull image "velero.io/vsphere": rpc error: code = NotFound desc = failed to pull and unpack image "velero.io/vsphere:latest": failed to resolve reference "velero.io/vsphere:latest": velero.io/vsphere:latest: not found

Has this been moved?

Deployment of the Velero Data Manager VM with a static IP

Describe the bug
It is currently not possible to deploy the velero DM VM with a static IP, even if the IP Allocation in the ova template points to '
Static - Manual'.

[A clear and concise description of what the bug is]
While deploying the VM within the ova template, after selecting the network in which to deploy the vm, I can see that '
IP allocation: Static - Manual' is being showed, but I cannot give an IP address from my network to be assigned the vm.

[Steps to reproduce the behavior]
In a vSphere 7 environment, just followed the installation process as stated in the docs (https://github.com/vmware-tanzu/velero-plugin-for-vsphere/blob/main/docs/supervisor-datamgr.md#data-manager-virtual-machine-install) and downloaded the ova template for the velero data manager. While configuring the vm within the wizard, it is not possible to give a static ip address for the vm, and the vm gets (in my case) an ip from the dhcp of the configured network.

Expected behavior
I can assign a static IP in the configuration wizard and the vm gets that ip assigned.

Velero Netapp S3 Connectivity

Describe the bug

Today we tried to deploy Velero in a vSphere 7 with Tanzu Supervisor Cluster and with Netapp Ontap 9.7P8 without Storage Grid as S3 Storage Provider. The deployment was flawless and both the backup-driver and velero pod became running.
Sadly the Backup Location stays unavailable and when creating backups the backup fails with the following error log:

<?xml version="1.0" encoding="UTF-8"?><Error><Code>MissingSecurityHeader</Code><Message>Your request is missing a required header.</Message></Error>

Sadly our velero pod does not provide any logs and thus the above error message is the only error message we have.

To Reproduce

Install Velero with Netapp as S3 Provider while using the following deployment script:

#!/bin/sh

NAMESPACE="velero"
BUCKET="<BUCKET>"
REGION=us-east-1
S3URL="http://<NETAPP-IP>/"
PublicURL="http://<NETAPP-IP>/"
VELERO_IMAGE=<Harbor-IP>/<Harbor-Namespace>/velero:v1.5.1
VSPHERE_PLUGIN=<Harbor-IP>/<Harbor-Namespace>/velero-plugin-for-vsphere:1.1.0
AWS_PLUGIN=<Harbor-IP>/<Harbor-Namespace>/velero-plugin-for-aws:v1.1.0

./velero-vsphere install \
    --namespace $NAMESPACE \
    --image $VELERO_IMAGE \
    --use-private-registry \
    --provider aws \
    --plugins $AWS_PLUGIN,$VSPHERE_PLUGIN \
    --bucket $BUCKET \
    --secret-file ./velero-minio-credentials \
    --snapshot-location-config region=$REGION \
    --backup-location-config region=$REGION,s3ForcePathStyle="true",s3Url=$S3URL,publicUrl=$PublicURL

Create a backup for a namespace

Expected behavior

Deploy Velero with Netapp as S3 Storage Provider.
The Backup Location should be Available.
Backups should succeed and not fail.

Anything else you would like to add:

Currently we are not sure if the Netapp as S3 Storage Provider is supported.
If it's not supported are there any plans to support it in the near future?

Currently used product versions:

vCenter 7.0 U1 (Build 17491160)
Netapp Ontap 9.7P8 without Storage Grid

Backup remains stuck in phase "Deleting", no additional delete requests honoured

Velero Version:
Velero 1.5.4
Velero Plugin for AWS 1.1.0
Velero VSphere Plugin 1.1.0

Configured S3 storage using minio on a separate node.

Create a namespace with 150 PVs and 150 Pods with total 2400+ resources.
Create multiple backups (more than 30)

Observations

Deleted one of the backups, the backup remained in phase "Deleting" forever. This happened without Velero getting restarted.
Submitted additional requests for delete, but these were not picked by Velero for processing. The corresponding CR (deleteBackuprequests) showed status not updated.
Is this because Velero Delete Custom Controller remains blocked on earlier backup which remained in "Deleting" phase ?
Restarted Velero after which the subsequent delete requests were picked up for deletion. However, the first backup still remained in phase "Deleting".
Subsequent delete backup request was served and could delete the backup.
However, next backup was stuck again in phase "Deleting"
Additional delete requests were not picked up by velero.

Install velero on K8s Cluster

After installing velero on our cluster (with the vsphere cloud provider) with the following command:

velero install --provider vsphereveleroplugin/velero-plugin-for-vsphere:1.0.0 --bucket backups --plugins vsphereveleroplugin/velero-plugin-for-vsphere:1.0.0 --secret-file secrets

The log of velero says the following:

kubectl -n velero logs velero-675c97cb48-gn5fb
time="2020-05-26T04:48:49Z" level=info msg="setting log-level to INFO" logSource="pkg/cmd/server/server.go:171"
time="2020-05-26T04:48:49Z" level=info msg="Starting Velero server v1.3.2 (55a9914a3e4719fb1578529c45430a8c11c28145)" logSource="pkg/cmd/server/server.go:173"
time="2020-05-26T04:48:49Z" level=info msg="No feature flags enabled" logSource="pkg/cmd/server/server.go:177"
time="2020-05-26T04:48:49Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/crd-remap-version
time="2020-05-26T04:48:49Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
time="2020-05-26T04:48:49Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pv
time="2020-05-26T04:48:49Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
time="2020-05-26T04:48:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pv-from-pvc
time="2020-05-26T04:48:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pvc-from-pod
time="2020-05-26T04:48:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-storage-class
time="2020-05-26T04:48:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/cluster-role-bindings
time="2020-05-26T04:48:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/crd-preserve-fields
time="2020-05-26T04:48:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/job
time="2020-05-26T04:48:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
time="2020-05-26T04:48:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/restic
time="2020-05-26T04:48:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/role-bindings
time="2020-05-26T04:48:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service
time="2020-05-26T04:48:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
time="2020-05-26T04:48:49Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-vsphere kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/vsphere
time="2020-05-26T04:48:49Z" level=info msg="Checking existence of namespace" logSource="pkg/cmd/server/server.go:337" namespace=velero
time="2020-05-26T04:48:49Z" level=info msg="Namespace exists" logSource="pkg/cmd/server/server.go:343" namespace=velero
time="2020-05-26T04:48:52Z" level=info msg="Checking existence of Velero custom resource definitions" logSource="pkg/cmd/server/server.go:372"
time="2020-05-26T04:48:56Z" level=info msg="All Velero custom resource definitions exist" logSource="pkg/cmd/server/server.go:406"
time="2020-05-26T04:48:56Z" level=info msg="Checking that all backup storage locations are valid" logSource="pkg/cmd/server/server.go:413"
An error occurred: some backup storage locations are invalid: error getting backup store for location "default": unable to locate ObjectStore plugin named vsphereveleroplugin/velero-plugin-for-vsphere:1.0.0

What the 💥 ☠️ did i miss...?

FailedMount (error processing PVC : PVC is not bound)

Did a velero backup and snapshots on a sample app using the plugin and looks like there is no issue. But upon restoration of the app, it looks like the mounting of PV is unsuccessful as shown under describe pod with a endless "ContainerCreating" status. Does anyone know how to solve it by getting the restored containers to mount/attach the PVC without any errors?

backup describe

$ velero backup describe wordpress-backup --details

Phase:  Completed

Errors:    0
Warnings:  0

Storage Location:  default

Velero-Native Snapshot PVs:  true

Started:    2020-09-09 16:09:37 +0800 +08
Completed:  2020-09-09 16:09:54 +0800 +08

Expiration:  2020-10-09 16:09:37 +0800 +08

Total items to be backed up:  18
Items backed up:              18

Resource List:
  apps/v1/Deployment:
    - wordpress/wordpress
    - wordpress/wordpress-mysql
  apps/v1/ReplicaSet:
    - wordpress/wordpress-747dd64758
    - wordpress/wordpress-mysql-6ccf9b4bb6
  v1/Endpoints:
    - wordpress/wordpress
    - wordpress/wordpress-mysql
  v1/Namespace:
    - wordpress
  v1/PersistentVolume:
    - pvc-a2716ad4-9113-43a9-bc1f-f3ee1450d597
    - pvc-ff40fc3e-235b-4d1a-accc-6d0465184678
  v1/PersistentVolumeClaim:
    - wordpress/mysql-pv-claim
    - wordpress/wp-pv-claim
  v1/Pod:
    - wordpress/wordpress-747dd64758-5ktsr
    - wordpress/wordpress-mysql-6ccf9b4bb6-b5qfv
  v1/Secret:
    - wordpress/default-token-7pf6w
    - wordpress/mysql-pass-ffgtbt8k66
  v1/Service:
    - wordpress/wordpress
    - wordpress/wordpress-mysql
  v1/ServiceAccount:
    - wordpress/default

Velero-Native Snapshots:
  pvc-a2716ad4-9113-43a9-bc1f-f3ee1450d597:
    Snapshot ID:        ivd:a2350ffa-a7f6-439d-82c3-9bcb15d2e333:3e6a98e3-372d-48ff-a474-e3322d1922fe
    Type:               ivd
    Availability Zone:
    IOPS:               100
  pvc-ff40fc3e-235b-4d1a-accc-6d0465184678:
    Snapshot ID:        ivd:8e2a0069-02b9-42c7-a346-c126928aa207:a677342d-8adf-4cf0-9439-676ce9adf562
    Type:               ivd
    Availability Zone:
    IOPS:               100

restore describe

$ velero restore describe wordpress-backup-20200909160413
Name:         wordpress-backup-20200909160413
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:     PartiallyFailed (run 'velero restore logs wordpress-backup-20200909160413' for more information)
Warnings:  <error getting warnings: file not found>

Errors:  <error getting errors: file not found>

Backup:  wordpress-backup

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

restore logs

$ velero restore logs wordpress-backup-20200909160413
An error occurred: file not found

describe pod

$ kubectl describe pod wordpress-747dd64758-5ktsr -n wordpress

Events:
  Type     Reason       Age                             From                                           Message
  ----     ------       ----                            ----                                           -------
  Normal   Scheduled    <invalid>                       default-scheduler                              Successfully assigned wordpress/wordpress-747dd64758-5ktsr to 2ba3e7b7-750f-4038-8339-0340e959d4a1
  Warning  FailedMount  <invalid> (x2 over <invalid>)   kubelet, 2ba3e7b7-750f-4038-8339-0340e959d4a1  Unable to attach or mount volumes: unmounted volumes=[wordpress-persistent-storage], unattached volumes=[default-token-69ck5 wordpress-persistent-storage]: error processing PVC wordpress/wp-pv-claim: PVC is not bound
  Warning  FailedMount  <invalid> (x14 over <invalid>)  kubelet, 2ba3e7b7-750f-4038-8339-0340e959d4a1  Unable to attach or mount volumes: unmounted volumes=[wordpress-persistent-storage], unattached volumes=[wordpress-persistent-storage default-token-69ck5]: error processing PVC wordpress/wp-pv-claim: PVC is not bound

storage class

$ kubectl get sc
NAME                         PROVISIONER                    AGE
csi-storageclass (default)   csi.vsphere.vmware.com         6h9m

Plugin tries to snapshot volumes that are not VMFS and are opted into restic backups

We have a mix of VMFS and NFS PVC's. When doing a backup via:

velero backup create test-backup-4 --include-namespaces=foo

Why does the vSphere plugin try to take a snapshot of PVC's that are not VMFS volumes? These pods are also annotated with the backup.velero.io/backup-volumes: data to indicate they are backed up with restic.

I can indicate --snapshot-volumes=false and the backup will succeed. It seems like the plugin would know not to try and backup volumes that are not CSI volume or at least could defer to restic if it sees that annotation.

Maybe there's some other configuration I'm missing between the two of them.

I'm using the plugin version v.1.1.1 (we are using Kubernetes 1.20.4 with Tanzu 1.3.0 on vSphere 6.7U3).

time="2021-05-01T13:24:32Z" level=info msg="1 errors encountered backup up item" backup=velero/test-foo-backup-2 logSource="pkg/backup/backup.go:427" name=foo-mysql-primary-0
time="2021-05-01T13:24:32Z" level=error msg="Error backing up item" backup=velero/test-foo-backup-2 error="pod volume backup failed: error getting snapshot id: error running command, stderr=Fatal: unable to open config file: Stat: Access Denied.\nIs there a repository at the following location?\ns3:http://172.23.60.49:9000/velero-test/restic/foo\n: exit status 1" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/restic/backupper.go:179" error.function="github.com/vmware-tanzu/velero/pkg/restic.(*backupper).BackupPodVolumes" logSource="pkg/backup/backup.go:431" name=foo-mysql-primary-0```

PVC Restore - Incorrect Permissions

We successfully performed a backup and restore but noticed that none of our vSphere CSI PVs were accessible after they were mounted.

Upon mounting them in a test pod with root permissions, we could see the datastore marked as follows:

drwx------    4 202105   202105        4096 Jan  3 17:03 datastore

The datastore is not accessible by using cd providing a Permission denied error.

Normally, permisssions should be:

drwxrwsr-x    4 202105   202105        4096 Jan  3 17:03 datastore

Both the backup and restore are marked as completed in velero but since the datastore is not accessible, the pod fails to launch. Why are the permissions not being retained within the PVs?

Thanks!

Documentation should show how to install the AWS plugin

Describe the problem/challenge you have

The install documentation currently discusses installing different object store plugins and does not call out that the AWS plugin is the only one supported and is required for use.

Describe the solution you'd like

Update the docs to show the install process completely.

Support for SAML token authentication

Hello,

we currently use the vSphere CSI Driver with SAML Token authentication (using a dedicated vSphere solution user per K8s cluster) in our environment, which is working fine.

However, it seems that the configured solution user cert/key from the csi-vsphere.conf in the vsphere-config-secret is not supported by the Velero vSphere plugin, as the Velero plugin throws a ServerFaultCode: Cannot complete login due to an incorrect user name or password upon trying to create a snapshot, whereas with a "normal" vSphere user/password combo it works.

So is there currently a way to get vSphere SAML token authentication working in Velero or does that require changes in the plugin first?

An error occurred: No such key VirtualCenter %s in params map

hi!

environment:
TKGI 1.7

looks like i messed something up during update to v1.5.0 (from v1.4.0). velero runs successful since version 1.2, now i receive following error - also when trying to go back to old versions

$ kubectl logs -f pod/datamgr-for-vsphere-plugin-llhcw -n velero     
time="2020-10-01T15:37:09Z" level=info msg="Starting data manager server  (-)" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/server/server.go:111"
time="2020-10-01T15:37:09Z" level=info msg="data manager server is started" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/server/server.go:174"
time="2020-10-01T15:37:09Z" level=info msg="Retrieved k8s secret, vsphere-config-secret" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:63"
time="2020-10-01T15:37:09Z" level=info msg="DataMover: vSphere VC credential is retrieved" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/dataMover/data_mover.go:42"
time="2020-10-01T15:37:09Z" level=info msg="DataMover: Velero Backup Storage Location is retrieved, region=minio, bucket=velero" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/dataMover/data_mover.go:49"
time="2020-10-01T15:37:09Z" level=info msg="Got true for s3ForcePathStyle, setting s3ForcePathStyle to true" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:204"
time="2020-10-01T15:37:09Z" level=info msg="Created S3 repo type=ivd bucket=velero prefix=plugins/vsphere-astrolabe-repo/" logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/s3repository/repository_protected_entity_type_manager.go:70"
time="2020-10-01T15:37:09Z" level=info msg="DataMover: Get s3PETM from the params map" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/dataMover/data_mover.go:58"
time="2020-10-01T15:37:09Z" level=info msg="Creating NewIVDProtectedEntityTypeManagerFromConfig." logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager.go:50"
time="2020-10-01T15:37:09Z" level=error msg="Error at creating new IVD PETM from s3URLBase: VOID_URL" error="No such key VirtualCenter %s in params map" error.file="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager_util.go:418" error.function=github.com/vmware-tanzu/astrolabe/pkg/ivd.GetStringFromParamsMap logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:172"
time="2020-10-01T15:37:09Z" level=error msg="Failed to get ivdPETM from params map." VirtualCenter="<nil>" bucket=velero error="No such key VirtualCenter %s in params map" error.file="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager_util.go:418" error.function=github.com/vmware-tanzu/astrolabe/pkg/ivd.GetStringFromParamsMap logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/dataMover/data_mover.go:67" port=443 region=minio
An error occurred: No such key VirtualCenter %s in params map

my vsphere-config-secret created from following vsphere.conf

[Global]
cluster-id = "<clustername>"

[VirtualCenter "<IP from Virtual Center>"]
insecure-flag = "true"
user = "<user@domain>"
password = "XXX"
port = "443"
datacenters = "<Datacentername>"

could anyone help how to find my wrong configuration? may the problem isn't vsphere-config-secret

broken link for Create a VolumeSnapshotLocation in the Installing the plugin section

https://github.com/vmware-tanzu/velero-plugin-for-vsphere#Install-the-plugin

Installing the plugin
Install Velero - please see the Velero documentation https://velero.io/docs/v1.3.1/basic-install
Install the AWS plugin
Install the plugin
Create a VolumeSnapshotLocation
Backup using the plugin

Create a VolumeSnapshotLocation point to :https://github.com/vmware-tanzu/velero-plugin-for-vsphere/blob/master/Create-a-VolumeSnapshotLocation which return 404

it should point to https://github.com/vmware-tanzu/velero-plugin-for-vsphere/#create-a-volumesnapshotlocation

Restore of a namespace with existing PVCs is going in infinite loop

velero-plugin-for-vsphere version 1.1.0
Velero version 1.5.4

Steps performed:

Created a new Namespace with PVC. (Used the sample provided in Velero package - nginx with PV).
Performed backup of this Namespace.
Did not make any changes to the Namespace and triggered a restore operation using the above backup with all the resources included.
Restore operation gets stuck in InProgress state.

Expected behaviour: Restore operation should be successful.

Few pointers which can help with the resolution of this issue:

The original PVCs were not changed and they still exist with the same name on the cluster.
the same operation was working fine with velero-plugin-for-vsphere version 1.0.2 and Velero version 1.5.4. The restore operation completes and provides warnings for skipping resources due to conflicts.
Download snapshot operations seems to be failing with the below error:
"Failed to download snapshot, ivd:0d79bc4c-2a06-4be1-83f4-49c2e087c858:4e8fbdb2-8ace-4f25-a119-57de0cbf5553, from durable object storage. Data copy failed from PE ivd:0d79bc4c-2a06-4be1-83f4-49c2e087c858, to PE ivd:0d79bc4c-2a06-4be1-83f4-49c2e087c858:4e8fbdb2-8ace-4f25-a119-57de0cbf5553: Open virtual disk file failed. The error code is 1. with error code: 1"
After the above failure download operation seems to be getting retried few more times. (6 times) Post this it gets marked as failed.
After every 20-21 minutes a new download CR is getting generated and the download operations gets triggered again with a new CR.
Restore operation never completes. Monitored for over 2 days.
If I delete the Velero restore job and fire a new restore job (with the same backup or different), then this job gets stuck in the New phase. This may be because earlier CloneFromSnapshot is still in the pending state.

Logs files attached with debug logs enabled.
RestoreNapespaceIssueLogs.zip

velero-plugin-for-vsphere image is not published

We would like to use the image but it can not be found on any publicly available image registry.

Commercial support

Hello,

I'd like to know if there is some way how to get commercial support for this plugin. We are really interested in it and would like to go to production with this backup solution.

Thank you.

copyInt failed: ServerFaultCode: InvalidArgument when restore

I can save a full namespace.


Name:         sftp-after
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  <none>

Phase:  Completed

Namespaces:
  Included:  test
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Snapshot PVs:  true

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2020-05-24 13:11:50 +0200 CEST
Completed:  2020-05-24 13:12:13 +0200 CEST

Expiration:  2020-06-23 13:11:50 +0200 CEST

Resource List:
  apiextensions.k8s.io/v1/CustomResourceDefinition:
    - ingressroutetcps.traefik.containo.us
  apps/v1/Deployment:
    - test/sftp
  apps/v1/ReplicaSet:
    - test/sftp-645f99df8
  discovery.k8s.io/v1beta1/EndpointSlice:
    - test/svc-sftp-r7gfl
  traefik.containo.us/v1alpha1/IngressRouteTCP:
    - test/sftp-tcp
  v1/Endpoints:
    - test/svc-sftp
  v1/Event:
    - test/pvc-sftp.1611f1f59032c99c
    - test/pvc-sftp.1611f1f59074e616
    - test/pvc-sftp.1611f1f5fb85590a
    - test/sftp-645f99df8-5vc6p.1611f1f6b0c5bb81
    - test/sftp-645f99df8-5vc6p.1611f1f702369b91
    - test/sftp-645f99df8-5vc6p.1611f1f7182d98ad
    - test/sftp-645f99df8-5vc6p.1611f1f724c99ef3
    - test/sftp-645f99df8-5vc6p.1611f1f724c9f858
    - test/sftp-645f99df8-5vc6p.1611f1f741d88205
    - test/sftp-645f99df8-5vc6p.1611f1f741d8acfe
    - test/sftp-645f99df8-vchvl.1611f24d5f4c8651
    - test/sftp-645f99df8-vchvl.1611f24da89eba6b
    - test/sftp-645f99df8-vchvl.1611f24dc9e92bb8
    - test/sftp-645f99df8-vchvl.1611f24debe97f7f
    - test/sftp-645f99df8-vchvl.1611f24defa9767b
    - test/sftp-645f99df8-vchvl.1611f24df9ae72c9
    - test/sftp-645f99df8.1611f1f6b07cabf6
    - test/sftp-645f99df8.1611f24d5f228f1a
    - test/sftp.1611f1f6b01dc51c
    - test/sftp.1611f24d5e074f34
  v1/Namespace:
    - test
  v1/PersistentVolume:
    - pvc-889b22ab-5d17-440c-a7de-9a06e29107d7
  v1/PersistentVolumeClaim:
    - test/pvc-sftp
  v1/Pod:
    - test/sftp-645f99df8-vchvl
  v1/Secret:
    - test/default-token-6ghcc
    - test/gitlab-registry
  v1/Service:
    - test/svc-sftp
  v1/ServiceAccount:
    - test/default

Persistent Volumes:
  pvc-889b22ab-5d17-440c-a7de-9a06e29107d7:
    Snapshot ID:        ivd:ee786fb2-3684-4dca-880e-b4ffddb0e430:1848f578-ce52-4811-9ee1-eebbb145a4c2
    Type:               ivd
    Availability Zone:
    IOPS:               <N/A>

But I can't restore the data.
Objects other than persistent volumes are well restored.

I encounter the following error

time="2020-05-24T13:00:38Z" level=error msg="Failed to copy from remote repository." Remote PEID="ivd:ee786fb2-3684-4dca-880e-b4ffddb0e430:1848f578-ce52-4811-9ee1-eebbb145a4c2" error="copyInt failed: ServerFaultCode: InvalidArgument:  is incorrectly formatted." error.file="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager.go:215" error.function="github.com/vmware-tanzu/astrolabe/pkg/ivd.(*IVDProtectedEntityTypeManager).Copy" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/dataMover/data_mover.go:118"
time="2020-05-24T13:00:38Z" level=info msg="Filtering out the retry download request which comes in before next retry time" controller=download generation=9 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/download_controller.go:105" name=download-1848f578-ce52-4811-9ee1-eebbb145a4c2-77224d47-20f4-41fa-b4c7-a62a0f5b7b60 namespace=velero phase=Retry

time="2020-05-24T13:50:08Z" level=info msg="CreateCnsVolumeInCluster called with args, metadata: {{{} {{{} {{} ee786fb2-3684-4dca-880e-b4ffddb0e430} pvc-889b22ab-5d17-440c-a7de-9a06e29107d7 2020-05-24 11:03:00.794 +0000 UTC 0xc0009343f8 0xc00093442c 0xc00093443c 0xc00093444d <nil> []} 5120 [disk] []}} Datastore:datastore-842 []}" logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager_util.go:437"
time="2020-05-24T13:50:08Z" level=error msg="No hosts can be found for node VMs" error="<nil>" logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager_util.go:121"
time="2020-05-24T13:50:08Z" level=error msg="Failed to find any datastore in the underlying vSphere" error="No hosts can be found for node VMs" error.file="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager_util.go:122" error.function=github.com/vmware-tanzu/astrolabe/pkg/ivd.findSharedDatastoresFromAllNodeVMs logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager_util.go:264"
time="2020-05-24T13:50:08Z" level=error msg="Failed to call createCnsVolumeWithClusterConfig" error="No hosts can be found for node VMs" error.file="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager_util.go:122" error.function=github.com/vmware-tanzu/astrolabe/pkg/ivd.findSharedDatastoresFromAllNodeVMs logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager_util.go:449"
Failed to fire hook: object logged as error does not satisfy error interface
time="2020-05-24T13:50:08Z" level=error msg="Failed to copy data from data source to newly-provisioned IVD protected entity" logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager.go:338

time="2020-05-24T14:39:32Z" level=error msg="Failed at calling SnapshotManager CreateVolumeFromSnapshot with peId ivd:993185c2-80cd-4983-a50e-ca122d1bb61f:4679625c-2ad8-49eb-be1c-215b00d44f9f" cmd=/plugins/velero-plugin-for-vsphere error="Create download cr failed." error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/snapshotmgr/snapshot_manager.go:319" error.function="github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/snapshotmgr.(*SnapshotManager).CreateVolumeFromSnapshot.func1" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/plugin/volume_snapshotter_plugin.go:103" pluginName=velero-plugin-for-vsphere restore=velero/sftp-vbu02-20200524162429

documentation is lacking in regard to vsphere integration and the plugin architecture

Describe the problem/challenge you have

[A description of the current limitation/problem/challenge that you are experiencing.]
I'm deploying the plugin on an openshift environment that with vsphere csi on a 6.7 vsphere cluster. Its been very hard to understand what are the responsibilities of each component in the plugin i.e the data-manager daemon set and the backup-driver.
in addition I cannot find what user the data-manager/backup-driver uses when authenticating with the esxi/vcenter. I need to know i order to give that user permission in the vcenter.

[A clear and concise description of what you want to happen.]
It might be a good idea to add the architecture of the plugin or at least the role of each component to make troubleshooting easier.
also, there is no documentation on how to get the plugin to work with the vcenter and just a link to the VDDK docs that apply to vcenter 7 and not 6.7

Uploads fail with errorcode 3014

All upload jobs are failing with Error code 3014.

I'm not sure if this is a bug or a configuration issue but I follow the docs and examples pretty closely. I can't seem to find any reference to error code 3014 in the plugin code. I did see some veeam/vmware references to this and they referenced vsphere permissions. I made the CSI account an administrator to rule that out and still received this error.

CSI 1.0.2
Vmware 6.7U3
Kubernetes: 1.18.3
Velero: 1.4

kubectl describe on an upload object returns the following:

Status:
  Current Back Off:      16
  Message:               Failed to upload snapshot, ivd:51935689-4adf-4acd-af7b-5b0d7493e841:fb597016-baa7-48c6-a06c-9fafc4f05c3f, to durable object storage. Prepare for access failed. The error code is 3014. with error code: 3014
  Next Retry Timestamp:  2020-07-02T12:27:32Z
  Phase:                 UploadError
  Processing Node:       test-worker-3
  Progress:
  Retry Count:      5
  Start Timestamp:  2020-07-02T12:04:26Z

Here are the logs from the datamgr pod

time="2020-07-02T12:11:32Z" level=error msg="Failed to upload snapshot, ivd:51935689-4adf-4acd-af7b-5b0d7493e841:fb597016-baa7-48c6-a06c-9fafc4f05c3f, to durable object storage. Prepare for access failed. The error code is 3014. with error code: 3014" controller=upload generation=9 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/upload_controller.go:280" name=upload-fb597016-baa7-48c6-a06c-9fafc4f05c3f namespace=velero phase=UploadError

backup to snapshot-location vsl-vsphere stuck InProgress

hi,

Environment:

k8s: v1.16.7
PKS 1.7
vSphere 6.7U3
CSI 1.0.2
velero 1.4
Minio S3

i setup single nginx pod with single csi attached 50Mi Disk, backup to vsl-vsphere stuck "inProgress" since 3 hours,
...
time="2020-07-16T09:17:23Z" level=info msg="Filtering out the retry upload request which comes in before next retry time" controller=upload generation=2 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/upload_controller.go:111" name=upload-b49b513b-b2fb-4f2e-bf1c-567bf50f17a0 namespace=velero phase=InProgress time="2020-07-16T09:17:23Z" level=info msg="Filtering out the upload request from nodes other than 90761767-67ec-41b6-a4b8-a96665821d24" controller=upload generation=2 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/upload_controller.go:121" name=upload-b49b513b-b2fb-4f2e-bf1c-567bf50f17a0 namespace=velero phase=InProgress 2020/07/16 09:17:23 pei = ivd:652527b8-4219-4d30-af9d-8be25b64c70d:b49b513b-b2fb-4f2e-bf1c-567bf50f17a0 time="2020-07-16T09:17:23Z" level=info msg="Current node: 90761767-67ec-41b6-a4b8-a96665821d24. Expected node for uploading the upload CR: f9aff106-13de-4f53-a58b-0c2966793232" controller=upload generation=2 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/upload_controller.go:140" name=upload-b49b513b-b2fb-4f2e-bf1c-567bf50f17a0 namespace=velero phase=InProgress

although it looks like files are visible in minio/s3:

minio-s3-pod:/export/velero $ find .
.
./backups
./backups/nginxback
./backups/nginxback/nginxback-podvolumebackups.json.gz
./backups/nginxback/nginxback-resource-list.json.gz
./backups/nginxback/nginxback-volumesnapshots.json.gz
./backups/nginxback/nginxback-logs.gz
./backups/nginxback/nginxback-csi-volumesnapshotcontents.json.gz
./backups/nginxback/nginxback.tar.gz
./backups/nginxback/velero-backup.json
./backups/nginxback/nginxback-csi-volumesnapshots.json.gz

i do not see any additional hints in vcenter, or minio or velero logs.
i miss the plugin dir in the velero bucket somehow

any ideas for this?

thank you!
regards
Philipp

Airgap: Cannot detect CSI driver version

When deploying on an airgapped infrastructure typically container images are fetched from an internal registry.
When velero-plugin-for-vsphere starts it try to deduce CSI driver version from the container image tags but fail:

2020-08-11T09:50:33.215009344Z velero is running in the namespace, velero
2020-08-11T09:50:33.215056383Z The Image LocalMode: false
2020-08-11T09:50:33.230916361Z Failed to get tag from image gcr.io/cloud-provider-vsphere/csi/release/driver
2020-08-11T09:50:33.230938601Z Failed to get tag from image cloudnativestorage/vsphere-csi
2020-08-11T09:50:33.230941356Z Failed to get tag from image gcr.io/cloud-provider-vsphere/csi/release/syncer
2020-08-11T09:50:33.230943614Z Failed to get tag from image cloudnativestorage/syncer
2020-08-11T09:50:33.230946007Z vSphere CSI driver version is prior to v1.0.2. Velero Plug-in for vSphere requires CSI driver version to be v1.0.2 or above
2020-08-11T09:50:33.239775529Z Using datamanger image vsphereveleroplugin/data-manager-for-plugin:1.0.1.
2020-08-11T09:50:33.254360283Z CustomResourceDefinition/downloads.veleroplugin.io: attempting to create resource
2020-08-11T09:50:33.285626302Z CustomResourceDefinition/downloads.veleroplugin.io: already exists, proceeding
2020-08-11T09:50:33.285637901Z CustomResourceDefinition/downloads.veleroplugin.io: created
2020-08-11T09:50:33.285640934Z CustomResourceDefinition/uploads.veleroplugin.io: attempting to create resource
2020-08-11T09:50:33.316118932Z CustomResourceDefinition/uploads.veleroplugin.io: already exists, proceeding
2020-08-11T09:50:33.316132616Z CustomResourceDefinition/uploads.veleroplugin.io: created
2020-08-11T09:50:33.316135038Z Waiting for resources to be ready in cluster...
2020-08-11T09:50:33.324856874Z DaemonSet/datamgr-for-vsphere-plugin: attempting to create resource
2020-08-11T09:50:33.344802172Z DaemonSet/datamgr-for-vsphere-plugin: already exists, proceeding
2020-08-11T09:50:33.344812003Z DaemonSet/datamgr-for-vsphere-plugin: created
2020-08-11T09:50:33.344814748Z Waiting for data manager daemonset to be ready.

These image names are hardcoded in the plugin code

velero-plugin-for-vsphere/pkg/cmd/utils.go

Line 139 in d859573

 csi_driver_version := GetVersionFromImage(containers, "gcr.io/cloud-provider-vsphere/csi/release/driver") 

In my case the images are named cloud-provider-vsphere/csi/release/driver:v2.0.0 or myregistry/cloud-provider-vsphere/csi/release/driver:v2.0.0

Failure on uploads/downloads

I'm working on doing backup/restores with velero as deployed by Tanzu Mission Control. My upload as result of a backup is complaining with:

Failed to upload snapshot, ivd:489750af-b01e-4b05-bb39-d5090cb1048f:53b4f953-6e3c-4bca-b8cd-2e71d6610ebf, to durable object storage. Failed to delete peinfo from bucket "vmware-tmc-dp-09965998": Unable to delete object "plugins/vsphere-astrolabe-repo/ivd/peinfo/ivd:489750af-b01e-4b05-bb39-d5090cb1048f:53b4f953-6e3c-4bca-b8cd-2e71d6610ebf" from bucket "vmware-tmc-dp-09965998": MissingRegion: could not find region configuration

I'm trying to figure out where I can specify region, as I thought I had already done that. Any pointers?

unable to locate VolumeSnapshotter plugin named velero.io/vsphere

Following the instructions to the letter. Backup attempt results in the error above.
In addition, velero plugin get only reports velero.io/aws as VolumeSnapshotter.
At the same time, repeating the plugin add command results in:
Duplicate value: "velero-plugin-for-vsphere"
Is it installed or not?

the server could not find the requested resource (get volumesnapshotcontents.snapshot.storage.k8s.io)

I've Velero 1.5.2 installed on Kubespray Kubernetes 1.18 on Vsphere 6.7 u3.

This is my installation:

velero install  \
--secret-file ./credentials-velero \
--image velero/velero:v1.5.2 \
--provider aws \
--bucket velero \
--plugins velero/velero-plugin-for-aws:v1.1.0,velero/velero-plugin-for-csi:v0.1.2 \
--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://"$(kubectl get service minio -n minio | awk 'NR > 1 {print "" $3 ""}')":9000 \
--use-restic \
--use-volume-snapshots=false

and enabled plugin support:

velero plugin add vsphereveleroplugin/velero-plugin-for-vsphere:1.0.2
velero snapshot-location create vsl-vsphere --provider velero.io/vsphere

Regarding the documentation ; my system meets the requirements:

My vsphere csi drivers:

Image:         quay.io/k8scsi/csi-attacher:v1.1.1
Image ID:      docker-pullable://quay.io/k8scsi/csi-attacher@sha256:e4db94969e1d463807162a1115192ed70d632a61fbeb3bdc97b40fe9ce78c831
Image:         gcr.io/cloud-provider-vsphere/csi/release/driver:v1.0.2
Image ID:      docker-pullable://gcr.io/cloud-provider-vsphere/csi/release/driver@sha256:149e87faaacda614ee95ec271b54c8bfdbd2bf5825abc12d45c654036b798229
Image:         quay.io/k8scsi/livenessprobe:v1.1.0
Image ID:      docker-pullable://quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5
Image:         gcr.io/cloud-provider-vsphere/csi/release/syncer:v1.0.2
Image ID:      docker-pullable://gcr.io/cloud-provider-vsphere/csi/release/syncer@sha256:4b36adf433b98aa84a8f2e58db1b3bea8dfcb05ee56e65c9e9388d56c25e8647
Image:         quay.io/k8scsi/csi-provisioner:v1.2.2
Image ID:      docker-pullable://quay.io/k8scsi/csi-provisioner@sha256:e3239de37c06d2bcd0e9e9648fe9a8b418d5caf9e89f243c649ff2394d3cbfef

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

I've also set vSphere Roles and Privileges as desribed in documentation.

The problem is that when I attempt to backup sample nginx-example with pv statefull application velero backup create backup1 --include-namespaces=nginx-example --snapshot-volumes --volume-snapshot-locations vsl-vsphere , the backup gets time-out and reports following error:

error getting VolumeSnapshotContent objects for backup backup1: the server could not find the requested resource (get volumesnapshotcontents.snapshot.storage.k8s.io)

Could you please advise what I'm missing?

Disabled migration during backup

When doing the backup, gvddk calls PrepareForAccess and fails if it is not sucessfull:
https://github.com/vmware-tanzu/astrolabe/blob/master/vendor/github.com/vmware/gvddk/gvddk-high/gvddk.go#L49

PrepareForAccess ( see docs ) disables the vSphere API method RelocateVM_Task which means the migration is not possible during this time and it also requires quite high privileges on vCenter (see resulting problems #156 and #108 and here https://github.com/vmware-tanzu/velero-plugin-for-vsphere/blame/master/README.md#L21)

First of all, as we don't administer our vSphere platform, our admins are very conservative about giving out privileges especially so high so we are not able to get these.
Second point is that if some failure condition happens, we won't be able to migrate the machine if it will be in the middle of long backup process which can affect our availability. We would much more prefer failing the backup and being able to migrate the machine anytime.

To make the backup work, I have patched gvddk like this:

--- a/vendor/github.com/vmware/gvddk/gvddk-high/gvddk.go
+++ b/vendor/github.com/vmware/gvddk/gvddk-high/gvddk.go
@@ -48,7 +48,8 @@ func OpenFCD(serverName string, thumbPrint string, userName string, password str
 func Open(globalParams gDiskLib.ConnectParams, logger logrus.FieldLogger) (DiskReaderWriter, gDiskLib.VddkError) {
        err := gDiskLib.PrepareForAccess(globalParams)
        if err != nil {
-               return DiskReaderWriter{}, err
+               logger.Warningf("Prepare For Access ended with '%s'\n", err)
+               //return DiskReaderWriter{}, err
        }
        conn, err := gDiskLib.ConnectEx(globalParams)
        if err != nil {
@@ -330,4 +331,4 @@ func (this DiskConnectHandle) Close() error {

 func (this DiskConnectHandle) Capacity() int64 {
        return int64(this.info.Capacity) * gDiskLib.VIXDISKLIB_SECTOR_SIZE

What is your opinion about this change?

Failed to upload snapshot in minio s3

Hi Team,
i am currently testing this feature for block provisioner csi.vsphere.vmware.com in my airgapped tkg environment on vsphere 6.7u3.

The problem here is , when i am taking backup as bellow

 velero backup create my-backup --include-namespaces=nginx-example --snapshot-volumes --volume-snapshot-locations vsphere

Backup completed partially success.

~$ velero backup describe my-backup
Name:         my-backup
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.18.3+vmware.1
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=18

Phase:  PartiallyFailed (run `velero backup logs my-backup` for more information)

Errors:    1
Warnings:  0

Namespaces:
  Included:  nginx-example
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  true

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2020-08-11 13:07:34 +0000 UTC
Completed:  2020-08-11 13:07:43 +0000 UTC

Expiration:  2020-09-10 13:07:34 +0000 UTC

Total items to be backed up:  27
Items backed up:              27

Velero-Native Snapshots:  1 of 1 snapshots completed successfully (specify --details for more information)

Now when i am runing to check snapshot upload it gives upload failed.

~$ kubectl get -n velero uploads.veleroplugin.io -o yaml

Bello is error

  status:
    currentBackOff: 32
    message: 'Failed to upload snapshot, ivd:11959489-aecd-48e0-8634-4728b7dc59a9:be658bf6-ed86-438d-abc9-829917bde84a,
      to durable object storage. Prepare for access failed. The error code is 3014.
      with error code: 3014'
    nextRetryTimestamp: "2020-08-11T13:50:46Z"
    phase: UploadError
    processingNode: gks-rat-stg-3-md-0-7659ffb99f-74m9g
    progress: {}
    retryCount: 6
    startTimestamp: "2020-08-11T13:07:43Z"

If you can help in what causing this error.

Failed to restore snapshot with error "Failed to find the list of all datastores in VC"

Versions:
vsphere: 6.7U3
velero vsphere plugin version: 1.0.1
TKG cluster: 1.1.2

I'm trying to restore a backup with a volume snapshot.

The restore fails with the following errors:

2020-08-18T04:31:49.842102158Z time="2020-08-18T04:31:49Z" level=info msg="Download starting" controller=download generation=13 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/download_controller.go:199" name=download-f4f6462c-710f-4d18-9522-1c7ebfea60ca-84889128-7b79-4cf1-81a7-273a5b6c041c namespace=velero phase=Retry
2020-08-18T04:31:49.844689472Z time="2020-08-18T04:31:49Z" level=info msg="Download request updated by retrieving from kubernetes API server" controller=download generation=13 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/download_controller.go:212" name=download-f4f6462c-710f-4d18-9522-1c7ebfea60ca-84889128-7b79-4cf1-81a7-273a5b6c041c namespace=velero phase=Retry
2020-08-18T04:31:49.848902501Z time="2020-08-18T04:31:49Z" level=info msg="Download status updated from Retry to InProgress" controller=download generation=13 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/download_controller.go:348" name=download-f4f6462c-710f-4d18-9522-1c7ebfea60ca-84889128-7b79-4cf1-81a7-273a5b6c041c namespace=velero phase=Retry
2020-08-18T04:31:49.848914791Z 2020/08/18 04:31:49 pei = ivd:7e15fbcf-aa75-4162-a11a-3a84e987ec1d:f4f6462c-710f-4d18-9522-1c7ebfea60ca
2020-08-18T04:31:49.848949065Z time="2020-08-18T04:31:49Z" level=info msg="Copying the snapshot from remote repository to local." Remote PEID="ivd:7e15fbcf-aa75-4162-a11a-3a84e987ec1d:f4f6462c-710f-4d18-9522-1c7ebfea60ca" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/dataMover/data_mover.go:106"
2020-08-18T04:31:49.849891438Z time="2020-08-18T04:31:49Z" level=info msg="Filtering out the retry download request which comes in before next retry time" controller=download generation=14 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/download_controller.go:105" name=download-f4f6462c-710f-4d18-9522-1c7ebfea60ca-84889128-7b79-4cf1-81a7-273a5b6c041c namespace=velero phase=InProgress
2020-08-18T04:31:49.849911758Z time="2020-08-18T04:31:49Z" level=info msg="Ignore retry download request which comes in before next retry time, download CR: download-f4f6462c-710f-4d18-9522-1c7ebfea60ca-84889128-7b79-4cf1-81a7-273a5b6c041c" controller=download currentTime="2020-08-18 04:31:49.849833935 +0000 UTC m=+43777.788578062" generation=14 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/download_controller.go:111" name=download-f4f6462c-710f-4d18-9522-1c7ebfea60ca-84889128-7b79-4cf1-81a7-273a5b6c041c namespace=velero nextRetryTime="2020-08-18 04:31:50 +0000 UTC" phase=InProgress
2020-08-18T04:31:54.854041621Z 2020/08/18 04:31:54 pei = ivd:7e15fbcf-aa75-4162-a11a-3a84e987ec1d:f4f6462c-710f-4d18-9522-1c7ebfea60ca
2020-08-18T04:31:54.861476565Z time="2020-08-18T04:31:54Z" level=info msg="CreateCnsVolumeInCluster called with args, metadata: {{{} {{{} {{} 7e15fbcf-aa75-4162-a11a-3a84e987ec1d} pvc-1ae5d8dd-73a4-435c-8b41-71e14d590023 2020-08-16 14:46:51.98 +0000 UTC 0xc000bac678 0xc000bac71c 0xc000bac74d 0xc000bac77d <nil> []} 20480 [disk] [{{} 707852bb-e0b9-44b4-8b4a-bae6ea39c199}]}} Datastore:datastore-322 []}" logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager_util.go:390"
2020-08-18T04:31:55.123174705Z time="2020-08-18T04:31:55Z" level=error msg="Failed to find the list of all datastores in VC" error="datastore '/TIL-Main/datastore/*' not found" logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager_util.go:144"
2020-08-18T04:31:55.123201563Z time="2020-08-18T04:31:55Z" level=error msg="Failed to find any datastore in the underlying vSphere" error="datastore '/TIL-Main/datastore/*' not found" logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager_util.go:232"
2020-08-18T04:31:55.123207091Z time="2020-08-18T04:31:55Z" level=error msg="Failed to call createCnsVolumeWithClusterConfig" error="datastore '/TIL-Main/datastore/*' not found" logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager_util.go:402"
2020-08-18T04:31:55.26462408Z time="2020-08-18T04:31:55Z" level=error msg="Failed to copy data from data source to newly-provisioned IVD protected entity" logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager.go:340"
2020-08-18T04:31:55.264651342Z time="2020-08-18T04:31:55Z" level=error msg="Failed to copy from remote repository." Remote PEID="ivd:7e15fbcf-aa75-4162-a11a-3a84e987ec1d:f4f6462c-710f-4d18-9522-1c7ebfea60ca" error="copyInt failed: ServerFaultCode: InvalidArgument:  is incorrectly formatted." error.file="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager.go:217" error.function="github.com/vmware-tanzu/astrolabe/pkg/ivd.(*IVDProtectedEntityTypeManager).Copy" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/dataMover/data_mover.go:118"
2020-08-18T04:31:55.272057105Z time="2020-08-18T04:31:55Z" level=info msg="Download status updated from InProgress to Retry" controller=download generation=14 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/download_controller.go:348" name=download-f4f6462c-710f-4d18-9522-1c7ebfea60ca-84889128-7b79-4cf1-81a7-273a5b6c041c namespace=velero phase=InProgress
2020-08-18T04:31:55.272070397Z time="2020-08-18T04:31:55Z" level=error msg="Failed to download snapshot, ivd:7e15fbcf-aa75-4162-a11a-3a84e987ec1d:f4f6462c-710f-4d18-9522-1c7ebfea60ca, from durable object storage. copyInt failed: ServerFaultCode: InvalidArgument:  is incorrectly formatted." controller=download generation=13 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/download_controller.go:246" name=download-f4f6462c-710f-4d18-9522-1c7ebfea60ca-84889128-7b79-4cf1-81a7-273a5b6c041c namespace=velero phase=Retry
2020-08-18T04:31:55.272122875Z time="2020-08-18T04:31:55Z" level=info msg="Processed Download." controller=download key=velero/download-f4f6462c-710f-4d18-9522-1c7ebfea60ca-84889128-7b79-4cf1-81a7-273a5b6c041c logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/download_controller.go:181"
2020-08-18T04:31:55.272221933Z time="2020-08-18T04:31:55Z" level=error msg="Error in syncHandler, re-adding item to queue" controller=download error="Failed to download snapshot, ivd:7e15fbcf-aa75-4162-a11a-3a84e987ec1d:f4f6462c-710f-4d18-9522-1c7ebfea60ca, from durable object storage. copyInt failed: ServerFaultCode: InvalidArgument:  is incorrectly formatted." error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/download_controller.go:247" error.function="github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller.(*downloadController).processDownload" key=velero/download-f4f6462c-710f-4d18-9522-1c7ebfea60ca-84889128-7b79-4cf1-81a7-273a5b6c041c logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/generic_controller.go:138"
2020-08-18T04:31:55.272253707Z time="2020-08-18T04:31:55Z" level=info msg="Running reEnqueueHandler for re-adding failed download CR" controller=download key=velero/download-f4f6462c-710f-4d18-9522-1c7ebfea60ca-84889128-7b79-4cf1-81a7-273a5b6c041c logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/download_controller.go:371"
2020-08-18T04:31:55.272277117Z time="2020-08-18T04:31:55Z" level=info msg="Re-adding failed download velero/download-f4f6462c-710f-4d18-9522-1c7ebfea60ca-84889128-7b79-4cf1-81a7-273a5b6c041c to the queue" controller=download key=velero/download-f4f6462c-710f-4d18-9522-1c7ebfea60ca-84889128-7b79-4cf1-81a7-273a5b6c041c logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/download_controller.go:372"

I am able to see the IVD files in the S3 storage and the upload and backup item both completed successfully.

problems with velero-vsphere-plugin in air-gapped environment

[A clear and concise description of what the bug is]
plugin does not upload snapshots to the s3 storage.
after creating a backup the upload resource phase is

Upload error

with the messege

Failed to upload snapshot ivd:[some random sequence] to durable object storage. open virtual disk file failed. with the error code 14009

and the snapshot resource phase is uploadFailed.

[Steps to reproduce the behavior]

install Velero in an air gapped environment on an Openshift using the velero install command with the aws plugin and providing it the custom ca bundle my s3 provider uses
add the vsphere plugin velero with the velero plugin command
velero plugin add vsphereveleroplugin/velero-plugin-for-vsphere:1.1.0
copied my custom s3 ca to /etc/ssl/certs in images of the backup-driver and the data-manager so it trusts it ( as the plugin does not yet provide a solution for this problem.
created snapshot location
velero snapshot-location create vsl-vsphere --provider velero.io/vsphere
try to backup namespace
velero backup create snapshot-mongodb --include-namespaces=test-backup\ --snapshot-volumes --volume-snapshot-locations vsl-vsphere

[A clear and concise description of what you expected to happen]
i expect the snapshot to finish and upload successfully to the s3 storage

[Miscellaneous information that will assist in solving the issue]
in addition when viewing the logs in /tmp/vmware-root/vixDiskLib-1.log in one of the data-mgr pod i see a lot of errors regarding missing vmware setting files in the image.
for example

cannt open file "/etc/vmware/vsphereFeatures/techFeatures/techPreview.cfg" No such file or directory
but it looks as it uses default values instead

also in the logs of the vsphere-plugin in the velero pod i can see that the plugin cannot get the tag of the image and uses the default csi version instead (v1.0.2) when in reality our csi version is 2.0.1
[troubleshooting]
velero version server: v1.5.3 client:v1.5.3
velero client config get features:
kubernetes version: v1.19 under openshift 4.6
csi version: v2.0.1
vsphere version: 6.7u

Unable to backup PVs with plugin

Currently using Velero 1.5.2 with AWS plugin 1.1.0 and vSphere plugin 1.1.0. The backup starts off okay and can be seen in MinIO but when it reaches a PV the below error is encounter:

time="2021-01-15T21:34:15Z" level=info msg="Processing item" backup=velero/default-snap-backup2 logSource="pkg/backup/backup.go:378" name=harbor-postgresql-0 namespace=harbor-system progress= resource=pods
time="2021-01-15T21:34:15Z" level=info msg="Backing up item" backup=velero/default-snap-backup2 logSource="pkg/backup/item_backupper.go:121" name=harbor-postgresql-0 namespace=harbor-system resource=pods
time="2021-01-15T21:34:15Z" level=info msg="Executing custom action" backup=velero/default-snap-backup2 logSource="pkg/backup/item_backupper.go:327" name=harbor-postgresql-0 namespace=harbor-system resource=pods
time="2021-01-15T21:34:15Z" level=info msg="Executing podAction" backup=velero/default-snap-backup2 cmd=/velero logSource="pkg/backup/pod_action.go:51" pluginName=velero
time="2021-01-15T21:34:15Z" level=info msg="Adding pvc data-harbor-postgresql-0 to additionalItems" backup=velero/default-snap-backup2 cmd=/velero logSource="pkg/backup/pod_action.go:67" pluginName=velero
time="2021-01-15T21:34:15Z" level=info msg="Done executing podAction" backup=velero/default-snap-backup2 cmd=/velero logSource="pkg/backup/pod_action.go:77" pluginName=velero
time="2021-01-15T21:34:15Z" level=info msg="Backing up item" backup=velero/default-snap-backup2 logSource="pkg/backup/item_backupper.go:121" name=data-harbor-postgresql-0 namespace=harbor-system resource=persistentvolumeclaims
time="2021-01-15T21:34:15Z" level=info msg="Executing custom action" backup=velero/default-snap-backup2 logSource="pkg/backup/item_backupper.go:327" name=data-harbor-postgresql-0 namespace=harbor-system resource=persistentvolumeclaims
time="2021-01-15T21:34:15Z" level=info msg="Executing PVCAction" backup=velero/default-snap-backup2 cmd=/velero logSource="pkg/backup/backup_pv_action.go:49" pluginName=velero
time="2021-01-15T21:34:15Z" level=info msg="Backing up item" backup=velero/default-snap-backup2 logSource="pkg/backup/item_backupper.go:121" name=pvc-b5e5c618-0a01-4f03-bfcc-2dacb9d0860b namespace= resource=persistentvolumes
time="2021-01-15T21:34:15Z" level=info msg="Executing takePVSnapshot" backup=velero/default-snap-backup2 logSource="pkg/backup/item_backupper.go:405" name=pvc-b5e5c618-0a01-4f03-bfcc-2dacb9d0860b namespace= resource=persistentvolumes
time="2021-01-15T21:34:15Z" level=info msg="label \"topology.kubernetes.io/zone\" is not present on PersistentVolume, checking deprecated label..." backup=velero/default-snap-backup2 logSource="pkg/backup/item_backupper.go:432" name=pvc-b5e5c618-0a01-4f03-bfcc-2dacb9d0860b namespace= persistentVolume=pvc-b5e5c618-0a01-4f03-bfcc-2dacb9d0860b resource=persistentvolumes
time="2021-01-15T21:34:15Z" level=info msg="label \"failure-domain.beta.kubernetes.io/zone\" is not present on PersistentVolume" backup=velero/default-snap-backup2 logSource="pkg/backup/item_backupper.go:435" name=pvc-b5e5c618-0a01-4f03-bfcc-2dacb9d0860b namespace= persistentVolume=pvc-b5e5c618-0a01-4f03-bfcc-2dacb9d0860b resource=persistentvolumes
time="2021-01-15T21:34:15Z" level=info msg="GetVolumeID called with unstructuredPV &{map[apiVersion:v1 kind:PersistentVolume metadata:map[annotations:map[pv.kubernetes.io/provisioned-by:csi.vsphere.vmware.com] creationTimestamp:2020-09-04T20:02:22Z finalizers:[kubernetes.io/pv-protection external-attacher/csi-vsphere-vmware-com] name:pvc-b5e5c618-0a01-4f03-bfcc-2dacb9d0860b resourceVersion:45231468 uid:aa3988a6-8594-4570-9fce-ae9fda460c0f] spec:map[accessModes:[ReadWriteOnce] capacity:map[storage:8Gi] claimRef:map[apiVersion:v1 kind:PersistentVolumeClaim name:data-harbor-postgresql-0 namespace:harbor-system resourceVersion:45231314 uid:b5e5c618-0a01-4f03-bfcc-2dacb9d0860b] csi:map[driver:csi.vsphere.vmware.com fsType:ext4 volumeAttributes:map[storage.kubernetes.io/csiProvisionerIdentity:1597498711248-8081-csi.vsphere.vmware.com type:vSphere CNS Block Volume] volumeHandle:c30194fb-0edf-4307-8d0a-785dfe123609] persistentVolumeReclaimPolicy:Delete storageClassName:vsphere-dsc-nvme volumeMode:Filesystem] status:map[phase:Bound]]}" backup=velero/default-snap-backup2 cmd=/plugins/velero-plugin-for-vsphere logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/plugin/volume_snapshotter_plugin.go:161" pluginName=velero-plugin-for-vsphere
time="2021-01-15T21:34:15Z" level=warning msg="Explicitly setting empty volume-id to prevent snapshot operations." backup=velero/default-snap-backup2 cmd=/plugins/velero-plugin-for-vsphere logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/plugin/volume_snapshotter_plugin.go:162" pluginName=velero-plugin-for-vsphere
time="2021-01-15T21:34:15Z" level=info msg="No volume ID returned by volume snapshotter for persistent volume" backup=velero/default-snap-backup2 logSource="pkg/backup/item_backupper.go:458" name=pvc-b5e5c618-0a01-4f03-bfcc-2dacb9d0860b namespace= persistentVolume=pvc-b5e5c618-0a01-4f03-bfcc-2dacb9d0860b resource=persistentvolumes volumeSnapshotLocation=vsl-vsphere
time="2021-01-15T21:34:15Z" level=info msg="Persistent volume is not a supported volume type for snapshots, skipping." backup=velero/default-snap-backup2 logSource="pkg/backup/item_backupper.go:469" name=pvc-b5e5c618-0a01-4f03-bfcc-2dacb9d0860b namespace= persistentVolume=pvc-b5e5c618-0a01-4f03-bfcc-2dacb9d0860b resource=persistentvolumes
time="2021-01-15T21:34:15Z" level=info msg="Executing custom action" backup=velero/default-snap-backup2 logSource="pkg/backup/item_backupper.go:327" name=data-harbor-postgresql-0 namespace=harbor-system resource=persistentvolumeclaims
time="2021-01-15T21:34:15Z" level=info msg="1 errors encountered backup up item" backup=velero/default-snap-backup2 logSource="pkg/backup/backup.go:451" name=harbor-postgresql-0
time="2021-01-15T21:34:15Z" level=error msg="Error backing up item" backup=velero/default-snap-backup2 error="error executing custom action (groupResource=persistentvolumeclaims, namespace=harbor-system, name=data-harbor-postgresql-0): rpc error: code = Unknown desc = Failed during IsObjectBlocked check: Could not translate selfLink  to CRD name" logSource="pkg/backup/backup.go:455" name=harbor-postgresql-0

To be clear, the plugin is able to load the CSI vSphere secret to authenticate with an admin account to vCenter but nothing ever happens.

Command executed is: velero backup create default-snap-backup2 --include-namespaces=harbor-system --snapshot-volumes --volume-snapshot-locations vsl-vsphere

Not sure what else to change at this point, MinIO is receiving data and the plugin can authenticate...

Thanks!

Can't create snapshot

Using TKG cluster version v1.18.5+vmware.1-tkg.1.c40d30d on vSphere7.0U1

Velero from tanzu helm repo.

ServerFaultCode: InvalidArgument: 52d1c374-34ae-4167-8b67-2274975026e3-7b2d9f8f-3230-4d0c-a7e6-0ec998fa7382 is incorrectly formatted.
During backup of StateFull app.

time="2020-12-07T13:23:22Z" level=info msg="Ready to call astrolabe Snapshot API. Will retry on InvalidState error once per second for an hour at maximum" backup=velero/ts-backup cmd=/plugins/velero-plugin-for-vsphere logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/snapshotmgr/snapshot_manager.go:122" pluginName=velero-plugin-for-vsphere
time="2020-12-07T13:23:22Z" level=info msg="CreateSnapshot called on IVD Protected Entity, ivd:52d1c374-34ae-4167-8b67-2274975026e3-7b2d9f8f-3230-4d0c-a7e6-0ec998fa7382" backup=velero/ts-backup cmd=/plugins/velero-plugin-for-vsphere logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity.go:288" pluginName=velero-plugin-for-vsphere
time="2020-12-07T13:23:22Z" level=info msg="Retrying CreateSnapshot on IVD Protected Entity, ivd:52d1c374-34ae-4167-8b67-2274975026e3-7b2d9f8f-3230-4d0c-a7e6-0ec998fa7382, for one hour at the maximum" backup=velero/ts-backup cmd=/plugins/velero-plugin-for-vsphere logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity.go:291" pluginName=velero-plugin-for-vsphere
time="2020-12-07T13:23:22Z" level=error msg="Failed to Snapshot PE for ivd:52d1c374-34ae-4167-8b67-2274975026e3-7b2d9f8f-3230-4d0c-a7e6-0ec998fa7382" backup=velero/ts-backup cmd=/plugins/velero-plugin-for-vsphere error="Failed to create a task for the CreateSnapshot invocation on IVD Protected Entity, ivd:52d1c374-34ae-4167-8b67-2274975026e3-7b2d9f8f-3230-4d0c-a7e6-0ec998fa7382: ServerFaultCode: InvalidArgument: 52d1c374-34ae-4167-8b67-2274975026e3-7b2d9f8f-3230-4d0c-a7e6-0ec998fa7382 is incorrectly formatted." error.file="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity.go:294" error.function=github.com/vmware-tanzu/astrolabe/pkg/ivd.IVDProtectedEntity.Snapshot.func1 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/snapshotmgr/snapshot_manager.go:139" pluginName=velero-plugin-for-vsphere
time="2020-12-07T13:23:22Z" level=error msg="Fail at calling SnapshotManager CreateSnapshot from peID :, tags map[velero.io/backup:ts-backup velero.io/pv:pvc-7b2d9f8f-3230-4d0c-a7e6-0ec998fa7382 velero.io/storage-location:default]" backup=velero/ts-backup cmd=/plugins/velero-plugin-for-vsphere error="Failed to create a task for the CreateSnapshot invocation on IVD Protected Entity, ivd:52d1c374-34ae-4167-8b67-2274975026e3-7b2d9f8f-3230-4d0c-a7e6-0ec998fa7382: ServerFaultCode: InvalidArgument: 52d1c374-34ae-4167-8b67-2274975026e3-7b2d9f8f-3230-4d0c-a7e6-0ec998fa7382 is incorrectly formatted." error.file="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity.go:294" error.function=github.com/vmware-tanzu/astrolabe/pkg/ivd.IVDProtectedEntity.Snapshot.func1 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/plugin/volume_snapshotter_plugin.go:117" pluginName=velero-plugin-for-vsphere

Thank you

csi-vsphere-config, vsphere-config-secret in kube-system are missing

Hi,

Both: csi-vsphere-config, vsphere-config-secret in kube-system are missing. What is the reason? Is it a problem with installation or is it expected behaviour and at least one shoule be created manually before/after installation?

Test case:

velero plugin add someprivateregistry.XYZ/vsphereveleroplugin/velero-plugin-for-vsphere:1.0.2

velero plugin get
NAME                                 KIND
velero.io/crd-remap-version          BackupItemAction
velero.io/pod                        BackupItemAction
velero.io/pv                         BackupItemAction
velero.io/service-account            BackupItemAction
velero.io/aws                        ObjectStore
velero.io/add-pv-from-pvc            RestoreItemAction
velero.io/add-pvc-from-pod           RestoreItemAction
velero.io/change-pvc-node-selector   RestoreItemAction
velero.io/change-storage-class       RestoreItemAction
velero.io/cluster-role-bindings      RestoreItemAction
velero.io/crd-preserve-fields        RestoreItemAction
velero.io/init-restore-hook          RestoreItemAction
velero.io/job                        RestoreItemAction
velero.io/pod                        RestoreItemAction
velero.io/restic                     RestoreItemAction
velero.io/role-bindings              RestoreItemAction
velero.io/service                    RestoreItemAction
velero.io/service-account            RestoreItemAction
velero.io/aws                        VolumeSnapshotter

kubectl get secrets --all-namespaces|grep -i vsph
0

velero snapshot-location create vsl-vsphere --provider velero.io/vsphere
Snapshot volume location "vsl-vsphere" configured successfully.

velero get plugin
NAME                                 KIND
velero.io/crd-remap-version          BackupItemAction
velero.io/pod                        BackupItemAction
velero.io/pv                         BackupItemAction
velero.io/service-account            BackupItemAction
velero.io/aws                        ObjectStore
velero.io/add-pv-from-pvc            RestoreItemAction
velero.io/add-pvc-from-pod           RestoreItemAction
velero.io/change-pvc-node-selector   RestoreItemAction
velero.io/change-storage-class       RestoreItemAction
velero.io/cluster-role-bindings      RestoreItemAction
velero.io/crd-preserve-fields        RestoreItemAction
velero.io/init-restore-hook          RestoreItemAction
velero.io/job                        RestoreItemAction
velero.io/pod                        RestoreItemAction
velero.io/restic                     RestoreItemAction
velero.io/role-bindings              RestoreItemAction
velero.io/service                    RestoreItemAction
velero.io/service-account            RestoreItemAction
velero.io/aws                        VolumeSnapshotter

kubectl get secrets --all-namespaces|grep -i vsph
0

tkgi cluster test
TKGI Version:             1.8.0-build.16
Name:                     test
K8s Version:              1.17.5
Plan Name:                small
UUID:                     sdfdgdfgffghfhfghfghfghfgjhfghf
Last Action:              UPGRADE
Last Action State:        succeeded
Last Action Description:  Instance upgrade completed
Kubernetes Master Host:   test.test
Kubernetes Master Port:   8443
Worker Nodes:             4
Kubernetes Master IP(s):  10.10.10.10
Network Profile Name:
Kubernetes Profile Name:
Tags:

kubectl get pods -n velero
NAME                               READY   STATUS              RESTARTS   AGE
datamgr-for-vsphere-plugin-fphrq   0/1     CrashLoopBackOff    3          2m9s
datamgr-for-vsphere-plugin-pck4v   0/1     ContainerCreating   0          2m9s
datamgr-for-vsphere-plugin-rm9cx   0/1     CrashLoopBackOff    3          2m9s
velero-7847684d89-plfxw            1/1     Running             0          8m20s
velero-cdd9dff64-bjlrq             0/1     Init:1/2            5          7m31s

kubectl logs datamgr-for-vsphere-plugin-rm9cx -n velero
time="2020-10-30T20:19:58Z" level=info msg="Starting data manager server  (-)" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/server/server.go:111"
time="2020-10-30T20:19:58Z" level=info msg="data manager server is started" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/server/server.go:174"
time="2020-10-30T20:20:28Z" level=info msg="Skipping k8s secret vsphere-config-secret as it does not exist" error="Get https://10.111.222.1:443/api/v1/namespaces/kube-system/secrets/vsphere-config-secret: dial tcp 10.111.222.1:443: i/o timeout" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:66"
time="2020-10-30T20:20:28Z" level=info msg="Skipping k8s secret csi-vsphere-config as it does not exist" error="secrets \"csi-vsphere-config\" not found" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:66"
time="2020-10-30T20:20:28Z" level=error msg="Failed to get k8s secret, [vsphere-config-secret csi-vsphere-config]" error="secrets \"csi-vsphere-config\" not found" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:71"
time="2020-10-30T20:20:28Z" level=error msg="Could not retrieve vsphere credential from k8s secret." error="secrets \"csi-vsphere-config\" not found" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/dataMover/data_mover.go:39"
An error occurred: secrets "csi-vsphere-config" not found

vmware-tanzu / velero-plugin-for-vsphere Goto Github PK

velero-plugin-for-vsphere's Introduction

Velero Plugin for vSphere

Overview

Releases

Kubernetes distributions

Important note on vSphere with Tanzu networking requirements

Architecture

Velero Plugin for vSphere Installation and Configuration details

Known issues

Troubleshooting

Developer Guide

velero-plugin-for-vsphere's People

Contributors

Stargazers

Watchers

Forkers

velero-plugin-for-vsphere's Issues

Overview

Additional information

velero deployment images

Persistent volume

Backup

minio pod and service

BackupStorageLocation

Velero and vsphere configuration

How to reproduce

Velero-Native Snapshots: 1 of 1 snapshots completed successfully (specify --details for more information)

Recommend Projects

Recommend Topics

Recommend Org

`velero` deployment images

`minio` pod and service