openebs-archive / upgrade Goto Github PK

View Code? Open in Web Editor NEW

10.0 14.0 19.0 21.62 MB

contains components that help with OpenEBS data engine upgrades

License: Apache License 2.0

Makefile 2.70% Shell 7.87% Dockerfile 2.31% Go 86.17% Jinja 0.94%

hacktoberfest openebs kubernetes golang kubernetes-jobs

upgrade's Introduction

Upgrade

Contains components that help with OpenEBS data engine upgrades

Upgrading OpenEBS reources

Below are the steps for upgrading the OpenEBS reources:

Note:

If current version of ndm-operator is 1.12.0 or below and using virtual disks as blockdevices for provisioning cStor pool please refer this doc before proceeding.
After upgrading the cStor or Jiva control plane, you have to upgrade Jiva/cStor pools and volumes to the latest control plane version as early as possible. While Jiva/cStor pools and volumes will continue to work, the management operations like Ongoing Pool/Volume Provisioning, Volume Expansion, Volume Replica Migration, cStor Pool Scaleup/Scaledown, cStor VolumeReplica Scaling, cStor Pool Expansion will not be supported due to difference in control plane and pools/volumes version.

Migrating cStor pools and volumes from SPC to CSPC

Below are the steps for migrating the OpenEBS cStor custom reources:

Migrating jiva volumes to CSI spec

Note:

If the Kubernetes cluster is on rancher and iscsi is running inside the kubelet container then it is mandatory to install iscsi service on the nodes and add extra binds to the kubelet container as mentioned here.
Minimum version of Kubernetes to migrate to CSPC pools / CSI volumes is 1.17.0.
If using virtual disks as blockdevices for provisioning cStorpool please refer this doc before proceeding.

License

upgrade's People

Contributors

Stargazers

Watchers

Forkers

shubham14bajpai kmova mayadata-io palash9561 sparkingdark infiniteoverflow sonasingh46 niladrih nsathyaseelan fossabot mittachaitu rajasahil ab-hishek abhishek-kumar09 zebardy mynktl abhilashshetty04 gnu-gnu si458

upgrade's Issues

Migration failure ended into data corruption

Describe the bug: Migration failure ended into data corruption.

Expected behaviour: Even if the migration failed the pool should have been renamed and data should not be corrupted.

Steps to reproduce the bug:

Created a SPC in 1.7.0 then upgraded it to 2.4.0

mayadata:upgrade$ kubectl get spc,csp
NAME                                     AGE
storagepoolclaim.openebs.io/cstor-pool   82m

NAME                                   ALLOCATED   FREE    CAPACITY   STATUS    READONLY   TYPE      AGE
cstorpool.openebs.io/cstor-pool-3w1a   334K        39.7G   39.8G      Healthy   false      striped   82m


Started migration and made it fail just after the CSPI became online

mayadata:upgrade$ k get cspc,cspi
NAME                                           HEALTHYINSTANCES   PROVISIONEDINSTANCES   DESIREDINSTANCES   AGE
cstorpoolcluster.cstor.openebs.io/cstor-pool   1                  1                      1                  40m

NAME                                                 HOSTNAME    FREE     CAPACITY    READONLY   PROVISIONEDREPLICAS   HEALTHYREPLICAS   STATUS   AGE
cstorpoolinstance.cstor.openebs.io/cstor-pool-972g   127.0.0.1   38500M   38500378k   false      1                     0                 ONLINE   40m

Checked the zpool status on CSPI
 
mayadata:upgrade$ kubectl -n openebs exec -it cstor-pool-972g-7f4cfdd794-z598d -c cstor-pool-mgmt -- bash
root@cstor-pool-972g-7f4cfdd794-z598d:/# zpool status
  pool: cstor-7d9da0d6-904b-4310-8d90-3da1aacf4774
 state: ONLINE
  scan: none requested
config:

	NAME                                        STATE     READ WRITE CKSUM
	cstor-7d9da0d6-904b-4310-8d90-3da1aacf4774  ONLINE       0     0     0
	  /var/openebs/sparse/3-ndm-sparse.img      ONLINE       0     0     0
	  /var/openebs/sparse/0-ndm-sparse.img      ONLINE       0     0     0
	  /var/openebs/sparse/1-ndm-sparse.img      ONLINE       0     0     0
	  /var/openebs/sparse/2-ndm-sparse.img      ONLINE       0     0     0

errors: No known data errors

Then scaled up the old CSP deploy and checked the pool status (unexpected behaviour)

mayadata:openebs$ kubectl -n openebs exec -it cstor-pool-3w1a-56695f78b7-x957h -c cstor-pool-mgmt -- bash
root@cstor-pool-3w1a-56695f78b7-x957h:/# zpool status
  pool: cstor-76aad699-4e5f-4bd5-9a1b-16008d0d5c54
 state: ONLINE
  scan: none requested
config:

	NAME                                        STATE     READ WRITE CKSUM
	cstor-76aad699-4e5f-4bd5-9a1b-16008d0d5c54  ONLINE       0     0     0
	  /var/openebs/sparse/3-ndm-sparse.img      ONLINE       0     0     0
	  /var/openebs/sparse/0-ndm-sparse.img      ONLINE       0     0     0
	  /var/openebs/sparse/1-ndm-sparse.img      ONLINE       0     0     0
	  /var/openebs/sparse/2-ndm-sparse.img      ONLINE       0     0     0

errors: No known data errors

Was able to still write data using the application pod

Restarted the CSPI pod and pool got imported but the status gives an error

mayadata:migrate$ k logs -f cstor-pool-972g-7f4cfdd794-2g8l2 -c cstor-pool-mgmt
+ rm /usr/local/bin/zrepl
+ pool_manager_pid=7
+ /usr/local/bin/pool-manager start
+ trap _sigint INT
+ trap _sigterm SIGTERM
+ wait 7
E0112 10:35:27.740140       7 pool.go:122] zpool status returned error in zrepl startup : exit status 1
I0112 10:35:27.740345       7 pool.go:123] Waiting for pool container to start...
E0112 10:35:30.751974       7 pool.go:122] zpool status returned error in zrepl startup : exit status 1
I0112 10:35:30.752010       7 pool.go:123] Waiting for pool container to start...
E0112 10:35:33.755995       7 pool.go:122] zpool status returned error in zrepl startup : exit status 1
I0112 10:35:33.756057       7 pool.go:123] Waiting for pool container to start...
E0112 10:35:36.770793       7 pool.go:122] zpool status returned error in zrepl startup : exit status 1
I0112 10:35:36.770879       7 pool.go:123] Waiting for pool container to start...
E0112 10:35:39.783352       7 pool.go:122] zpool status returned error in zrepl startup : exit status 1
I0112 10:35:39.783374       7 pool.go:123] Waiting for pool container to start...
E0112 10:35:42.787035       7 pool.go:122] zpool status returned error in zrepl startup : exit status 1
I0112 10:35:42.787113       7 pool.go:123] Waiting for pool container to start...
E0112 10:35:45.797694       7 pool.go:122] zpool status returned error in zrepl startup : exit status 1
I0112 10:35:45.797771       7 pool.go:123] Waiting for pool container to start...
I0112 10:35:45.809247       7 controller.go:109] Setting up event handlers for CSPI
I0112 10:35:45.809704       7 controller.go:115] will set up informer event handlers for cvr
I0112 10:35:45.810120       7 new_restore_controller.go:105] Setting up event handlers for restore
I0112 10:35:45.886391       7 controller.go:110] Setting up event handlers for backup
I0112 10:35:45.893357       7 runner.go:38] Starting CStorPoolInstance controller
I0112 10:35:45.893409       7 runner.go:41] Waiting for informer caches to sync
I0112 10:35:45.909280       7 common.go:262] CStorPool found: [cannot open 'name': no such pool ]
I0112 10:35:45.909483       7 run_restore_controller.go:38] Starting CStorRestore controller
I0112 10:35:45.909525       7 run_restore_controller.go:41] Waiting for informer caches to sync
I0112 10:35:45.909556       7 run_restore_controller.go:53] Started CStorRestore workers
I0112 10:35:45.909674       7 runner.go:39] Starting CStorVolumeReplica controller
I0112 10:35:45.909706       7 runner.go:42] Waiting for informer caches to sync
I0112 10:35:45.909727       7 runner.go:47] Starting CStorVolumeReplica workers
I0112 10:35:45.909749       7 runner.go:54] Started CStorVolumeReplica workers
I0112 10:35:45.909893       7 runner.go:38] Starting CStorBackup controller
I0112 10:35:45.909926       7 runner.go:41] Waiting for informer caches to sync
I0112 10:35:45.993629       7 runner.go:45] Starting CStorPoolInstance workers
I0112 10:35:45.993667       7 runner.go:51] Started CStorPoolInstance workers
I0112 10:35:46.010362       7 runner.go:53] Started CStorBackup workers
I0112 10:35:46.017415       7 import.go:73] Importing pool 764d0038-cb8d-4b34-8ef8-5fb1efa80081 cstor-7d9da0d6-904b-4310-8d90-3da1aacf4774
I0112 10:35:51.166697       7 event.go:281] Event(v1.ObjectReference{Kind:"CStorPoolInstance", Namespace:"openebs", Name:"cstor-pool-972g", UID:"764d0038-cb8d-4b34-8ef8-5fb1efa80081", APIVersion:"cstor.openebs.io/v1", ResourceVersion:"9230", FieldPath:""}): type: 'Normal' reason: 'Pool Imported' Pool Import successful: cstor-7d9da0d6-904b-4310-8d90-3da1aacf4774
^C
mayadata:migrate$ k exec -it cstor-pool-972g-7f4cfdd794-2g8l2 -c cstor-pool-mgmt -- bash
root@cstor-pool-972g-7f4cfdd794-2g8l2:/# zpool status
  pool: cstor-7d9da0d6-904b-4310-8d90-3da1aacf4774
 state: ONLINE
status: One or more devices has experienced an error resulting in data
	corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
	entire pool from backup.
   see: http://zfsonlinux.org/msg/ZFS-8000-8A
  scan: none requested
config:

	NAME                                        STATE     READ WRITE CKSUM
	cstor-7d9da0d6-904b-4310-8d90-3da1aacf4774  ONLINE       0     0     7
	  /var/openebs/sparse/3-ndm-sparse.img      ONLINE       0     0    14
	  /var/openebs/sparse/0-ndm-sparse.img      ONLINE       0     0     0
	  /var/openebs/sparse/1-ndm-sparse.img      ONLINE       0     0     0
	  /var/openebs/sparse/2-ndm-sparse.img      ONLINE       0     0     0

errors: 1 data errors, use '-v' for a list 

Still able to write the data using the pod.

Restarted the CSP pool pod the import failed (expected behaviour)

mayadata:upgrade$ k logs -f cstor-pool-3w1a-56695f78b7-nb2zp -c cstor-pool-mgmt
+ rm /usr/local/bin/zrepl
+ exec /usr/local/bin/cstor-pool-mgmt start
E0112 11:09:32.888080       7 pool.go:501] zpool status returned error in zrepl startup : exit status 1
I0112 11:09:32.888334       7 pool.go:502] Waiting for zpool replication container to start...
E0112 11:09:35.896036       7 pool.go:501] zpool status returned error in zrepl startup : exit status 1
I0112 11:09:35.896298       7 pool.go:502] Waiting for zpool replication container to start...
E0112 11:09:38.903751       7 pool.go:501] zpool status returned error in zrepl startup : exit status 1
I0112 11:09:38.903805       7 pool.go:502] Waiting for zpool replication container to start...
E0112 11:09:41.912888       7 pool.go:501] zpool status returned error in zrepl startup : exit status 1
I0112 11:09:41.912968       7 pool.go:502] Waiting for zpool replication container to start...
E0112 11:09:44.920051       7 pool.go:501] zpool status returned error in zrepl startup : exit status 1
I0112 11:09:44.920155       7 pool.go:502] Waiting for zpool replication container to start...
E0112 11:09:47.928038       7 pool.go:501] zpool status returned error in zrepl startup : exit status 1
I0112 11:09:47.928138       7 pool.go:502] Waiting for zpool replication container to start...
I0112 11:09:47.983445       7 common.go:218] CStorPool CRD found
I0112 11:09:47.987162       7 common.go:236] CStorVolumeReplica CRD found
I0112 11:09:47.987794       7 new_pool_controller.go:103] Setting up event handlers
I0112 11:09:47.988014       7 new_replica_controller.go:118] will set up informer event handlers for cvr
I0112 11:09:47.988181       7 new_backup_controller.go:104] Setting up event handlers for backup
I0112 11:09:47.990730       7 new_restore_controller.go:103] Setting up event handlers for restore
I0112 11:09:47.993062       7 run_pool_controller.go:43] Starting CStorPool controller
I0112 11:09:47.993095       7 run_pool_controller.go:46] Waiting for informer caches to sync
I0112 11:09:47.996167       7 new_pool_controller.go:125] cStorPool Added event : cstor-pool-3w1a, 76aad699-4e5f-4bd5-9a1b-16008d0d5c54
I0112 11:09:47.997357       7 event.go:281] Event(v1.ObjectReference{Kind:"CStorPool", Namespace:"", Name:"cstor-pool-3w1a", UID:"76aad699-4e5f-4bd5-9a1b-16008d0d5c54", APIVersion:"openebs.io/v1alpha1", ResourceVersion:"13474", FieldPath:""}): type: 'Normal' reason: 'Synced' Received Resource create event
W0112 11:09:47.997459       7 common.go:271] CStorPool not found. Retrying after 5s, err: <nil>
I0112 11:09:47.997871       7 handler.go:598] cVR 'pvc-cb2f311d-b114-4927-bf1b-ab30738a270d-cstor-pool-3w1a': uid '6109daf9-a239-4049-b255-1aaf9671a7e0': phase 'Healthy': is_empty_status: false
I0112 11:09:47.998211       7 event.go:281] Event(v1.ObjectReference{Kind:"CStorVolumeReplica", Namespace:"openebs", Name:"pvc-cb2f311d-b114-4927-bf1b-ab30738a270d-cstor-pool-3w1a", UID:"6109daf9-a239-4049-b255-1aaf9671a7e0", APIVersion:"openebs.io/v1alpha1", ResourceVersion:"13475", FieldPath:""}): type: 'Normal' reason: 'Synced' Received Resource create event
I0112 11:09:48.093300       7 run_pool_controller.go:50] Starting CStorPool workers
I0112 11:09:48.093360       7 run_pool_controller.go:56] Started CStorPool workers
I0112 11:09:48.236208       7 new_pool_controller.go:167] cStorPool Modify event : cstor-pool-3w1a, 76aad699-4e5f-4bd5-9a1b-16008d0d5c54
I0112 11:09:48.237655       7 event.go:281] Event(v1.ObjectReference{Kind:"CStorPool", Namespace:"", Name:"cstor-pool-3w1a", UID:"76aad699-4e5f-4bd5-9a1b-16008d0d5c54", APIVersion:"openebs.io/v1alpha1", ResourceVersion:"13490", FieldPath:""}): type: 'Normal' reason: 'Synced' Received Resource modify event
E0112 11:09:48.574618       7 run_pool_controller.go:117] error syncing 'cstor-pool-3w1a': expected csp object but got 
cstorpool {null
}
W0112 11:09:53.005226       7 common.go:271] CStorPool not found. Retrying after 5s, err: <nil>
W0112 11:09:58.013215       7 common.go:271] CStorPool not found. Retrying after 5s, err: <nil>
W0112 11:10:03.021787       7 common.go:271] CStorPool not found. Retrying after 5s, err: <nil>
^C
mayadata:upgrade$ k exec -it cstor-pool-3w1a-56695f78b7-nb2zp -- bash
Defaulting container name to cstor-pool.
Use 'kubectl describe pod/cstor-pool-3w1a-56695f78b7-nb2zp -n openebs' to see all of the containers in this pod.
root@cstor-pool-3w1a-56695f78b7-nb2zp:/# zpool status
no pools available
root@cstor-pool-3w1a-56695f78b7-nb2zp:/# zpool import
2021-01-12/11:10:45.346 Iterating over all the devices to find zfs devices using blkid
2021-01-12/11:10:45.377 Iterated over cache devices to find zfs devices
no pools available to import
root@cstor-pool-3w1a-56695f78b7-nb2zp:/# 


Again restarted the CSPI pool pod and ended up in the user issue

mayadata:upgrade$ k logs -f cstor-pool-972g-7f4cfdd794-f2lsm -c cstor-pool-mgmt
+ rm /usr/local/bin/zrepl
+ pool_manager_pid=8
+ trap _sigint INT
+ /usr/local/bin/pool-manager start
+ trap _sigterm SIGTERM
+ wait 8
E0112 11:13:02.634184       8 pool.go:122] zpool status returned error in zrepl startup : exit status 1
I0112 11:13:02.634240       8 pool.go:123] Waiting for pool container to start...
E0112 11:13:05.637713       8 pool.go:122] zpool status returned error in zrepl startup : exit status 1
I0112 11:13:05.637805       8 pool.go:123] Waiting for pool container to start...
E0112 11:13:08.653611       8 pool.go:122] zpool status returned error in zrepl startup : exit status 1
I0112 11:13:08.653714       8 pool.go:123] Waiting for pool container to start...
E0112 11:13:11.668001       8 pool.go:122] zpool status returned error in zrepl startup : exit status 1
I0112 11:13:11.668128       8 pool.go:123] Waiting for pool container to start...
E0112 11:13:14.680239       8 pool.go:122] zpool status returned error in zrepl startup : exit status 1
I0112 11:13:14.680294       8 pool.go:123] Waiting for pool container to start...
E0112 11:13:17.690164       8 pool.go:122] zpool status returned error in zrepl startup : exit status 1
I0112 11:13:17.690218       8 pool.go:123] Waiting for pool container to start...
E0112 11:13:20.702640       8 pool.go:122] zpool status returned error in zrepl startup : exit status 1
I0112 11:13:20.702696       8 pool.go:123] Waiting for pool container to start...
E0112 11:13:23.717248       8 pool.go:122] zpool status returned error in zrepl startup : exit status 1
I0112 11:13:23.717277       8 pool.go:123] Waiting for pool container to start...
I0112 11:13:23.723416       8 controller.go:109] Setting up event handlers for CSPI
I0112 11:13:23.723781       8 controller.go:115] will set up informer event handlers for cvr
I0112 11:13:23.724125       8 new_restore_controller.go:105] Setting up event handlers for restore
I0112 11:13:23.733100       8 controller.go:110] Setting up event handlers for backup
I0112 11:13:23.737086       8 runner.go:38] Starting CStorPoolInstance controller
I0112 11:13:23.737111       8 runner.go:41] Waiting for informer caches to sync
I0112 11:13:23.743502       8 common.go:262] CStorPool found: [cannot open 'name': no such pool ]
I0112 11:13:23.743575       8 run_restore_controller.go:38] Starting CStorRestore controller
I0112 11:13:23.743584       8 run_restore_controller.go:41] Waiting for informer caches to sync
I0112 11:13:23.743595       8 run_restore_controller.go:53] Started CStorRestore workers
I0112 11:13:23.743643       8 runner.go:39] Starting CStorVolumeReplica controller
I0112 11:13:23.743655       8 runner.go:42] Waiting for informer caches to sync
I0112 11:13:23.743662       8 runner.go:47] Starting CStorVolumeReplica workers
I0112 11:13:23.743670       8 runner.go:54] Started CStorVolumeReplica workers
I0112 11:13:23.743719       8 runner.go:38] Starting CStorBackup controller
I0112 11:13:23.743732       8 runner.go:41] Waiting for informer caches to sync
I0112 11:13:23.743742       8 runner.go:53] Started CStorBackup workers
I0112 11:13:23.837328       8 runner.go:45] Starting CStorPoolInstance workers
I0112 11:13:23.837409       8 runner.go:51] Started CStorPoolInstance workers
I0112 11:13:23.891344       8 import.go:73] Importing pool 764d0038-cb8d-4b34-8ef8-5fb1efa80081 cstor-7d9da0d6-904b-4310-8d90-3da1aacf4774
E0112 11:13:24.039603       8 import.go:94] Failed to import pool by reading cache file: cannot import 'cstor-7d9da0d6-904b-4310-8d90-3da1aacf4774': I/O error
	Recovery is possible, but will result in some data loss.
	Returning the pool to its state as of Tue Jan 12 11:13:10 2021
	should correct the problem.  Approximately 5 seconds of data
	must be discarded, irreversibly.  Recovery can be attempted
	by executing 'zpool import -F cstor-7d9da0d6-904b-4310-8d90-3da1aacf4774'.  A scrub of the pool
	is strongly recommended after recovery.
 : exit status 1
E0112 11:13:25.375807       8 import.go:114] Failed to import pool by scanning directory: 2021-01-12/11:13:24.042 Verifying pool existence on the device /var/openebs/sparse/0-ndm-sparse.img
2021-01-12/11:13:24.042 Verifying pool existence on the device /var/openebs/sparse/3-ndm-sparse.img
2021-01-12/11:13:24.042 Verifying pool existence on the device /var/openebs/sparse/4-ndm-sparse.img
2021-01-12/11:13:24.043 Verifying pool existence on the device /var/openebs/sparse/2-ndm-sparse.img
2021-01-12/11:13:24.043 Skipping /var/openebs/sparse/4-ndm-sparse.img device due to no labels on device
2021-01-12/11:13:24.043 Verifying pool existence on the device /var/openebs/sparse/shared-cstor-pool
2021-01-12/11:13:24.043 ERROR Skipping /var/openebs/sparse/shared-cstor-pool device due to failure in read stats or it is not a regular file/block device
2021-01-12/11:13:24.042 Verifying pool existence on the device /var/openebs/sparse/1-ndm-sparse.img
2021-01-12/11:13:25.069 Verified the device /var/openebs/sparse/1-ndm-sparse.img for pool existence
2021-01-12/11:13:25.081 Verified the device /var/openebs/sparse/3-ndm-sparse.img for pool existence
2021-01-12/11:13:25.092 Verified the device /var/openebs/sparse/2-ndm-sparse.img for pool existence
2021-01-12/11:13:25.107 Verified the device /var/openebs/sparse/0-ndm-sparse.img for pool existence
cannot import 'cstor-7d9da0d6-904b-4310-8d90-3da1aacf4774': I/O error
	Recovery is possible, but will result in some data loss.
	Returning the pool to its state as of Tue Jan 12 11:13:10 2021
	should correct the problem.  Approximately 5 seconds of data
	must be discarded, irreversibly.  Recovery can be attempted
	by executing 'zpool import -F cstor-7d9da0d6-904b-4310-8d90-3da1aacf4774'.  A scrub of the pool
	is strongly recommended after recovery.
 : exit status 1

The user had hundreds of restarts on his pods and his node went down a couple of times.

The suspected reason for the lock not working is that the path is not same for csp and cspi deployments

mayadata:upgrade$ k get deploy cstor-pool-3w1a cstor-pool-972g -oyaml
apiVersion: v1
items:
- apiVersion: apps/v1
  kind: Deployment
  metadata:
    annotations:
      cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
      deployment.kubernetes.io/revision: "2"
      openebs.io/monitoring: pool_exporter_prometheus
    creationTimestamp: "2021-01-12T09:33:27Z"
    generation: 4
    labels:
      app: cstor-pool
      openebs.io/cas-template-name: cstor-pool-create-default-1.7.0
      openebs.io/cstor-pool: cstor-pool-3w1a
      openebs.io/storage-pool-claim: cstor-pool
      openebs.io/version: 2.4.0
      manager: kube-controller-manager
      operation: Update
      time: "2021-01-12T11:17:10Z"
    name: cstor-pool-3w1a
    namespace: openebs
    ownerReferences:
    - apiVersion: openebs.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: CStorPool
      name: cstor-pool-3w1a
      uid: 76aad699-4e5f-4bd5-9a1b-16008d0d5c54
    resourceVersion: "14526"
    selfLink: /apis/apps/v1/namespaces/openebs/deployments/cstor-pool-3w1a
    uid: d4a15972-8ede-4f07-abd8-fc51e9061f19
  spec:
    progressDeadlineSeconds: 600
    replicas: 1
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app: cstor-pool
    strategy:
      type: Recreate
    template:
      metadata:
        annotations:
          cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
          openebs.io/monitoring: pool_exporter_prometheus
          prometheus.io/path: /metrics
          prometheus.io/port: "9500"
          prometheus.io/scrape: "true"
        creationTimestamp: null
        labels:
          app: cstor-pool
          openebs.io/cstor-pool: cstor-pool-3w1a
          openebs.io/storage-pool-claim: cstor-pool
          openebs.io/version: 2.4.0
      spec:
        containers:
        - env:
          - name: OPENEBS_IO_CSTOR_ID
            value: 76aad699-4e5f-4bd5-9a1b-16008d0d5c54
          image: quay.io/openebs/cstor-pool:2.4.0
          imagePullPolicy: IfNotPresent
          lifecycle:
            postStart:
              exec:
                command:
                - /bin/sh
                - -c
                - sleep 2
          livenessProbe:
            exec:
              command:
              - /bin/sh
              - -c
              - timeout 120 zfs set io.openebs:livenesstimestamp="$(date +%s)" cstor-$OPENEBS_IO_CSTOR_ID
            failureThreshold: 3
            initialDelaySeconds: 300
            periodSeconds: 60
            successThreshold: 1
            timeoutSeconds: 150
          name: cstor-pool
          ports:
          - containerPort: 12000
            protocol: TCP
          - containerPort: 3233
            protocol: TCP
          - containerPort: 3232
            protocol: TCP
          resources:
            limits:
              memory: 4Gi
            requests:
              memory: 2Gi
          securityContext:
            privileged: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /dev
            name: device
          - mountPath: /var/openebs/cstor-pool
            name: storagepath
          - mountPath: /tmp
            name: tmp
          - mountPath: /var/openebs/sparse
            name: sparse
          - mountPath: /run/udev
            name: udev
          - mountPath: /var/tmp/sock
            name: sockfile
        - env:
          - name: OPENEBS_IO_CSTOR_ID
            value: 76aad699-4e5f-4bd5-9a1b-16008d0d5c54
          - name: POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          - name: RESYNC_INTERVAL
            value: "30"
          image: quay.io/openebs/cstor-pool-mgmt:2.4.0
          imagePullPolicy: IfNotPresent
          name: cstor-pool-mgmt
          resources: {}
          securityContext:
            privileged: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /dev
            name: device
          - mountPath: /tmp
            name: tmp
          - mountPath: /var/openebs/cstor-pool
            name: storagepath
          - mountPath: /var/openebs/sparse
            name: sparse
          - mountPath: /run/udev
            name: udev
          - mountPath: /var/tmp/sock
            name: sockfile
        - args:
          - -e=pool
          command:
          - maya-exporter
          image: quay.io/openebs/m-exporter:2.4.0
          imagePullPolicy: IfNotPresent
          name: maya-exporter
          ports:
          - containerPort: 9500
            protocol: TCP
          resources: {}
          securityContext:
            privileged: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /dev
            name: device
          - mountPath: /tmp
            name: tmp
          - mountPath: /var/openebs/cstor-pool
            name: storagepath
          - mountPath: /var/openebs/sparse
            name: sparse
          - mountPath: /run/udev
            name: udev
          - mountPath: /var/tmp/sock
            name: sockfile
        dnsPolicy: ClusterFirst
        nodeSelector:
          kubernetes.io/hostname: 127.0.0.1
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext: {}
        serviceAccount: openebs-maya-operator
        serviceAccountName: openebs-maya-operator
        terminationGracePeriodSeconds: 30
        volumes:
        - hostPath:
            path: /dev
            type: Directory
          name: device
        - hostPath:
            path: /var/openebs/cstor-pool/cstor-pool
            type: DirectoryOrCreate
          name: storagepath
        - emptyDir: {}
          name: sockfile
        - hostPath:
            path: /var/openebs/sparse/shared-cstor-pool
            type: DirectoryOrCreate
          name: tmp
        - hostPath:
            path: /var/openebs/sparse
            type: DirectoryOrCreate
          name: sparse
        - hostPath:
            path: /run/udev
            type: Directory
          name: udev
  status:
    conditions:
    - lastTransitionTime: "2021-01-12T09:33:28Z"
      lastUpdateTime: "2021-01-12T09:50:42Z"
      message: ReplicaSet "cstor-pool-3w1a-56695f78b7" has successfully progressed.
      reason: NewReplicaSetAvailable
      status: "True"
      type: Progressing
    - lastTransitionTime: "2021-01-12T11:17:10Z"
      lastUpdateTime: "2021-01-12T11:17:10Z"
      message: Deployment does not have minimum availability.
      reason: MinimumReplicasUnavailable
      status: "False"
      type: Available
    observedGeneration: 4
    replicas: 1
    unavailableReplicas: 1
    updatedReplicas: 1
- apiVersion: apps/v1
  kind: Deployment
  metadata:
    annotations:
      deployment.kubernetes.io/revision: "1"
      openebs.io/monitoring: pool_exporter_prometheus
    creationTimestamp: "2021-01-12T10:16:15Z"
    generation: 1
    labels:
      app: cstor-pool
      openebs.io/cstor-pool-cluster: cstor-pool
      openebs.io/cstor-pool-instance: cstor-pool-972g
      openebs.io/version: 2.4.0
    manager: kube-controller-manager
      operation: Update
      time: "2021-01-12T11:13:07Z"
    name: cstor-pool-972g
    namespace: openebs
    ownerReferences:
    - apiVersion: cstor.openebs.io/v1
      blockOwnerDeletion: true
      controller: true
      kind: CStorPoolInstance
      name: cstor-pool-972g
      uid: 764d0038-cb8d-4b34-8ef8-5fb1efa80081
    resourceVersion: "13982"
    selfLink: /apis/apps/v1/namespaces/openebs/deployments/cstor-pool-972g
    uid: d0c27829-288b-4300-9301-b5da91841ebc
  spec:
    progressDeadlineSeconds: 600
    replicas: 1
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app: cstor-pool
    strategy:
      type: Recreate
    template:
      metadata:
        annotations:
          cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
          openebs.io/monitoring: pool_exporter_prometheus
          prometheus.io/path: /metrics
          prometheus.io/port: "9500"
          prometheus.io/scrape: "true"
        creationTimestamp: null
        labels:
          app: cstor-pool
          openebs.io/cstor-pool-cluster: cstor-pool
          openebs.io/cstor-pool-instance: cstor-pool-972g
          openebs.io/version: 2.4.0
      spec:
        containers:
        - env:
          - name: OPENEBS_IO_CSPI_ID
            value: 764d0038-cb8d-4b34-8ef8-5fb1efa80081
          - name: RESYNC_INTERVAL
            value: "30"
          - name: POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          - name: OPENEBS_IO_POOL_NAME
            value: 7d9da0d6-904b-4310-8d90-3da1aacf4774
          image: openebs/cstor-pool-manager:2.4.0
          imagePullPolicy: IfNotPresent
          name: cstor-pool-mgmt
          resources: {}
          securityContext:
            privileged: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /dev
            name: device
          - mountPath: /tmp
            name: tmp
          - mountPath: /run/udev
            name: udev
          - mountPath: /var/openebs/cstor-pool
            name: storagepath
          - mountPath: /var/tmp/sock
            name: sockfile
          - mountPath: /var/openebs/sparse
            name: sparse
        - env:
          - name: OPENEBS_IO_CSTOR_ID
            value: 764d0038-cb8d-4b34-8ef8-5fb1efa80081
          - name: OPENEBS_IO_POOL_NAME
            value: 7d9da0d6-904b-4310-8d90-3da1aacf4774
          image: openebs/cstor-pool:2.4.0
          imagePullPolicy: IfNotPresent
          lifecycle:
            postStart:
              exec:
                command:
                - /bin/sh
                - -c
                - sleep 2
          livenessProbe:
            exec:
              command:
              - /bin/sh
              - -c
              - timeout 120 zfs set io.openebs:livenesstimestamp="$(date +%s)" cstor-$OPENEBS_IO_POOL_NAME
            failureThreshold: 3
            initialDelaySeconds: 300
            periodSeconds: 60
            successThreshold: 1
            timeoutSeconds: 150
          name: cstor-pool
          ports:
          - containerPort: 12000
            protocol: TCP
          - containerPort: 3232
            protocol: TCP
          - containerPort: 3233
            protocol: TCP
          resources:
            limits:
              memory: 4Gi
            requests:
              memory: 2Gi
          securityContext:
            privileged: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /dev
            name: device
          - mountPath: /tmp
            name: tmp
          - mountPath: /run/udev
            name: udev
          - mountPath: /var/openebs/cstor-pool
            name: storagepath
          - mountPath: /var/tmp/sock
            name: sockfile
          - mountPath: /var/openebs/sparse
            name: sparse
        - args:
          - -e=pool
          command:
          - maya-exporter
          image: openebs/m-exporter:2.4.0
          imagePullPolicy: IfNotPresent
          name: maya-exporter
          ports:
          - containerPort: 9500
            protocol: TCP
          resources: {}
          securityContext:
            privileged: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /dev
            name: device
          - mountPath: /tmp
            name: tmp
          - mountPath: /run/udev
            name: udev
          - mountPath: /var/openebs/cstor-pool
            name: storagepath
          - mountPath: /var/tmp/sock
            name: sockfile
          - mountPath: /var/openebs/sparse
            name: sparse
        dnsPolicy: ClusterFirst
        nodeSelector:
          kubernetes.io/hostname: 127.0.0.1
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext: {}
        serviceAccount: openebs-maya-operator
        serviceAccountName: openebs-maya-operator
        terminationGracePeriodSeconds: 30
        volumes:
        - hostPath:
            path: /dev
            type: Directory
          name: device
        - hostPath:
            path: /run/udev
            type: Directory
          name: udev
        - hostPath:
            path: /var/openebs/cstor-pool/cstor-pool
            type: DirectoryOrCreate
          name: tmp
        - hostPath:
            path: /var/openebs/sparse
            type: DirectoryOrCreate
          name: sparse
        - hostPath:
            path: /var/openebs/cstor-pool/cstor-pool
            type: DirectoryOrCreate
          name: storagepath
        - emptyDir: {}
          name: sockfile
  status:
    availableReplicas: 1
    conditions:
    - lastTransitionTime: "2021-01-12T10:16:15Z"
      lastUpdateTime: "2021-01-12T10:16:39Z"
      message: ReplicaSet "cstor-pool-972g-7f4cfdd794" has successfully progressed.
      reason: NewReplicaSetAvailable
      status: "True"
      type: Progressing
    - lastTransitionTime: "2021-01-12T11:13:07Z"
      lastUpdateTime: "2021-01-12T11:13:07Z"
      message: Deployment has minimum availability.
      reason: MinimumReplicasAvailable
      status: "True"
      type: Available
    observedGeneration: 1
    readyReplicas: 1
    replicas: 1
    updatedReplicas: 1
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

To track we can follow up with this thread: https://kubernetes.slack.com/archives/CUAKPFU78/p1608665319368100

Environment details:

OpenEBS version (use kubectl get po -n openebs --show-labels): 2.4.0
Kubernetes version (use kubectl version): 1.18
Cloud provider or hardware configuration: Rancher
OS (e.g: cat /etc/os-release): Centos
kernel (e.g: uname -a):
others: VMware Virtual disks

Move e2e tests to github actions

Upgrade of csi failed due to the changes in the csi provisioner spec

While Upgrade the csi operator to 2.5.0 failed because of the recent changes in the failed attachRequired in csi operator spec

customresourcedefinition.apiextensions.k8s.io/cstorvolumeattachments.cstor.openebs.io unchanged
customresourcedefinition.apiextensions.k8s.io/volumesnapshotclasses.snapshot.storage.k8s.io configured
customresourcedefinition.apiextensions.k8s.io/volumesnapshotcontents.snapshot.storage.k8s.io configured
customresourcedefinition.apiextensions.k8s.io/volumesnapshots.snapshot.storage.k8s.io configured
The CSIDriver "cstor.csi.openebs.io" is invalid: spec: Invalid value: storage.CSIDriverSpec{AttachRequired:(*bool)(0xc02de20f80), PodInfoOnMount:(*bool)(0xc02de20f81), VolumeLifecycleModes:[]storage.VolumeLifecycleMode{"Persistent", "Ephemeral"}}: field is immutable

Old csidriver plugin

apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1beta1","kind":"CSIDriver","metadata":{"annotations":{},"name":"cstor.csi.openebs.io"},"spec":{"attachRequired":true,"podInfoOnMount":true,"volumeLifecycleModes":["Persistent","Ephemeral"]}}
  creationTimestamp: "2021-01-10T03:53:25Z"
  name: cstor.csi.openebs.io
  resourceVersion: "3437"
  selfLink: /apis/storage.k8s.io/v1beta1/csidrivers/cstor.csi.openebs.io
  uid: 9b436edc-88cf-42d0-82b2-715b2a6f3407
spec:
  attachRequired: true
  podInfoOnMount: true
  volumeLifecycleModes:
  - Persistent
  - Ephemeral

Modified/Updated csi-driver spec

apiVersion: v1
items:
- apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1beta1","kind":"CSIDriver","metadata":{"annotations":{},"name":"cstor.csi.openebs.io"},"spec":{"attachRequired":false,"podInfoOnMount":true,"volumeLifecycleModes":["Persistent","Ephemeral"]}}
  creationTimestamp: "2021-01-11T09:46:53Z"
  name: cstor.csi.openebs.io
  resourceVersion: "3346390"
  selfLink: /apis/storage.k8s.io/v1beta1/csidrivers/cstor.csi.openebs.io
  uid: 309fc90f-e61b-455f-b17f-2763a1834d8f
spec:
  attachRequired: false
  podInfoOnMount: true
  volumeLifecycleModes:
  - Persistent
  - Ephemeral

Add integration test for jiva-csi upgrades

We need to add an e2e test like the ones we have for cstor to the github action to validate any changes we make in the future.

Multi-arch build for upgrade repo

Error on upgrade from 2.12.2 to 3.0.0 - failed to get controller deployment for volume : no deployments found for label: in openebs namespace

Describe the bug:

I'm trying to upgrade volumes from a 2.12.2 installation, following instructions from here: https://github.com/openebs/upgrade/blob/develop/docs/upgrade.md#jiva-csi-volumes

Current 2.x installation was done using the official helm chart, without any modifications to the original values.yaml.

helm upgrade --install --create-namespace -n openebs  openebs openebs/openebs --version 2.12.2

OpenEBS was upgraded with a custom values.yaml to enable Jiva Operator and legacy controllers: openebs-values.yaml.zip

helm upgrade --install --create-namespace -n openebs  openebs openebs/openebs --version 3.0.3 -f openebs-values.yaml

At this point, legacy jiva PVs are still working and new PVs can be provisioned by the Jiva Operator.

The following error comes up while trying to upgrade legacy PVs, using the upgrade tool,

E1118 02:25:42.970856       1 jiva_volume.go:71] failed to get controller deployment for volume pvc-6f494ad3-18e2-459a-b062-668c3f5fad57: no deployments found for label: openebs.io/component=jiva-controller,openebs.io/persistent-volume=pvc-6f494ad3-18e2-459a-b062-668c3f5fad57 in openebs namespace

Inspecting the pv controller, I can see the expected labels are not present in the controller pods.

kubectl get deploy -n openebs pvc-6f494ad3-18e2-459a-b062-668c3f5fad57-ctrl -oyaml

** snipped **

  labels:
    monitoring: volume_exporter_prometheus
    openebs.io/cas-template-name: jiva-volume-create-default-2.12.0
    openebs.io/cas-type: jiva
    openebs.io/controller: jiva-controller
    openebs.io/persistent-volume: pvc-6f494ad3-18e2-459a-b062-668c3f5fad57
    openebs.io/persistent-volume-claim: asdf
    openebs.io/storage-engine-type: jiva
    openebs.io/version: 2.12.0

I tried manually adding the missing openebs.io/component=jiva-controller label to the deployment, but that creates another error message.

E1118 02:49:49.631009       1 jiva_volume.go:71] failed to list replica statefulset for volumepvc-6f494ad3-18e2-459a-b062-668c3f5fad57: no statefulsets found for label: openebs.io/component=jiva-replica,openebs.io/persistent-volume=pvc-6f
494ad3-18e2-459a-b062-668c3f5fad57 in openebs namespace

Expected behaviour: I expected the upgrade tool to work without much modification from the default installation. Even the serviceaccount name in the example does not match what the helm chart created.

Steps to reproduce the bug:
Install OpenEBS using 2.12.2 chart.

helm upgrade --install --create-namespace -n openebs  openebs openebs/openebs --version 2.12.2

For simplicity, create a StorageClass with 1 replica.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: jiva1replica
  annotations:
    cas.openebs.io/config: |
      - name: FSType
        value: "xfs"
      - name: ReplicaCount
        value: "1"
      - name: StoragePool
        value: default
    openebs.io/cas-type: jiva
provisioner: openebs.io/provisioner-iscsi
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true

Create a PV using that storage class.

Upgrade to version 3.0.3 with Jiva enabled and legacy flag activated.

# Legacy components will be installed if it is enabled.
# Legacy components are - admission-server, maya api-server, snapshot-operator
# and k8s-provisioner
legacy:
  enabled: true

helm upgrade --install --create-namespace -n openebs  openebs openebs/openebs --version 3.0.3 -f openebs-values.yaml

Attempt to migrate PV to the new Jiva operator.

SPC to CSPC cstor pool migration failed due to old cstorrestore objects

Describe the bug:
I performed a SPC to CSPC pool migration following the docs here. The migration of the first CSP deployment failed because of:

I1101 09:57:10.186089       1 pool.go:396] Migrating csp cstor-disk-pool-7it6 to cspi cstor-disk-pool-4mrz
I1101 09:57:10.186731       1 pool.go:478] Scaling down csp deployment cstor-disk-pool-7it6
I1101 09:57:10.385936       1 pool.go:528] waiting for csp cstor-disk-pool-7it6 deployment to scale down
[...]
I1101 09:57:50.974154       1 pool.go:423] waiting for cspi cstor-disk-pool-4mrz to come to ONLINE state, got OFFLINE
[...]
I1101 09:59:52.102599       1 pool.go:611] Updating cvr pvc-60c6a231-8b85-43c4-8031-32bd422f81f0-cstor-disk-pool-7it6 with cspi cstor-disk-pool-4mrz info.
[...]
E1101 09:59:52.808333       1 cstor_pool.go:86] failed to migrate cspi cstor-disk-pool-4mrz: failed to create v1 cstorrestore for volumes-full-20201011-043725-0359f610-7763-428c-b0fc-a92c2f5463c6: resourceVersion should not be set on objects to be created
F1101 09:59:52.808469       1 cstor_pool.go:49] Failed to migrate cStor SPC : cstor-disk-pool

Expected behaviour: A successful migration.

Steps to reproduce the bug:
I still had cstorrestore objects from somewhere in ~2020. Unfortunately, I didn't gather a YAML of them before I deleted them. But, I think I must have been running OpenEBS 2.1.0 and Velero 1.5.1 at the time of creating them.

So, I think these steps will likely reproduce the bug:

Create SPC and a cStor volume
Create cstorrestore objects for the cStor volume as would be created by OpenEBS 2.1.0 with the Velero plugin (same version)
Perform SPC to CSPC migration.

The output of the following commands will help us better understand what's going on:

kubectl get pods -n <openebs_namespace> --show-labels: https://pastebin.com/HQYtm7DE
kubectl logs <upgrade_job_pod> -n <openebs_namespace>: https://pastebin.com/2rY78gdc

After this first attempt, I deleted the cstorbackups and restarted the job (hoping it would continue), here is the output of run 2: https://pastebin.com/gCzPyewT

Environment details:

OpenEBS version (use kubectl get po -n openebs --show-labels): 2.11.0
Kubernetes version (use kubectl version): 1.19.11+k3s1
Cloud provider or hardware configuration: bare-metal k3os
OS (e.g: cat /etc/os-release): k3os v0.19.11-k3s1r0
kernel (e.g: uname -a): 4.19.75-v8+
velero v1.6.0

Upgrade of cstor cspi pool is failed from 2.7.0 or lesser cstor operator version

Upgrade of cspc pool is not succeeded upgrade job running too long. Here is the log for cstor-pool-mgmt container log for the pool

cstor-pool-mgmt.log

upgrade job log
upgrade-job.log

Local PV hostpath upgrading from 2.7.0 to 3.4.0

1、What are the good plans for Local PV hostpath upgrading from 2.7.0 to 3.3.0
2、What is the impact of Local PV hostpath upgrading from 2.7.0 to 3.3.0 on stock data

Environment:

kubectl get po -n openebs --show-labels

NAME READY STATUS RESTARTS AGE LABELS
maya-apiserver-867994d7dd-5p5zd 1/1 Running 0 32d name=maya-apiserver,openebs.io/component-name=maya-apiserver,openebs.io/version=2.7.0,pod-template-hash=867994d7dd
openebs-admission-server-78458d9ff6-h6fbs 1/1 Running 0 18d app=admission-webhook,openebs.io/component-name=admission-webhook,openebs.io/version=2.7.0,pod-template-hash=78458d9ff6
openebs-localpv-provisioner-548c7dfbf7-mv8hn 1/1 Running 0 32d name=openebs-localpv-provisioner,openebs.io/component-name=openebs-localpv-provisioner,openebs.io/version=2.7.0,pod-template-hash=548c7dfbf7
openebs-ndm-4crhj 1/1 Running 4 231d controller-revision-hash=6cbb7dbf5f,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=2.7.0,pod-template-generation=1
openebs-ndm-4grcj 1/1 Running 2 231d controller-revision-hash=6cbb7dbf5f,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=2.7.0,pod-template-generation=1
openebs-ndm-57qlq 1/1 Running 2 231d controller-revision-hash=6cbb7dbf5f,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=2.7.0,pod-template-generation=1
openebs-ndm-cfqfb 1/1 Running 2 231d controller-revision-hash=6cbb7dbf5f,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=2.7.0,pod-template-generation=1
openebs-ndm-cluster-exporter-5c985f8b77-9zrzw 1/1 Running 0 32d name=openebs-ndm-cluster-exporter,openebs.io/component-name=ndm-cluster-exporter,openebs.io/version=3.0.0,pod-template-hash=5c985f8b77
openebs-ndm-j4wxx 1/1 Running 4 524d controller-revision-hash=6cbb7dbf5f,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=2.7.0,pod-template-generation=1
openebs-ndm-mt2z4 1/1 Running 2 231d controller-revision-hash=6cbb7dbf5f,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=2.7.0,pod-template-generation=1
openebs-ndm-node-exporter-4lv6r 1/1 Running 1 231d controller-revision-hash=6f95f5c64d,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-node-exporter-5v2cf 1/1 Running 1 231d controller-revision-hash=6f95f5c64d,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-node-exporter-c7662 1/1 Running 2 524d controller-revision-hash=6f95f5c64d,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-node-exporter-dnzj4 1/1 Running 1 231d controller-revision-hash=6f95f5c64d,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-node-exporter-fcf4m 1/1 Running 2 231d controller-revision-hash=6f95f5c64d,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-node-exporter-pl24h 1/1 Running 1 231d controller-revision-hash=6f95f5c64d,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-node-exporter-zrtvm 1/1 Running 1 231d controller-revision-hash=6f95f5c64d,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-operator-67876b4dc4-n9dm7 1/1 Running 0 32d name=openebs-ndm-operator,openebs.io/component-name=ndm-operator,openebs.io/version=2.7.0,pod-template-hash=67876b4dc4
openebs-ndm-wv496 1/1 Running 2 231d controller-revision-hash=6cbb7dbf5f,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=2.7.0,pod-template-generation=1
openebs-provisioner-54c6dbf8bb-sz4tl 1/1 Running 3 32d name=openebs-provisioner,openebs.io/component-name=openebs-provisioner,openebs.io/version=2.7.0,pod-template-hash=54c6dbf8bb
openebs-snapshot-operator-d9855ff67-5mg25 2/2 Running 0 32d name=openebs-snapshot-operator,openebs.io/component-name=openebs-snapshot-operator,openebs.io/version=2.7.0,pod-template-hash=d9855ff67

kubectl version

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T14:30:33Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

Automate virtual disk correction while cStor pool migration

Migration of jiva external provisioner fails due to old volume directory is not found in the node

Had Trying the non-csi jiva migration in a five node cluster followed by the document https://github.com/openebs/upgrade/blob/master/docs/migration.md#migrating-jiva-external-provisioned-volumes-to-jiva-csi-volumes-experimental

Jiva volume policy mentioned in the migration document is not working

apiVersion: openebs.io/v1alpha1
kind: JivaVolumePolicy
metadata:
  name: example-jivavolumepolicy
  namespace: openebs
spec:
  replicaSC: openebs-hostpath
  target:
    # This should be same as the ReplicaCount
    # in the old StorageClass 
    replicationFactor: 1

Instead of that used the below jiva csi volume policy.

apiVersion: openebs.io/v1alpha1
kind: JivaVolumePolicy
metadata:
  name: migrate-jivavolumepolicy
  namespace: openebs
spec:
  replicaSC: openebs-hostpath
  enableBufio: false
  autoScaling: false
  target:
    # This should be same as the ReplicaCount
    # in the old StorageClass 
    replicationFactor: 3

Then followed the steps in the document mentioned that after scaled down the csi sts replica

SSH into each node. Remove the files from the new localpv hostpath and copy the files from old hostpath to the new localpv hostpath While trying not able to saw the non-csi volume path in the some nodes because of the following reason- Non csi replicas (old volume rep) presented in N1,N2,N3 and the new volume present in N2,N4,N5 so in N4 and N5 there is no old rep to move the files.

Add status reporting for migration via CR

Migration currently runs as a job and provides only logs for status updates. For any higher operator to make use of the migration job a CustomResource needs to be added for status reporting.

PRs
openebs-archive/api#64 got merged 2.1.0
Implementation will be taken in next release.

Add jiva non-csi to csi migration doc

This should also include an e2e test which we can run in github action.

Add support for jiva csi volume upgrades

Upgrade support for jiva csi volumes on a release basis.

Add repo documentation

Add upgrade related test cases

List of tests that should be covered:

unit tests for some logic related functions for upgrade.
unit tests for some logic related functions for migrate.
travis sanity test migrating from 1.11.0 to ci.
travis sanity test upgrading from 1.10.0 to ci.

Add support to deprecate images with -amd64 suffix

Describe the problem/challenge you have
[A description of the current limitation/problem/challenge that you are experiencing.]

Describe the solution you'd like
[A clear and concise description of what you want to happen.]

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

OpenEBS version (use kubectl get po -n openebs --show-labels):
Kubernetes version (use kubectl version):
Cloud provider or hardware configuration:
OS (e.g: cat /etc/os-release):
kernel (e.g: uname -a):
others:

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.