Confirm functionality of the automated database backup feature, set retention policy about cpho-phase2 HOT 7 CLOSED

Think it might be due to a change in the referenced cluster's name:

Looks like these changes were not applied as the resource logs still point to the old name:

...
Spec:
  Cluster:
    Name:  cpho-postgres-cluster
Events:
  Type     Reason          Age                     From                   Message
  ----     ------          ----                    ----                   -------
  Warning  FindingCluster  4m32s (x1582 over 13h)  cloudnative-pg-backup  Unknown cluster cpho-postgres-cluster, will retry in 30 seconds

Flux is not watching these manifests so a manual apply is required. I've applied the necessary changes.

For future, we can separate out the Backup (triggers an on-demand backup) resource from ScheduledBackup and have flux sync the latter. What do you think?

from cpho-phase2.

Syncing ScheduledBackup sounds good, should keep it more reliable.

So setting/keeping the necessary service account role is a click-ops job right now? If it's not documented anywhere, could you add it as a comment or note somewhere, for future maintainers? We'll have to make that part of the bootstrap/IaC down the line.

from cpho-phase2.

The ability to load from backups works as well.

> k get cluster -n server
NAME                                     AGE     INSTANCES   READY   STATUS                     PRIMARY
cpho-postgres14-cluster                  28d     3           3       Cluster in healthy state   cpho-postgres14-cluster-2
cpho-postgres14-cluster-object-storage   9m37s   3           2       Creating a new replica     cpho-postgres14-cluster-object-storage-1

Again, the workloadIdentityUser was required for the backup resource's k8s service account. In general service accounts are created with the resource's metadata.name.

@Stephen-ONeil couid you please confirm if the data is consistent? I'm almost sure about this but it'd be good have a second set of eyes. Once done, we can take the recovered instance down.

from cpho-phase2.

The backups were failing with walArchiveFailing error after applying the updates. I guess this was somewhat related, since the K8s service account name was changed, therefore, the workloadIdentityUser role had to be re-applied to the cloud service account.

from cpho-phase2.

Spot checks look good, go ahead and kill the backups. No one's using or relying on the app being live right now, so I think it's worth recovering to the in-use DB for the confidence that there won't be any gotchas to that too.

from cpho-phase2.

I can confirm that all recovery mechanisms seem to be working as expected. I've pushed some docs regarding the resources to #162 and added some comments at relevant places.

re: retention policy; is there something specific you have in mind aside from -

Here's some more info on what it implies.

from cpho-phase2.

Awesome! That's sufficient for retention policy, thanks 👍

from cpho-phase2.