Comments (7)
Think it might be due to a change in the referenced cluster's name:
cpho-phase2/k8s/server/postgres/backup/postgres-cluster-backup.yaml
Lines 19 to 20 in 0dd669a
Looks like these changes were not applied as the resource logs still point to the old name:
...
Spec:
Cluster:
Name: cpho-postgres-cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FindingCluster 4m32s (x1582 over 13h) cloudnative-pg-backup Unknown cluster cpho-postgres-cluster, will retry in 30 seconds
Flux is not watching these manifests so a manual apply is required. I've applied the necessary changes.
For future, we can separate out the Backup
(triggers an on-demand backup) resource from ScheduledBackup
and have flux sync the latter. What do you think?
from cpho-phase2.
Syncing ScheduledBackup
sounds good, should keep it more reliable.
So setting/keeping the necessary service account role is a click-ops job right now? If it's not documented anywhere, could you add it as a comment or note somewhere, for future maintainers? We'll have to make that part of the bootstrap/IaC down the line.
from cpho-phase2.
The ability to load from backups works as well.
> k get cluster -n server
NAME AGE INSTANCES READY STATUS PRIMARY
cpho-postgres14-cluster 28d 3 3 Cluster in healthy state cpho-postgres14-cluster-2
cpho-postgres14-cluster-object-storage 9m37s 3 2 Creating a new replica cpho-postgres14-cluster-object-storage-1
Again, the workloadIdentityUser
was required for the backup resource's k8s service account. In general service accounts are created with the resource's metadata.name
.
@Stephen-ONeil couid you please confirm if the data is consistent? I'm almost sure about this but it'd be good have a second set of eyes. Once done, we can take the recovered instance down.
from cpho-phase2.
The backups were failing with walArchiveFailing
error after applying the updates. I guess this was somewhat related, since the K8s service account name was changed, therefore, the workloadIdentityUser
role had to be re-applied to the cloud service account.
from cpho-phase2.
Spot checks look good, go ahead and kill the backups. No one's using or relying on the app being live right now, so I think it's worth recovering to the in-use DB for the confidence that there won't be any gotchas to that too.
from cpho-phase2.
I can confirm that all recovery mechanisms seem to be working as expected. I've pushed some docs regarding the resources to #162 and added some comments at relevant places.
re: retention policy; is there something specific you have in mind aside from -
Here's some more info on what it implies.
from cpho-phase2.
Awesome! That's sufficient for retention policy, thanks 👍
from cpho-phase2.
Related Issues (20)
- Configure k8s postgres to autoscale storage capacity
- Infrastructure as Code/Data for non-k8s infrastructure HOT 1
- Review the k8s logs
- Pre-prod (dev/test) environment HOT 3
- Configure Istio HTTP health checks
- Flux GitHub commit status updates HOT 2
- logging out redirects to admin view logout
- Streamline project specific configurations as Kustomize patch HOT 1
- Dependent pods should restart when SealedSecret resources are modified. HOT 3
- Disable username/pw login when ENABLE_LEGACY_LOGIN is false
- Use k8s ResourceQuota's HOT 1
- GKE upgrade breaks postgres cluster HOT 2
- Path based vs sub domain based routing HOT 3
- CR00001: Search Functionality
- Use IaD for Cloud Armor Security Policy
- figure out sequential-ID requirements
- Add cleanup policies for Artifact Registry HOT 3
- API: figure out how to output text in both languages
- API: expose available years
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cpho-phase2.