kubernetes/kubernetes#7890

This can be handy during recovery operation.

The main issue is what the app prefix?

--prune
--dry-run
-- tag

• Side-car watches for TPR and updates cron

cc: @sadlil

Build a cli to turn various multi step process into once command:

• repo initialization
• recover process for a kubernetes deployment.

This could be even part of https://github.com/appscode/osm

Somethings I noticed during the demo;

• Soidecar container should use PullIfAbsent
• Use Restik from our appscode repo.

We are going to have 2 sets of installation options:

• Use a stash-operator service account with reduced permissions.

Operator will use the same serviceAccountName (default or stash-operator) installed in the operator pod with pods, secrets, etc.

https://coreos.com/operators/prometheus/docs/latest/user-guides/getting-started.html

https://restic.readthedocs.io/en/latest/manual.html#temporary-files

In Kube 1.6,

• Deployments changed from extensions/v1beta1 to apps/v1beta1.

These needs to be supported.

This will require restarting the pods

https://github.com/kubernetes/community/blob/master/contributors/design-proposals/thirdpartyresources.md

kubernetes/enhancements#95 (comment)

See the proposal for details; #1

Blocked on kubernetes/community#132

• AWS
• GCS
• Azure
• Snapshots endpoints
• Swift
• Rest server
• BackBlaze B2
• Prometheus Pushgateway Scrapping

Motivation:

If you are running production workloads in Kubernetes, you want to take backup of your disks. Traditional tools like Bacula are too complex to setup and maintain manually in a dynamic compute environment like Kubernetes. So, we plan to implement a TPR controller based on restic to address these issues.

Goals:

• Take backup of Persistent Volumes or Persistent Volume claims from any running pod in Kubernetes.
• Take backup of host paths from Nodes in Kubernetes.

Non-goals:

• Take bakup of unbound PV or PVCs in Kubernetes.
• Taking backup of consistent disk snapshots. This is a non goal for initial implementation. See here #4

Why Restic

• Fewer moving parts.
• Cryptographically secure backups.
• It has limited support for cloud storage backends. But we can fix those given it is written in GO.

Design:

Taking Backups:

To take a backup 4 piece of information is required:

• Destination: Where the restic repository will live?
• Source: Which files or folders will be backed up?
• Schedule: How frequently backup command will be run?
• GC: Some policy based garbage collection of old snapshots.

This a how a TPR controller can implement backups:

• User deploys restik TPR controller. This will automatically create TPR if not present.
• User creates a TPR object defining the above 4 piece of information needed for taking backups.
• Now, user has to connect the TPR with a RC/Deployment. We can do this in 2 ways. We need to discuss this:
• Add pod selectors to TPR. TPR controller can watch for that and make modifications.
• Use adds an label RCs, Deployments, etc that TPR controller watches for. It could be:

storage.appscode.com/backup: true
backup.appscode.com/config: <name_of_tpr>

Filtering for a custom controller (one running its own, singular list/watch) will reduce deserialization costs. So, restik TOR will watch for RCs, etc. with label storage.appscode.com/backup: true.

I think the label based option has some advantages:

• TPR controller can easily identify which RC/ Deployment PodTemplate to updated, instead of doing a reverse mapping from selector to RC/Deployment in option 1.
• Dashboards like Pharm can also show if a RC etc in backed up or not easily.
• Since PV/PVCs are related to a single app, Restic TPR object will be also related a single application. So, asking users to create a TPR for each app is probably reasonable.
• Given that we have to modify the PodTemplate directly, there is not much benefit of using pod selectors unlike in Prometheus opreator.

Once TPR controller finds RC etc that has enabled bckup, it will add a sidecar container with restic. Currently there is no way to add sidecard container to a running pod. So, restic will restart the pods for the first time. This is not ideal but this the best we can do at this time.

If the RC and TPR assocation is later removed, TPR controller will also remove the side car container.

Scheuling backups:

I think we can just use cron to run backups on a schedule inside the docker container. This sounds simple enough.

Entrypoint:

Since restic process will be run on a scheule, some process will be needed to be running as the entrypoint. This will be a kloader type process that watches restic TPR and translates that into the restic compatiable config. eg,

• update cron schedule,
• modify the parameteres passed to restic cli, etc.
  We should ensure that updated config does not impact a running backup process.

Essentially, we can create a new restik (with k) cli with 3 commands:

• restik run: This is the main TPR controller entrypoint that is run as a single replica in Kubernetes.
• restik watch: This will be the entrypoint. This will watch Kubernetes restic TPR and write a cron script like below:

#!/bin/sh
RESTIC_PASSWORD=secret restic -r xyz /myapp --json | restik log

• restik log: This command will read the json output from restic cli. It can do few things with that:
  • Write a Kubernetes event to record backup completion.
  • It can send an alert using Searchlight in case of failure.
  • It can export stats to Prometheus, records how long it took to run backup. How much data was added, etc. I need to check what is available in the output json.

Supported Kubernetes Objects:

We should any Kubernetes object types that have PodTemplate in their schema. These are:

• RC
• Replica Set
• StaefulSet
• Deployment
• DaemonSet

Since new side car containers can't added once a Pod is created, Pod types will not be supported.

Backup Nodes

Users might be interested in take backup of host paths. This can be done by deploying a DaemonSet with a do nothing busybox container. Restic TPR controller can use that as a vessel for running restic side card containers.

Restarting pods

As mentioned before, first time side car containers are added, pods will be restarted by controller. Who performs the restart will be done on a case-by-case basis. For example, Kubernetes itself will restarts pods behind a deployment. In such cases, TPR controller will let Kubernetes do that.

Implementers should be aware of the controller-ref concept to properly identify the pods managed by a RC, etc.

This situation can be improved once Kubernetes adds support for extensible admission controller. Restic TPRC can become an admission controller and modify the PodTemplate accordingly. This is avoid the first restart issue. This will also allow us to support Pods directly. See tracking bug #5.

Repo initialization

For initial implementation, users will be expected to initialize the repository. We can revisit this later. See tracking bug #6 .

HTTP api

An HTTP api server can be implemented to expose readonly data for repositories. This can be used to build web Dashboards. This is outside the scope for initial implementation. Tracking bug #7.

Recovery process

The recover process will be left to users for now. Later we can think about automation.

CLI

A cli can be created that can simplify repo initialization, recover process for a Kubernetes deployment. This is outside the scope for initial implementation. Also, note that we are building a cli called osm for managing various cloud bucket operations. Tracking bug #8

Currently users can manually update stash side-car. To automate this we could add a resync: true annotation on TPR or Kube objects. Stash operator can resync those automatically.

The obvious issue is this will not work for DaemonSets & StatefulSets.

The upside is not quite clear but this could be useful.

Also, should be a tag

Side car container can't be added to a StatefulSet once it is created. So, we can add an option to
backup.appscode.com/recreate: true to delete and recreate SS with side car. This is an advanced option. we can do this later.

It is not possible at the moment to have restic create a new bucket in a different location, so you need to create it using a different program. Afterwards, the S3 server (s3.amazonaws.com) will redirect restic to the correct endpoint.

https://restic.readthedocs.io/en/stable/Manual/

This will be useful for building web dashboards. There can be done even at the upstream level.

restic/restic#832

https://github.com/k8sdb/apimachinery/blob/master/api/types.go#L33

Like KubeDB, we want to only use cloud storage destination. We can keep the option to store on a disk (for local testing & using mountable storage, like NFS volumes, etc).

• Duration of snapshot.
• Snapot success / fail
• Amount of data backed up

Given restic executions are short-lived jobs, we should use the Pushgateway:
https://prometheus.io/docs/instrumenting/pushing/

• Operator itself should expose metrics via /metrics port

UA-62096468-15