Giter Club home page Giter Club logo

tergum's Introduction

Tergum: Universal Backup Tool

SikaLabs SikaLabs SikaLabs

Why Tergum?

Tergum is simple tool provides centralized backup solution with multiple sources (databases, files, S3, ...) and multiple backup storages (S3, filesystem, ...). Tergum has native backup monitoring and alerts you when backup fails. Tergum also support backup encryption, compression and automatic recovery testing.

Tergum is under active development, not all features are already implemented. Check current project state

Do you want to start using Tergum? Give us a call

Let's discuss Tergum in your project in 30 min call

What "Tergum" means?

Tergum means backup in latin.

Tergum Cloud: Bring Your Backups into Cloud

Tergum Cloud allow you to manage your backup using UI & Terraform and store your backups securely in our AWS.

Are you interested in our public beta? Drop us email [email protected]

Tergum Enterprise: Use Tergum Cloud in Your Private Infrastructure

Tergum Enterprise brings our cloud platform behind your filewall. For an inquiry, contact our sales [email protected]

Install

Install using Brew:

brew install sikalabs/tap/tergum

Autocomplete

See: tergum completion

Bash

source <(tergum completion bash)

CLI Usage

Generated CLI Docs on Github

See: https://github.com/sikalabs/tergum-cli-docs/blob/master/tergum.md#tergum

Generate CLI Docs

Generate Markdown CLI docs to ./cobra-docs

tergum generate-docs

Tergum Config File

Tergum supports only JSON config file, but we're working on YAML support.

Config file examples are in misc/example/config directory

Basic Config Structure

Meta:
  SchemaVersion: 3
Cloud: <Cloud>
Notification: <Notification>
Backups:
  - <Backup>
  - <Backup>
  - ...

Backup Block

ID: <UniqueBackupID>
Source:
  Mysql: <BackupSourceMysqlConfiguration>
  MysqlServer: <BackupSourceMysqlServerConfiguration>
  Postgres: <BackupSourcePostgresConfiguration>
  PostgresServer: <BackupSourcePostgresServerConfiguration>
  Mongo: <BackupSourceMongoConfiguration>
  SingleFile: <BackupSourceSingleFileConfiguration>
  Dir: <BackupSourceDirConfiguration>
  KubernetesTLSSecret: <BackupSourceKubernetesTLSSecret>
  Kubernetes: <BackupSourceKubernetes>
  Notion: <BackupSourceNotion>
  FTP: <BackupSourceFTP>
  Redis: <BackupSourceRedis>
  Vault: <BackupSourceVault>
  Dummy: <BackupSourceDummy>
  Gitlab: <BackupSourceGitlab>
  Consul: <BackupSourceConsul>
Middlewares:
  - <MiddlewareConfiguration>
  - ...
Destinations:
  - ID: <UniqueBackupDestinationID>
    Middlewares:
      - <MiddlewareConfiguration>
      - ...
    FilePath: <BackupDestinationFilePathConfiguration>
    File: <BackupDestinationFileConfiguration>
    S3: <BackupDestinationS3Configuration>
    AzureBlob: <BackupDestinationAzureBlobConfiguration>
  - ...
SleepBefore: <sleep time befor backup job in seconds>

GzipMiddlewareConfiguration

Gzip: {}
SymmetricEncryption:
  Passphrase: "passphrase"

Example BackupSourceMysqlConfiguration Block

Host: "127.0.0.1"
Port: "3306"
User: "root"
Password: "root"
Database: "default"

With extra args

Host: "127.0.0.1"
Port: "3306"
User: "root"
Password: "root"
Database: "default"
MysqldumpExtraArgs:
  - --column-statistics=0

Example BackupSourceMysqlServerConfiguration Block

Host: "127.0.0.1"
Port: "3306"
User: "root"
Password: "root"

With extra args

Host: "127.0.0.1"
Port: "3306"
User: "root"
Password: "root"
MysqldumpExtraArgs:
  - --column-statistics=0

Example BackupSourcePostgresConfiguration Block

Host: "127.0.0.1"
Port: "15432"
User: "postgres"
Password: "pg"
Database: "postgres"

With extra args

Host: "127.0.0.1"
Port: "15432"
User: "postgres"
Password: "pg"
Database: "postgres"
PgdumpExtraArgs:
  - --ignore-version

Example BackupSourcePostgresServerConfiguration Block

Host: "127.0.0.1"
Port: "15432"
User: "postgres"
Password: "pg"

With extra args

Host: "127.0.0.1"
Port: "15432"
User: "postgres"
Password: "pg"
PgdumpallExtraArgs:
  - --ignore-version

Example BackupSourceMongoConfiguration Block

Dump all dbs & no auth

Host: "127.0.0.1"
Port: "27017"

Dump all dbs with auth

Host: "127.0.0.1"
Port: "27017"
User: "root"
Password: "root"

Dump single db with auth

Host: "127.0.0.1"
Port: "27017"
User: "root"
Password: "root"
Database: "test"

Dump single db with auth and custom Authentication Database

Host: "127.0.0.1"
Port: "27017"
User: "root"
Password: "root"
AuthenticationDatabase: "test" # default is admin
Database: "test"

Example BackupSourceKubernetesTLSSecret Block

Backup all TLS secrets

Server: https://kubernetes-api.example.com
Token: foo-bar-baz
Namespace: default

Backup single TLS secret

Server: https://kubernetes-api.example.com
Token: foo-bar-baz
Namespace: default
SecretName: tls-example-com

Example BackupSourceKubernetes Block

Backup all resources (pods)

Server: https://kubernetes-api.example.com
Token: foo-bar-baz
Namespace: default
Resource: pod

Backup single resource (hello-world pod)

Server: https://kubernetes-api.example.com
Token: foo-bar-baz
Namespace: default
Resource: pod
Name: hello-world

Example BackupSourceSingleFileConfiguration Block

Path: /data/export/dump.sql

Example BackupSourceDirConfiguration Block

Path: /data

Example BackupSourceNotion Block

Token: <Notion token_v2>
SpaceID: <Notion Space UID>
Format: <Fotmat of export ("html" or "markdown")>

Example BackupSourceFTP Block

Host: <FTP host>
User: <FTP user>
Password: <FTP password>

Example BackupSourceRedis Block

Host: <host>
Port: <port>

Example BackupSourceVault Block

Addr: <vault address>
Token: <vault token>

Example BackupSourceDummy Block

Content: <backup content>

Example BackupSourceGitlab Block

NamePrefix: <prefix Gitlab backup file in /var/opt/gitlab/backups>
Skip: <skip (for example registry)>

Example BackupSourceConsul Block

Addr: <host>
Token: <token>

Example without ACL

Addr: http://127.0.0.1:8500

Example with ACL requires token

Addr: http://127.0.0.1:8500
Token: 51047cd1-c243-a969-2bf1-a845405e4da9

Example BackupDestinationFilePathConfiguration Block

Path: "/backup/mysql-default.sql"

Example BackupDestinationFileConfiguration Block

Dir: "/backup/"
Prefix: "mysql-default"
Suffix: "sql"

Example BackupDestinationS3Configuration Block

AWS:

AccessKey: "admin"
SecretKey: "asdfasdf"
Endpoint: "https://minio.example.com"
BucketName: "tergum-backups"
Prefix: "mysql-default"
Suffix: "sql"

Minio:

accessKey: "aws_access_key_id"
secretKey: "aws_secret_access_key"
region: "eu-central-1"
bucketName: "tergum-backups"
prefix: "mysql-default"
suffix: "sql"

Example BackupDestinationAzureBlobConfiguration Block

AccountName: account_name
AccountKey: account_key
ContainerName: container_name
Prefix: "mysql-default"
Suffix: "sql"

Notification Block

Backends: {
  Email:  <NotificationBackendEmail>
Target:
  - <NotificationTarget>
  - <NotificationTarget>
  - ...

Example NotificationBackendEmail Block

SmtpHost: "mail.example.com"
SmtpPort: "25"
Usename: "aaa"
Password: "aaa/bbb"
From: "[email protected]"

NotificationTarget Block

Email: <NotificationEmailTarget>
SlackWebhook: <NotificationSlackWebhookTarget>

Example NotificationEmailTarget Block

Emails:
  - [email protected]
  - [email protected]
SendOK: false
  • SendOK=true will send email notification for all tergum runs (failed & OK runs)

Example NotificationSlackWebhookTarget Block

URLs:
  - https://hooks.slack.com/services/xxx/yyy/zzz
SendOK: false
  • SendOK=true will send email notification for all tergum runs (failed & OK runs)

Cloud Block

Email: <email of tergum cloud account>

Current Project State

Backup Sources

  • SingleFile
  • Files (Dir)
  • Postgres
  • PostgresServer
  • MySQL
  • MySQLServer
  • Oracle (Enterprise)
  • S3
  • Ceph RBD
  • CephFS
  • MongoDB
  • Gitlab
  • Proxmox
  • Kubernetes Resource
    • Kubernetes TLS Secret
  • Container Image
  • Redis
  • Notion
  • FTP Server (for old school hostings)
  • Hashicorp Vault
  • Hashicorp Consul
  • Dummy (for testing)

Passwords Sources

  • YAML
  • Environment Variables
  • Hashicorp Vault
  • AWS Secrets Manager
  • Azure Key Vault

Backup Processors

  • GZIP Compression
  • Symmetric Encryption
  • AsymmetricEncryption
  • GPG Encryption
  • GPG Signatures

Backup Storage

  • Files
  • S3
  • Tergum Cloud
  • Azure Blob
  • GCS (Google Cloud Storage)
  • Container Registry

Notification

  • Email
  • Slack
  • Microsoft Teams
  • Pagerduty

tergum's People

Contributors

kaplan-michael avatar ondrejsika avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

tergum's Issues

Tergum k8s operator

Create a tergum k8s operator.
probably based on operator-sdk with controller-gen for CRD generation.

we'll ideally need a "global" image with all the needed tools embedded

metadata for backup files

Filename

  • file.tar.gz
  • file.tar.gz.tergum-meta.json

Format

{
  meta: {
    version: 1
  },
  created: 123445555,
}

Check free disk space on tmp volume before backup

When you will try to backup huge dir, which will not fit into tmp volume tergum will fail without explanation. We should check if there is approximately enough space and return warning if tmp dir might be too small.

outputFile, err := os.CreateTemp("", "tergum-tar-gz-")

As workaround you can set variable TMPDIR to point to volume with enough space.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.