A command-line tool for distributed backup and restoration of the TiDB cluster data
Home Page: https://pingcap.com/docs/dev/how-to/maintain/backup-and-restore/br/
License: Apache License 2.0
This repository has been moved to https://github.com/pingcap/tidb/tree/master/br.
Current BR will use utils.WithRetry function to retry on function downloadSST and ingestSST. but there are not all error need this retry, for example, file NotFound error.
we need a function to distinguish this error and stop retry.
To improve user experience, we can:
utils.WithRetryAfter pingcap/tidb#15274 merged, we can do backup and restore job in SQL.
we have following things to do:
Sometimes user need one certain timestamp in different backup, such as
current timeago can't solve above problem.
support input timestamp on backup
Try new BR restore with below command:
./br restore full --connect "root@tcp(10.233.58.235:4000)/" --pd 10.233.9.221:2379 --storage "local:///var/lib/tikv/backup"
It fails as below (the logs are repeated continuously):
[2019/11/18 11:11:00.796 +00:00] [WARN] [import.go:93] ["download file failed"] [file="name:\"1_2_22_51_write.sst\" sha256:\"3dbe57eaeeeadae7272239064cbb22dd555d39e2b177b9ce08d5604894af938d\" start_key:\"t\\200\\000\\000\\000\\000\\000\\0003\" end_key:\"t\\200\\000\\000\\000\\000\\000\\0004\" start_version:412632649740320770 end_version:412632649740320770 crc64xor:3733601071017700193 total_kvs:100000 total_bytes:3800000 "] [region="id:100 start_key:\"t\\200\\000\\000\\000\\000\\000\\000\\3770_i\\200\\000\\000\\000\\000\\377\\000\\000\\001\\000\\000\\000\\000\\000\\372\" end_key:\"t\\200\\000\\000\\000\\000\\000\\000\\3770_r\\000\\000\\000\\000\\000\\372\" region_epoch:<conf_ver:5 version:24 > peers:<id:101 store_id:1 > peers:<id:102 store_id:43 > peers:<id:103 store_id:38 > "] [scanStartKey="t\ufffd\u0000\u0000\u0000\u0000\u0000\u0000\ufffd0\u0000\u0000\u0000\u0000\u0000\u0000\u0000\ufffd"] [scanEndKey="t\ufffd\u0000\u0000\u0000\u0000\u0000\u0000\ufffd1\u0000\u0000\u0000\u0000\u0000\u0000\u0000\ufffd"] [error="rpc error: code = Unknown desc = CannotReadExternalStorage(\"local:///var/lib/tikv/backup\", \"1_2_22_51_write.sst\", Os { code: 2, kind: NotFound, message: \"No such file or directory\" })"]
[2019/11/18 11:11:00.795 +00:00] [WARN] [import.go:93] ["download file failed"] [file="name:\"4_1005_22_49_default.sst\" sha256:\"f4b4504a40a436bcbcc3115c0505a6a09ac4533c3597e575621e894d3a0ea501\" start_key:\"t\\200\\000\\000\\000\\000\\000\\0001\" end_key:\"t\\200\\000\\000\\000\\000\\000\\0002\" start_version:412632649740320770 end_version:412632649740320770 crc64xor:8739028299443176440 total_kvs:100000 total_bytes:21200000 "] [region="id:92 start_key:\"t\\200\\000\\000\\000\\000\\000\\000\\377-_i\\200\\000\\000\\000\\000\\377\\000\\000\\001\\000\\000\\000\\000\\000\\372\" end_key:\"t\\200\\000\\000\\000\\000\\000\\000\\377-_r\\000\\000\\000\\000\\000\\372\" region_epoch:<conf_ver:5 version:24 > peers:<id:93 store_id:1 > peers:<id:94 store_id:43 > peers:<id:95 store_id:38 > "] [scanStartKey="t\ufffd\u0000\u0000\u0000\u0000\u0000\u0000\ufffd-\u0000\u0000\u0000\u0000\u0000\u0000\u0000\ufffd"] [scanEndKey="t\ufffd\u0000\u0000\u0000\u0000\u0000\u0000\ufffd.\u0000\u0000\u0000\u0000\u0000\u0000\u0000\ufffd"] [error="rpc error: code = Unknown desc = CannotReadExternalStorage(\"local:///var/lib/tikv/backup\", \"4_1005_22_49_default.sst\", Os { code: 2, kind: NotFound, message: \"No such file or directory\" })"]
^C
But I do have backup sst files under the correct directory:
/var/lib/tikv/backup # pwd
/var/lib/tikv/backup
/var/lib/tikv/backup # ls
1_2_22_51_default.sst 4_1005_22_49_default.sst backupmeta
1_2_22_51_write.sst 4_1005_22_49_write.sst localtmp
The interface ExternalStorage force us to save the entire file into memory when reading or writing.
// ExternalStorage represents a kind of file system storage
type ExternalStorage interface {
// Write file to storage
Write(ctx context.Context, name string, data []byte) error
// Read storage file
Read(ctx context.Context, name string) ([]byte, error)
// FileExists return true if file exists
FileExists(ctx context.Context, name string) (bool, error)
}Suggest a new interface
// ExternalStorage represents a kind of file system storage
type ExternalStorage interface {
// Write file to storage
NewWriter(ctx context.Context, name string) (io.WriteCloser, error)
// Read storage file
NewReader(ctx context.Context, name string) (rc io.ReadCloser, size int64, err error)
// FileExists return true if file exists
FileExists(ctx context.Context, name string) (bool, error)
}Currently, BR outputs many logs to console by default and, as a command-line tool, they are meaningless and noisy to users.
To improve user experience, we can:
Note:
Integrate BR with common cloud object storage (S3, GCS, Azure Blob storage etc).
Currently, BR supports local storage where backup files are stored on local directory. But the backup files need to be collected together and copied to every TiKV node. This is difficult to use in practice, so it's better to mount an NFS like filesystem to every TiKV node and BR node. However, mounting NFS to every node is difficult to set up and error-prone.
Alternatively, object storage is better for this scenario, especially that it's quite common to backup to S3/GCS on public cloud.
Right now, restoring to a cluster to which a backup has already been restored generates a lot of "splitting regions failed" and ["Full Restore"] [progress=0.00%] messages.
br should be able to detect this situation and exit earlier and without so many confusing and repetitive messages.
We need an integration E2E test against S3 storage, to ensure the implementation is correct. cc #89.
Please answer these questions before submitting your issue. Thanks!
What did you do?
If possible, provide a recipe for reproducing the error.
BR didn't rebase auto_random after restore
auto_random to true.create table t(a int key auto_random); insert into t values (), (), ();br backup table --db t --table t --pd xxx -s xxxdrop table tbr restore table --db t --table t --pd xxx -s xxxinsert into t values (),(),();before backup:
MySQL [t]> select a & b'0000011111111111111111111111111' from t.t;
+----------------------------------------+
| a & b'0000011111111111111111111111111' |
+----------------------------------------+
| 1 |
| 2 |
| 3 |
+----------------------------------------+
after restore:
select a & b'0000011111111111111111111111111' from t.t;
+----------------------------------------+
| a & b'0000011111111111111111111111111' |
+----------------------------------------+
| 4 |
| 5 |
| 6 |
| 1 |
| 2 |
| 3 |
+----------------------------------------+
after restore:
select a & b'0000011111111111111111111111111' from t.t;
+----------------------------------------+
| a & b'0000011111111111111111111111111' |
+----------------------------------------+
| 1 |
| 2 |
| 3 |
| 1 |
| 2 |
| 3 |
+----------------------------------------+
BR is currently a CLI program. To some DBAs, this interface is not as straightforward as managing via SQL directly.
We would like to invoke BR directly from TiDB via SQL. This involves several parts:
pingcap/parser need to recognize the BR-related SQL statements.pingcap/tidb need to forward the requests to the BR library.pingcap/br (this repository) need to refactor a bit so TiDB can import it.This issue mainly focuses on the last item.
BR itself imports various packages from TiDB. If such TiDB packages depend on BR again, we get an "import cycle". We need to modify BR to avoid those packages. (The cycle is checked at package level, not module level, so go.mod is hopefully irrelevant.)
According to the current implementation, the argument for --storage will be stored in backup meta and will be retrieved and sent to TiKV during restore.
In this case, for backup & restore with cloud storage, e.g. AWS S3, keys will also be included in the backup meta, which is insecure and should be avoided.
@kennytm @overvenus @tennix What do you think?
Currently, BR only support three modes of backup:
Please answer these questions before submitting your issue. Thanks!
What did you do?
run integration test br_ddl_full.
What did you expect to see?
The test passes in acceptable time duration.
What did you see instead?
On the restore process, the process bar frozen at 100% for a very long period(about 1hr.).
And BR will repeat trying to call DownloadSST and try to ingest it.
Eventually the test will pass, but restore a real strange size of file, alone with huge kv pair size:
Download RPC was totally called about 1,660 times, and at the last SST file, scan regions shows that there are about 834 regions, and grep the log "scan regions", summing the count field, the answer is about 1,600.Hi, I know BR only supports TiDB v3.1 and later versions.
Can we support TiDB v3.0?
db.go#L107 what if create table statement contains comment like /*xxx*/ CREATE /*xxx*/ TABLE
Currently BR (especially restore) requires all stores to be online. If a TiKV store is removed, the whole process would fail.
BR should adapt to scaling up and down, especially for online restore.
None yet.
None yet.
Current BR will break immediately when tikv node get down during backup.we need to handle this error.
To improve user experience, we can:
Support backing up multiple tables in a single command without needing the --backupts trick.
A simplified version of #173. We support the following commands to backup/restore multiple databases or tables.
./br backup database --db firstdb --db seconddb
# ^~~~~~~~~~~~~
./br backup table --???? 'firstdb.firsttable' --???? '`second.db`.`2,tbl`'Because the table name can contain a ., we need to support quoting by `.
Unresolved question:
database --db 'db1,db2'? (currently backing up a single database named "db1,db2"database --db '`db`'? (currently backing up a single database named "`db`"--???? parameter be named?table --db db1 --table tbl1 form?table --???? 'db1.tbl1,db2.tbl2'?table --db db1 --???? tbl1 --???? tbl2?table --db db1 --db db2 --???? tbl1 --???? tbl2?table --???? '"db1,db2"."tbl1,tbl2"'?table --???? '!@#$.%^&*'?Don't support this and go straight to #173.
If we decide to name --???? as --table we'll need to rewrite all documentations.
Please answer these questions before submitting your issue. Thanks!
Backup from a cluster where some TiFlash nodes are running
Success.
Backup failed with "Unimplemented" error in SendBackup(), on this line:
Line 684 in 328ad69
Since TiFlash cannot participate in BR anyway, we should just filter these nodes out.
3.1.0-beta.1
Please answer these questions before submitting your issue. Thanks!
What did you do?
max-index-length to 12288;create table t1 (a varchar(3072) primary key);What did you expect to see?
restore success
What did you see instead?
Error: [ddl:1071]Specified key was too long; max key length is 3072 bytes
What version of BR and TiDB/TiKV/PD are you using?
br master
tidb master
when user already has a table A in cluster.
sometimes if table A partially broken.
user need recover table A from BR but don't want to drop origin table A.
so a kind of table route function is needed.
table route in restore.
not sure if we need introduce config file to support this funciton.
BR only supports full backup & restore for now, in this project we require to support incremental backup & restore, this is related to BR and TiKV.
Currently, the procedure of backup is:
And the procedure of restore is:
Several problems are raised as we support incremental backup & restore:
At least support the basic function of incremental backup & restore.
DeltaScanner (500 points / medium)2700
VIEW is an important feature of TiDB. For now, BR only support for independent VIEWs. We need to support more complex VIEWs(depending on other tables and views).
TiDB has a concept called "bootstrap version" (tidb_server_version). It represents the schemas of the system tables. If a TiDB server with a newer bootstrap version (e.g. v41) is started on a storage with older version (e.g. v40), the TiDB server will perform a "rolling upgrade" (execute some DDL statements) to migrate the schemas to v41.
This action is also performed when BR uses its own TiDB to bootstrap the session.
Line 133 in 56e206e
This brings us to the problem:
Thus the only safe solution is to ensure BR version = cluster TiDB version.
How should our version management cope with this?
This was revealed in #158 (https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/br_ghpr_unit_and_integration_test/detail/br_ghpr_unit_and_integration_test/828/pipeline/) due to pingcap/tidb#14598 renaming the column password (v40) to authentication_string (v41) in the system table mysql.user.
local:// is confusing because SST files are distributed into multiple machines on backup, and must be manually copied on restore.
An alternative without requiring cloud storage is that we automatically set up a local file server. TiKV would then read and write directly on the BR machine.
Since we have already implemented S3 read/write, we could embed a MinIO server into BR and advertise it directly.
We should add a new scheme br:///path/local/backup/archive, which will
:port, where the port is taken from --advertise-addr (-A)s3://generated_bucket/ when sent downstream to TiKV. The bucket links directly to /path/local/backup/archive--s3.provider=minio.Besides S3, we could try other protocols like TFTP, NFS, custom HTTP, or custom gRPC. But in all these cases we need to implement more new code.
--s3.endpoint can be auto-generated using get_if_addrs, but the output may not be reliable.
Replace all mentions of local:// by the local S3 server.
When TiDB cluster enables TLS connection, BR won't work.
BR needs to support TLS connection for security.
This is a tracking issue of tikv/tikv#7209. Please discuss over there.
The TLS test was disabled by #204 so that other PRs can move forward. This PR should be reverted once tikv/tikv#7209 is fixed.
When I do a full backup, it keeps doing backup and printing progress=100% for a long time after the it first prints progress=100%.
Logs are like this:
[2019/12/20 15:58:26.674 +08:00] [INFO] [progress.go:100] ["Database backup"] [progress=100.00%]
[2019/12/20 15:58:26.874 +08:00] [INFO] [progress.go:100] ["Database backup"] [progress=100.00%]
[2019/12/20 15:58:27.074 +08:00] [INFO] [progress.go:100] ["Database backup"] [progress=100.00%]
[2019/12/20 15:58:27.274 +08:00] [INFO] [progress.go:100] ["Database backup"] [progress=100.00%]
[2019/12/20 15:58:27.474 +08:00] [INFO] [progress.go:100] ["Database backup"] [progress=100.00%]
[2019/12/20 15:58:27.674 +08:00] [INFO] [progress.go:100] ["Database backup"] [progress=100.00%]
[2019/12/20 15:58:27.702 +08:00] [INFO] [client.go:683] ["range backuped"] [StartKey="dIAAAAAAAAFQX3KAAAAAAAZfmw=="] [EndKey="dIAAAAAAAAF
QX3KAAAAAAAvJXQ=="]
[2019/12/20 15:58:27.874 +08:00] [INFO] [progress.go:100] ["Database backup"] [progress=100.00%]
[2019/12/20 15:58:28.074 +08:00] [INFO] [progress.go:100] ["Database backup"] [progress=100.00%]
[2019/12/20 15:58:28.274 +08:00] [INFO] [progress.go:100] ["Database backup"] [progress=100.00%]
[2019/12/20 15:58:28.407 +08:00] [INFO] [client.go:683] ["range backuped"] [StartKey="dIAAAAAAAAFQX3KAAAAAAAvJXQ=="] [EndKey="dIAAAAAAAAF
QX3L//////////wA="]
[2019/12/20 15:58:28.408 +08:00] [INFO] [client.go:676] ["backup streaming finish"] [StoreID=1]
[2019/12/20 15:58:28.408 +08:00] [INFO] [client.go:379] ["finish backup push down"] [Ok=4]
[2019/12/20 15:58:28.408 +08:00] [INFO] [client.go:392] ["backup time range"] [StartVersion=0] [EndVersion=413356152248860683]
[2019/12/20 15:58:28.408 +08:00] [INFO] [client.go:405] ["backup range finished"] [take=3.237989556s]
[2019/12/20 15:58:28.408 +08:00] [INFO] [client.go:349] ["backup started"] [StartKey="dIAAAAAAAAFQX2mAAAAAAAAAAQA="] [EndKey="dIAAAAAAAAFQX2mAAAAAAAAAAfs="] [RateLimit=0] [Concurrency=4]
[2019/12/20 15:58:28.409 +08:00] [INFO] [push.go:24] ["new backup client"]
[2019/12/20 15:58:28.409 +08:00] [INFO] [client.go:664] ["try backup"] ["backup request"="{\"cluster_id\":6771082893009782418,\"start_key\":\"dIAAAAAAAAFQX2mAAAAAAAAAAQA=\",\"end_key\":\"dIAAAAAAAAFQX2mAAAAAAAAAAfs=\",\"end_version\":413356152248860683,\"concurrency\":4,\"storage_backend\":{\"Backend\":{\"Local\":{\"path\":\"/data2/yilin/bak/full\"}}}}"]
[2019/12/20 15:58:28.474 +08:00] [INFO] [progress.go:100] ["Database backup"] [progress=100.00%]
[2019/12/20 15:58:28.674 +08:00] [INFO] [progress.go:100] ["Database backup"] [progress=100.00%]
[2019/12/20 15:58:28.874 +08:00] [INFO] [progress.go:100] ["Database backup"] [progress=100.00%]
[2019/12/20 15:58:29.074 +08:00] [INFO] [progress.go:100] ["Database backup"] [progress=100.00%]
[2019/12/20 15:58:29.274 +08:00] [INFO] [progress.go:100] ["Database backup"] [progress=100.00%]
[2019/12/20 15:58:29.385 +08:00] [INFO] [client.go:683] ["range backuped"] [StartKey="dIAAAAAAAAFQX2mAAAAAAAAAAQA="] [EndKey="dIAAAAAAAAFQX2mAAAAAAAAAAfs="]
[2019/12/20 15:58:29.385 +08:00] [INFO] [client.go:676] ["backup streaming finish"] [StoreID=1]
Integration test fails #214 (comment)
pretty printed backtrace
[2020-03-30T09:59:53.436Z] [2020/03/30 17:59:53.217 +08:00] [ERROR] [restore.go:238] ["split regions failed"] [error="split region failed: region=id:3828 start_key:\"t\\200\\000\\000\\000\\000\\000\\003\\377\\275_r\\000\\000\\000\\000\\000\\372\" end_key:\"t\\200\\000\\000\\000\\000\\000\\003\\377\\277\\000\\000\\000\\000\\000\\000\\000\\370\" region_epoch:<conf_ver:20 version:1049 > peers:<id:3829 store_id:6 > peers:<id:3830 store_id:1 > peers:<id:3831 store_id:5 > , err=message:\"peer is not leader for region 3828, leader may Some(id: 3831 store_id: 5)\" not_leader:<region_id:3828 leader:<id:3831 store_id:5 > > "] [errorVerbose="split region failed: region=id:3828 start_key:\"t\\200\\000\\000\\000\\000\\000\\003\\377\\275_r\\000\\000\\000\\000\\000\\372\" end_key:\"t\\200\\000\\000\\000\\000\\000\\003\\377\\277\\000\\000\\000\\000\\000\\000\\000\\370\" region_epoch:<conf_ver:20 version:1049 > peers:<id:3829 store_id:6 > peers:<id:3830 store_id:1 > peers:<id:3831 store_id:5 > , err=message:\"peer is not leader for region 3828, leader may Some(id: 3831 store_id: 5)\" not_leader:<region_id:3828 leader:<id:3831 store_id:5 > >
github.com/pingcap/br/pkg/restore.(*pdClient).BatchSplitRegions
/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/pkg/restore/split_client.go:230
github.com/pingcap/br/pkg/restore.(*RegionSplitter).splitAndScatterRegions
/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/pkg/restore/split.go:316
github.com/pingcap/br/pkg/restore.(*RegionSplitter).Split
/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/pkg/restore/split.go:118
github.com/pingcap/br/pkg/restore.SplitRanges
/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/pkg/restore/util.go:344
github.com/pingcap/br/pkg/task.RunRestore
/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/pkg/task/restore.go:236
github.com/pingcap/br/cmd.runRestoreCommand
/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/cmd/restore.go:21
github.com/pingcap/br/cmd.newDbRestoreCommand.func1
/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/cmd/restore.go:93
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/[email protected]/command.go:826
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/[email protected]/command.go:914
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
github.com/pingcap/br.main
/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main.go:54
github.com/pingcap/br.TestRunMain.func1
/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main_test.go:39
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/log.Error
/go/pkg/mod/github.com/pingcap/[email protected]/global.go:42
github.com/pingcap/br/pkg/task.RunRestore
/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/pkg/task/restore.go:238
github.com/pingcap/br/cmd.runRestoreCommand
/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/cmd/restore.go:21
github.com/pingcap/br/cmd.newDbRestoreCommand.func1
/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/cmd/restore.go:93
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/[email protected]/command.go:826
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/[email protected]/command.go:914
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
github.com/pingcap/br.main
/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main.go:54
github.com/pingcap/br.TestRunMain.func1
/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main_test.go:39"]
br/pkg/restore/split_client.go
Lines 218 to 231 in e476c82
BR should retry on RegionError:
Currently we'll reject BR restore if the cluster contains TiFlash, because TiFlash doesn't support all features of sst_importer yet.
We should be able to skip TiFlash nodes during (full, offline) restore.
What did you do?
Perform restore with dataset 2T size, restore progress bar is inaccurate, after check the code found that progress has increase more than len(keys) when split meet error, need to fix it
What did you expect to see?
accurate progress bar during restore .
What did you see instead?
100% progress for a long time.
What version of BR and TiDB/TiKV/PD are you using?
BR master
Mydumper will output the commitTS after backup finished, it's better that BR also supports this which is useful to know the backup position.
Started dump at: 2019-06-13 10:00:04
SHOW MASTER STATUS:
Log: tidb-binlog
Pos: 409054741514944513
GTID:
Finished dump at: 2019-06-13 10:00:04
The current tests fail CI almost every time, we need to find out why and stabilize it.
create table ta(a int(11) NOT NULL /*T!30100 AUTO_RANDOM(5) */, PRIMARY KEY (a))Invalid auto random: auto_random is an experimental feature, which can only be used when allow-auto-random is enabled. This can be changed in the configuration.
What did you expect to see?
incremental restore success
What did you see instead?
Invalid auto random: auto_random is an experimental feature, which can only be used when allow-auto-random is enabled. This can be changed in the configuration.
When we backup a empty TiDB cluster, It will generate a empty bucket folder in storage, and when I restore it , it will failed.
I want the backup of empty TiDB cluster can be restore
pd.go#L102 also need to check group ID?
Currently, if we restore from local:// storage, the entire backup archive must be copied to every TiKV machines. While we did warn for this caveat, not every user follow the instruction correctly. In particular, users testing BR who just backup and then immediately restore will encounter error.
Copying also means the actual storage space is multiplied by the number of machines, which is not affordable in some use cases.
Rather than letting users to copy the files themselves (which is error prone), we try to automatically transfer the SST files if it is missing.
We will add two gRPC APIs to sst_importer (or a new service):
FileExists(local_path_name: string) -> boolReadFile(local_path_name: string) -> stream<bytes>When reading from local:// storage, we will perform the following:
function read(local_path_name: string) -> stream<bytes> {
if (file_exists(local_path_name)) {
return open(local_path_name);
}
let peers_with_file = get_peers().filter(p => p.FileExists(local_path_name));
if (peers_with_file.is_empty()) {
throw FileNotFound(local_path_name);
}
return peers_with_file[0].ReadFile(local_path_name);
}Implementation details:
peers_with_file[0] can be replaced by better balancing strategyFileExists returning "unimplemented" or other errors should be treated as "false". Therefore this strategy is backward-compatible.An alternative is deprecate local:// entirely and stop mentioning it in all documentations.
We could stop "requiring" copying SSTs to every machine, but it is still recommended to reduce network traffic.
Try new br with full backup.
Here is the cluster:
[[email protected] tidb-cluster]$kubectl get pod -n dan5
NAME READY STATUS RESTARTS AGE
dan5-pd-0 1/1 Running 0 112m
dan5-pd-1 1/1 Running 0 113m
dan5-pd-2 1/1 Running 0 114m
dan5-tidb-0 2/2 Running 0 42m
dan5-tidb-1 2/2 Running 0 43m
dan5-tikv-0 1/1 Running 0 111m
dan5-tikv-1 1/1 Running 0 112m
dan5-tikv-2 1/1 Running 0 112m
Run backup with below command:
./br backup full --pd 10.233.40.168:2379 --storage local:///var/lib/tikv/backup
Check result:
[[email protected] tidb-cluster]$kubectl exec -it dan5-tikv-0 -n dan5 -- sh -c "ls -lartR /var/lib/tikv/backup"
/var/lib/tikv/backup:
total 20324
drwxr-xr-x 7 root root 102 Nov 18 09:17 ..
drwxr-xr-x 2 root root 6 Nov 18 09:17 localtmp
-rw-r--r-- 1 root root 1731074 Nov 18 09:17 4_1005_22_49_write.sst
-rw-r--r-- 1 root root 19077269 Nov 18 09:17 4_1005_22_49_default.sst
drwxr-xr-x 3 root root 84 Nov 18 09:17 .
/var/lib/tikv/backup/localtmp:
total 0
drwxr-xr-x 3 root root 84 Nov 18 09:17 ..
drwxr-xr-x 2 root root 6 Nov 18 09:17 .
[[email protected] tidb-cluster]$kubectl exec -it dan5-tikv-2 -n dan5 -- sh -c "ls -lartR /var/lib/tikv/backup"
/var/lib/tikv/backup:
total 20328
drwxr-xr-x 7 root root 112 Nov 18 08:21 ..
drwxr-xr-x 2 root root 6 Nov 18 09:17 localtmp
-rw-r--r-- 1 root root 4589 Nov 18 09:17 backupmeta
-rw-r--r-- 1 root root 1726923 Nov 18 09:17 1_2_22_51_write.sst
-rw-r--r-- 1 root root 19076989 Nov 18 09:17 1_2_22_51_default.sst
drwxr-xr-x 3 root root 96 Nov 18 09:17 .
/var/lib/tikv/backup/localtmp:
total 0
drwxr-xr-x 3 root root 96 Nov 18 09:17 ..
drwxr-xr-x 2 root root 6 Nov 18 09:17 .
[[email protected] tidb-cluster]$kubectl exec -it dan5-tikv-1 -n dan5 -- sh -c "ls -lartR /var/lib/tikv/backup"
/var/lib/tikv/backup:
total 0
drwxr-xr-x 2 root root 6 Nov 18 09:17 localtmp
drwxr-xr-x 7 root root 102 Nov 18 09:17 ..
drwxr-xr-x 3 root root 22 Nov 18 09:17 .
/var/lib/tikv/backup/localtmp:
total 0
drwxr-xr-x 3 root root 22 Nov 18 09:17 ..
drwxr-xr-x 2 root root 6 Nov 18 09:17 .
However, there is no backup file on tikv-1 even if there are 20 region leaders of 22 regions totally on it:
[[email protected] test]$curl http://10.233.40.168:2379/pd/api/v1/store/5
{
"store": {
"id": 5,
"address": "dan5-tikv-1.dan5-tikv-peer.dan5.svc:20160",
"labels": [
{
"key": "host",
"value": "172.16.4.249"
}
],
"version": "3.0.3",
"state_name": "Up"
},
"status": {
"capacity": "440.9GiB",
"available": "427.7GiB",
"leader_count": 20,
"leader_weight": 1,
"leader_score": 20,
"leader_size": 20,
"region_count": 22,
"region_weight": 1,
"region_score": 112,
"region_size": 112,
"start_ts": "2019-11-18T08:01:55Z",
"last_heartbeat_ts": "2019-11-18T09:59:56.491943663Z",
"uptime": "1h58m1.491943663s"
}
}
Please help confirm if this is expected behavior since per design, the sst files should be created by region leaders.
BTW, checksum fails because br wants to check the files on remote tikv, not sure if I run the command missing some options:
/var/lib/tikv # ./br meta checksum -s local:///var/lib/tikv/backup/
[2019/11/18 09:47:46.458 +00:00] [INFO] [version.go:22] ["Welcome to Backup & Restore (BR)"]
[2019/11/18 09:47:46.458 +00:00] [INFO] [version.go:23] [BR] [release-version=]
[2019/11/18 09:47:46.458 +00:00] [INFO] [version.go:24] [BR] [git-hash=bbd137c28cefc595ba8a3c8c98db378028a7b553]
[2019/11/18 09:47:46.459 +00:00] [INFO] [version.go:25] [BR] [git-branch=master]
[2019/11/18 09:47:46.459 +00:00] [INFO] [version.go:26] [BR] [utc-build-time="2019-11-18 08:12:51"]
[2019/11/18 09:47:46.459 +00:00] [INFO] [version.go:43] [arguments] [ca=] [cert=] [help=false] [key=] [log-file=] [log-level=info] [pd=127.0.0.1:2379] [slow-log-file=] [status-addr=] [storage=local:///var/lib/tikv/backup/]
[2019/11/18 09:47:46.472 +00:00] [INFO] [meta.go:85] ["file info"] [table=sbtest1] [file=4_1005_22_49_default.sst] [crc64xor=8739028299443176440] [totalKvs=100000] [totalBytes=21200000] [startVersion=412632649740320770] [endVersion=412632649740320770] [startKey=dIAAAAAAAAAx] [endKey=dIAAAAAAAAAy]
Error: open /var/lib/tikv/backup/4_1005_22_49_default.sst: no such file or directory
open /var/lib/tikv/backup/4_1005_22_49_default.sst: no such file or directory
github.com/pingcap/errors.AddStack
/root/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174
github.com/pingcap/errors.Trace
/root/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15
github.com/pingcap/br/cmd.NewMetaCommand.func2
/home/dev/br/cmd/meta.go:99
github.com/spf13/cobra.(*Command).execute
/root/go/pkg/mod/github.com/spf13/[email protected]/command.go:826
github.com/spf13/cobra.(*Command).ExecuteC
/root/go/pkg/mod/github.com/spf13/[email protected]/command.go:914
github.com/spf13/cobra.(*Command).Execute
/root/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main
/home/dev/br/main.go:56
runtime.main
/usr/local/go/src/runtime/proc.go:203
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357
Please answer these questions before submitting your issue. Thanks!
First set up a TiDB with alter-primary-key = true.
Create a table:
create table test.a (a int primary key, b int unique);
insert into test.a values (42, 42);Run backup
bin/br backup table --db test -t a -s 'local:///tmp/brapk'Drop the table
drop table test.a;Restore
bin/br restore table --db test -t a -s 'local:///tmp/brapk'Restore success and checksum pass
An "ranges overlapped" error
[2020/03/27 13:38:22.085 +00:00] [ERROR] [restore.go:225] ["split regions failed"] [error="ranges overlapped: [74800000000000003b5f69800000000000000100 74800000000000003b5f698000000000000001fb],
Cluster: 4.0
BR: master
In this project we needs focus on TiKV and BR project. and we need to
For now TiKV is an independent CNCF project, the ability of backup and restore data is become more important, so we need to implement backup and restore data in TiKV with BR.
Raw KV is a concept of TiKV and irrelevant with tidb, current BR already support backup Txn KV from TiDB TiKV cluster and restore Txn KV data to new cluster.
we can reuse above process to implement Raw KV backup and restore, so the main process can be divided into two section.
Backup:
Restore:
Please answer these questions before submitting your issue. Thanks!
What did you do?
sometimes after backup/restore succeed, br still log some error logs
What did you expect to see?
cleaned log
What did you see instead?
lots of
[2020/03/07 16:35:47.768 +08:00] [INFO] [coprocessor.go:855] ["[TIME_COP_PROCESS] resp_time:805.319684ms txnStartTS:415123386115555329 region_id:1231 store_addr:172.16.5.204:20191 kv_process_ms:734 scan_total_write:415965 scan_processed_write:415964 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"]
{"level":"warn","ts":"2020-03-07T16:35:50.309+0800","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-ee6aac04-29bd-4a17-a93b-fd32c6aa18cb/172.16.5.204:2379","attempt":30,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
BR should provide the total size of one backup considering following cases:
Maybe TiKV can return the size of the files it created for each backup to BR and BR calculates the total size from all the TiKVs and writes the size to backup meta.
Currently, BR only supports offline restore, but some users need online restore.
The goal of this project: to support online restore, during which the online
query should not have a significant impact.
BR will adjust TiKV to import mode during restore. In this mode, TiKV can
quickly import a large amount of data through RocksDB IngestSST, but this mode
will affect the online query seriously.
During restore, BR will split a large number of small regions and randomly
scatter them to each TiKV. This will affect PD scheduling decisions, which will
have an unpredictable impact on the online query.
After restore is complete, the entire cluster executes full Compaction,
which will affect the IO/CPU greatly.
Before the restore is completed, the restore data is inconsistent, but there is
no restriction to prevent TiDB from operating on the data.
Online import scenarios can be roughly divided into two categories:
For case 1, we can:
For case 2, we can:
In both cases, TiDB needs to support making the restored database/table
invisible to the user.
GanttStart: YYYY-MM-DD
GanttDue: YYYY-MM-DD
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.