aerospike / aerospike-tools-backup Goto Github PK

View Code? Open in Web Editor NEW

25.0 48.0 20.0 2.41 MB

Aerospike backup and restore utility

Home Page: http://www.aerospike.com/docs/tools/backup/

License: Apache License 2.0

Makefile 1.66% C 76.15% C# 1.70% Shell 0.61% Python 12.28% C++ 7.60%

aerospike-tools-backup's Issues

Small mistake in the documentation

http://www.aerospike.com/docs/tools/backup/asrestore.html

this example:
$ asrestore --host 1.2.3.4 --directory backup_2015_08_24 --namespace test,prod

should be:
$ asrestore --host 1.2.3.4 --directory backup_2015_08_24 --namespace test prod

ASBackup not working C-4.5.0.9

Hi,

I am not able to back up my aerospike data. I have posted a question on aerospike forum but got no answer yet.

https://discuss.aerospike.com/t/asbackup-not-working-c-4-5-0-5/6077

Thanks

How can I deserialize the list bins?

I want to use to use this tool to generate a backup, but also I want to be able to process that backup file in order to build a secondary index in a different database. I know I can perform an scan, but for my particular problem, processing the file might be more interesting.

asrestore: Unexpected character \x0a in backup block (line 4, col 46), expected " "

Hi!

I did backup of namespace with asbackup 3.10.0 in aerospike 6.0.0.1 and tried to restore with asrestore 3.12.0 in aerospike 6.1.0.3.

Here is redacted output:

2022-10-10 10:22:16 GMT [INF] [26865] Starting restore to localhost:3000 (bins: [all], sets: [all]) from ...
2022-10-10 10:22:16 GMT [INF] [26865] Processing 3 node(s)
2022-10-10 10:22:16 GMT [INF] [26865] Restoring ...
2022-10-10 10:22:16 GMT [INF] [26865] Opened backup file ...
2022-10-10 10:22:17 GMT [INF] [26865] Restoring records
2022-10-10 10:22:17 GMT [ERR] [26942] Unexpected character \x0a in backup block (line 4, col 46), expected " "
2022-10-10 10:22:17 GMT [ERR] [26942] Error while restoring backup file ...
2022-10-10 10:22:17 GMT [INF] [26901] 0 UDF file(s), 0 secondary index(es), 0 record(s) (0 rec/s, 253 KiB/s, 0 B/rec, retries: 0)
2022-10-10 10:22:17 GMT [INF] [26901] Expired 0 : skipped 0 : err_ignored 0 : inserted 0: failed 0 (existed 0 , fresher 0)
2022-10-10 10:22:17 GMT [INF] [26901] 0% complete, ~1d4h6m38s remaining

I've successfully restored backup with asrestore 3.10.0.

Backup file format documentation should state that space after bytes-length is optional

it looks like space after bytes-length in bin data in records section (["-"] [SP] ["B"] ["!"]? [SP] [escape({bin-name})] [SP] [{bytes-length}] [SP] [{bytes-data}] [LF]) is optional if bytes-length is 0. This should be documented in format.

Is this the case with record header (["+"] [SP] ["k"] [SP] ["B"] ["!"]? [SP] [{bytes-length}] [SP] [{bytes-data}] [LF]) as well?

readme does not mention which versions of dependencies to install

The readme implies that the latest version of each dependency is needed to successfully build asbackup. However, looking at the mac-artifact.yml workflow it requires specific versions of each package

Add S3 checksum option for enhanced data integrity

Our client wants to ensure the data integrity of objects being uploaded to S3 buckets is not compromised during the upload. AWS S3 SDK has the ability to automatically create a checksum of the object on client-side to ensure data integrity during transfer. The checksum is then checked at the end of the upload.

Adding an option like --s3-checksum HASH_FUNC or similar to asbackup, would set the checksum algorithm for uploading the object with AWS SDK, hence enabling this feature. With hashes printed out to the console, we could make sure the objects arrive to the S3 bucket without compromising the integrity of the data.

Here are the links to the documentation about the S3 checksum feature:

asrestore (3.9.1) sometimes doesn't restore indexes

asrestore (3.9.1) sometimes doesn't restore indexes.

In order to demonstrate this problem, I installed fresh aerospike on my machine (Ubuntu 16.04 LTS).

Package info:

$ dpkg -l | grep aero | awk '{print $2 " " $3}'
aerospike-server-community 3.9.1-1
aerospike-tools 3.9.1

$ md5sum Downloads/aerospike-server-community-3.9.1-ubuntu16.04.tgz 
ff7fe8665f84e582ed0319b54d827c2f  Downloads/aerospike-server-community-3.9.1-ubuntu16.04.tgz

Config info (everything else is default config):

namespace backuptest {
    replication-factor   2
    memory-size          1G
    default-ttl          0
    ldt-enabled          true

    storage-engine device {

        file /opt/aerospike/data/backuptest.dat

        filesize            2G

        data-in-memory      true
    }
}

Inserted one record and one index:

$ aql
Aerospike Query Client
Version 3.9.1
Copyright 2012-2016 Aerospike. All rights reserved.

aql> show sets
0 rows in set (0.001 secs)
OK

aql> insert into backuptest.aaa(PK, bin) values (111 , 'val')
OK, 1 record affected.

aql> show sets
+------------------+--------------+----------------+---------+-------------------+-------+-------------------+----------+
| disable-eviction | ns           | set-enable-xdr | objects | stop-writes-count | set   | memory_data_bytes | deleting |
+------------------+--------------+----------------+---------+-------------------+-------+-------------------+----------+
| "false"          | "backuptest" | "use-default"  | 1       | 0                 | "aaa" | 22                | "false"  |
+------------------+--------------+----------------+---------+-------------------+-------+-------------------+----------+
1 row in set (0.001 secs)
OK

aql> show indexes backuptest
0 rows in set (0.000 secs)
OK

aql> create index idx on backuptest.aaa (PK) numeric
OK, 1 index added.

aql> show indexes backuptest
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
| ns           | bin  | indextype | set   | state | indexname | path | sync_state | type      |
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
| "backuptest" | "PK" | "NONE"    | "aaa" | "RW"  | "idx"     | "PK" | "synced"   | "NUMERIC" |
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
1 row in set (0.001 secs)
OK

Created backup:

$ asbackup -v --host 127.0.0.1 --namespace backuptest --directory backuptest
2016-08-22 13:21:41 GMT [INF] [11298] Starting 100% backup of 127.0.0.1:3000 (namespace: backuptest, set: [all], bins: [all]) to backuptest
2016-08-22 13:21:41 GMT [VER] [11298] Connecting to cluster
2016-08-22 13:21:41 GMT [INF] [11298] [src/main/aerospike/as_cluster.c:440][as_cluster_seed_nodes] Add node BB9A31ECA9D7B50 127.0.0.1:3000
2016-08-22 13:21:41 GMT [VER] [11298] [src/main/aerospike/as_node.c:541][as_node_process_response] Node BB9A31ECA9D7B50 partition generation changed: 1
2016-08-22 13:21:41 GMT [VER] [11298] [src/main/aerospike/as_cluster.c:931][as_cluster_add_seeds] Add seed 127.0.0.1:3000
2016-08-22 13:21:41 GMT [VER] [11298] Adding node BB9A31ECA9D7B50
2016-08-22 13:21:41 GMT [INF] [11298] Processing 1 node(s)
2016-08-22 13:21:41 GMT [VER] [11298] Getting cluster object count
2016-08-22 13:21:41 GMT [INF] [11298] Node ID             Objects        Replication    
2016-08-22 13:21:41 GMT [VER] [11298] Getting object count for node BB9A31ECA9D7B50
2016-08-22 13:21:41 GMT [VER] [11298] Getting info value namespace/backuptest for node BB9A31ECA9D7B50
2016-08-22 13:21:41 GMT [VER] [11298] Parsing info
2016-08-22 13:21:41 GMT [INF] [11298] BB9A31ECA9D7B50     1              1              
2016-08-22 13:21:41 GMT [INF] [11298] Namespace contains 1 record(s)
2016-08-22 13:21:41 GMT [VER] [11298] Preparing backup directory backuptest
2016-08-22 13:21:41 GMT [INF] [11298] Directory backuptest does not exist, creating
2016-08-22 13:21:41 GMT [INF] [11298] Directory backuptest prepared for backup
2016-08-22 13:21:41 GMT [VER] [11298] Creating counter thread
2016-08-22 13:21:41 GMT [VER] [11298] Pushing 1 job(s) to job queue
2016-08-22 13:21:41 GMT [VER] [11298] Creating 1 backup thread(s)
2016-08-22 13:21:41 GMT [VER] [11298] Waiting for 1 backup thread(s)
2016-08-22 13:21:41 GMT [VER] [11317] Entering backup thread
2016-08-22 13:21:41 GMT [INF] [11317] Starting backup for node BB9A31ECA9D7B50
2016-08-22 13:21:41 GMT [VER] [11317] 1 remaining record(s), 0 B/rec average size
2016-08-22 13:21:41 GMT [VER] [11317] Opening backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:21:41 GMT [VER] [11317] Creating backup file
2016-08-22 13:21:41 GMT [VER] [11316] Entering counter thread
2016-08-22 13:21:41 GMT [VER] [11317] Checking disk space on backuptest for 0 byte(s)
2016-08-22 13:21:41 GMT [INF] [11317] Created new backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:21:41 GMT [VER] [11317] Initializing backup file
2016-08-22 13:21:41 GMT [VER] [11317] Picked up first job, doing one shot work
2016-08-22 13:21:41 GMT [VER] [11317] Processing secondary indexes
2016-08-22 13:21:41 GMT [INF] [11317] Backing up 1 secondary index(es)
2016-08-22 13:21:41 GMT [VER] [11317] Storing index idx
2016-08-22 13:21:41 GMT [VER] [11317] Processing UDFs
2016-08-22 13:21:41 GMT [INF] [11317] Backing up 0 UDF file(s)
2016-08-22 13:21:41 GMT [VER] [11317] Signaling one shot work completion
2016-08-22 13:21:41 GMT [VER] [11317] Received scan end marker
2016-08-22 13:21:41 GMT [INF] [11317] Completed backup for node BB9A31ECA9D7B50, records: 1, size: 168 (~168 B/rec)
2016-08-22 13:21:41 GMT [VER] [11317] Closing backup file
2016-08-22 13:21:41 GMT [VER] [11317] Closing file descriptor
2016-08-22 13:21:41 GMT [VER] [11317] File size is 168
2016-08-22 13:21:41 GMT [VER] [11317] Job queue is empty
2016-08-22 13:21:41 GMT [VER] [11317] Leaving backup thread
2016-08-22 13:21:41 GMT [VER] [11298] Waiting for counter thread
2016-08-22 13:21:42 GMT [INF] [11316] Backed up 1 record(s), 1 secondary index(es), 0 UDF file(s) from 1 node(s), 168 byte(s) in total (~168 B/rec)
2016-08-22 13:21:42 GMT [VER] [11316] Leaving counter thread
2016-08-22 13:21:42 GMT [VER] [11298] Exiting with status code 0

Check index is still there:

$ aql
Aerospike Query Client
Version 3.9.1
Copyright 2012-2016 Aerospike. All rights reserved.
aql> show indexes backuptest
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
| ns           | bin  | indextype | set   | state | indexname | path | sync_state | type      |
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
| "backuptest" | "PK" | "NONE"    | "aaa" | "RW"  | "idx"     | "PK" | "synced"   | "NUMERIC" |
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
1 row in set (0.003 secs)
OK

I ran several times asrestore and index is still alive:

$ asrestore -v --host 127.0.0.1 --directory backuptest -g
2016-08-22 13:23:14 GMT [INF] [11549] Starting restore to 127.0.0.1:3000 (bins: [all], sets: [all]) from backuptest
2016-08-22 13:23:14 GMT [VER] [11549] Connecting to cluster
2016-08-22 13:23:14 GMT [VER] [11549] Adding node BB9A31ECA9D7B50
2016-08-22 13:23:14 GMT [INF] [11549] Processing 1 node(s)
2016-08-22 13:23:14 GMT [VER] [11549] Creating counter thread
2016-08-22 13:23:14 GMT [VER] [11549] Listing backup files in backuptest
2016-08-22 13:23:14 GMT [VER] [11567] Entering counter thread
2016-08-22 13:23:14 GMT [INF] [11549] Found 1 backup file(s) in backuptest
2016-08-22 13:23:14 GMT [VER] [11549] Triaging 1 backup file(s)
2016-08-22 13:23:14 GMT [VER] [11549] Opening backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:23:14 GMT [VER] [11549] Getting file descriptor
2016-08-22 13:23:14 GMT [INF] [11549] Opened backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:23:14 GMT [VER] [11549] Validating backup file version
2016-08-22 13:23:14 GMT [VER] [11549] Found first backup file: backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:23:14 GMT [VER] [11549] Parsing global index and UDF data (line 4)
2016-08-22 13:23:14 GMT [VER] [11549] Parsing index in line 4
2016-08-22 13:23:14 GMT [VER] [11549] Index: idx (on set aaa)
2016-08-22 13:23:14 GMT [VER] [11549] Closing backup file
2016-08-22 13:23:14 GMT [VER] [11549] Closing file descriptor
2016-08-22 13:23:14 GMT [VER] [11549] Pushing 1 exclusive job(s) to job queue
2016-08-22 13:23:14 GMT [INF] [11549] Restoring 0 UDF file(s)
2016-08-22 13:23:14 GMT [INF] [11549] Restoring 1 secondary index(es)
2016-08-22 13:23:14 GMT [VER] [11549] Restoring index backuptest:aaa:idx (PK)
2016-08-22 13:23:14 GMT [INF] [11549] Restoring records
2016-08-22 13:23:14 GMT [VER] [11549] Creating 1 restore thread(s)
2016-08-22 13:23:14 GMT [VER] [11549] Waiting for 1 restore thread(s)
2016-08-22 13:23:14 GMT [VER] [11568] Entering restore thread
2016-08-22 13:23:14 GMT [INF] [11568] Restoring backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:23:14 GMT [VER] [11568] Opening backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:23:14 GMT [VER] [11568] Getting file descriptor
2016-08-22 13:23:14 GMT [INF] [11568] Opened backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:23:14 GMT [VER] [11568] Validating backup file version
2016-08-22 13:23:14 GMT [VER] [11568] Existence policy is default
2016-08-22 13:23:14 GMT [VER] [11568] Generation policy is default
2016-08-22 13:23:14 GMT [VER] [11568] Parsing index in line 4
2016-08-22 13:23:14 GMT [VER] [11568] Index: idx (on set aaa)
2016-08-22 13:23:14 GMT [VER] [11568] Ignoring index block
2016-08-22 13:23:14 GMT [VER] [11568] 1 per-thread record(s) (1 rec/s), read latency: 12 (0) us, store latency: 102 (1) us
2016-08-22 13:23:14 GMT [VER] [11568] Encountered end of file (line 12, col 1)
2016-08-22 13:23:14 GMT [VER] [11568] End of backup file reached
2016-08-22 13:23:14 GMT [VER] [11568] Closing backup file
2016-08-22 13:23:14 GMT [VER] [11568] Closing file descriptor
2016-08-22 13:23:14 GMT [VER] [11568] Job queue is empty
2016-08-22 13:23:14 GMT [VER] [11568] Leaving restore thread
2016-08-22 13:23:14 GMT [VER] [11549] Freeing 1 index(es)
2016-08-22 13:23:14 GMT [VER] [11549] Freeing 0 UDF file(s)
2016-08-22 13:23:14 GMT [VER] [11549] Waiting for counter thread
2016-08-22 13:23:15 GMT [INF] [11567] 0 UDF file(s), 1 secondary index(es), 1 record(s) (0 KiB/s, 1 rec/s, 168 B/rec, backed off: 0)
2016-08-22 13:23:15 GMT [INF] [11567] Expired 0 : skipped 0 : inserted 1 : failed 0 (existed 0, fresher 0)
2016-08-22 13:23:15 GMT [INF] [11567] 100% complete, ~0s remaining
2016-08-22 13:23:15 GMT [VER] [11567] Leaving counter thread
2016-08-22 13:23:15 GMT [VER] [11549] Exiting with status code 0

$ aql
Aerospike Query Client
Version 3.9.1
Copyright 2012-2016 Aerospike. All rights reserved.
aql> show indexes backuptest
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
| ns           | bin  | indextype | set   | state | indexname | path | sync_state | type      |
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
| "backuptest" | "PK" | "NONE"    | "aaa" | "RW"  | "idx"     | "PK" | "synced"   | "NUMERIC" |
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
1 row in set (0.002 secs)
OK

But after Nth attempt to restore from backup I lost indexes:

$ asrestore -v --host 127.0.0.1 --directory backuptest -g
2016-08-22 13:25:40 GMT [INF] [11677] Starting restore to 127.0.0.1:3000 (bins: [all], sets: [all]) from backuptest
2016-08-22 13:25:40 GMT [VER] [11677] Connecting to cluster
2016-08-22 13:25:40 GMT [VER] [11677] Adding node BB9A31ECA9D7B50
2016-08-22 13:25:40 GMT [INF] [11677] Processing 1 node(s)
2016-08-22 13:25:40 GMT [VER] [11677] Creating counter thread
2016-08-22 13:25:40 GMT [VER] [11677] Listing backup files in backuptest
2016-08-22 13:25:40 GMT [VER] [11695] Entering counter thread
2016-08-22 13:25:40 GMT [INF] [11677] Found 1 backup file(s) in backuptest
2016-08-22 13:25:40 GMT [VER] [11677] Triaging 1 backup file(s)
2016-08-22 13:25:40 GMT [VER] [11677] Opening backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:25:40 GMT [VER] [11677] Getting file descriptor
2016-08-22 13:25:40 GMT [INF] [11677] Opened backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:25:40 GMT [VER] [11677] Validating backup file version
2016-08-22 13:25:40 GMT [VER] [11677] Found first backup file: backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:25:40 GMT [VER] [11677] Parsing global index and UDF data (line 4)
2016-08-22 13:25:40 GMT [VER] [11677] Parsing index in line 4
2016-08-22 13:25:40 GMT [VER] [11677] Index: idx (on set aaa)
2016-08-22 13:25:40 GMT [VER] [11677] Closing backup file
2016-08-22 13:25:40 GMT [VER] [11677] Closing file descriptor
2016-08-22 13:25:40 GMT [VER] [11677] Pushing 1 exclusive job(s) to job queue
2016-08-22 13:25:40 GMT [INF] [11677] Restoring 0 UDF file(s)
2016-08-22 13:25:40 GMT [INF] [11677] Restoring 1 secondary index(es)
2016-08-22 13:25:40 GMT [VER] [11677] Restoring index backuptest:aaa:idx (PK)
2016-08-22 13:25:40 GMT [INF] [11677] Restoring records
2016-08-22 13:25:40 GMT [VER] [11677] Creating 1 restore thread(s)
2016-08-22 13:25:40 GMT [VER] [11677] Waiting for 1 restore thread(s)
2016-08-22 13:25:40 GMT [VER] [11696] Entering restore thread
2016-08-22 13:25:40 GMT [INF] [11696] Restoring backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:25:40 GMT [VER] [11696] Opening backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:25:40 GMT [VER] [11696] Getting file descriptor
2016-08-22 13:25:40 GMT [INF] [11696] Opened backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:25:40 GMT [VER] [11696] Validating backup file version
2016-08-22 13:25:40 GMT [VER] [11696] Existence policy is default
2016-08-22 13:25:40 GMT [VER] [11696] Generation policy is default
2016-08-22 13:25:40 GMT [VER] [11696] Parsing index in line 4
2016-08-22 13:25:40 GMT [VER] [11696] Index: idx (on set aaa)
2016-08-22 13:25:40 GMT [VER] [11696] Ignoring index block
2016-08-22 13:25:40 GMT [VER] [11696] 1 per-thread record(s) (1 rec/s), read latency: 10 (0) us, store latency: 45 (0) us
2016-08-22 13:25:40 GMT [VER] [11696] Encountered end of file (line 12, col 1)
2016-08-22 13:25:40 GMT [VER] [11696] End of backup file reached
2016-08-22 13:25:40 GMT [VER] [11696] Closing backup file
2016-08-22 13:25:40 GMT [VER] [11696] Closing file descriptor
2016-08-22 13:25:40 GMT [VER] [11696] Job queue is empty
2016-08-22 13:25:40 GMT [VER] [11696] Leaving restore thread
2016-08-22 13:25:40 GMT [VER] [11677] Freeing 1 index(es)
2016-08-22 13:25:40 GMT [VER] [11677] Freeing 0 UDF file(s)
2016-08-22 13:25:40 GMT [VER] [11677] Waiting for counter thread
2016-08-22 13:25:41 GMT [INF] [11695] 0 UDF file(s), 1 secondary index(es), 1 record(s) (0 KiB/s, 1 rec/s, 168 B/rec, backed off: 0)
2016-08-22 13:25:41 GMT [INF] [11695] Expired 0 : skipped 0 : inserted 1 : failed 0 (existed 0, fresher 0)
2016-08-22 13:25:41 GMT [INF] [11695] 100% complete, ~0s remaining
2016-08-22 13:25:41 GMT [VER] [11695] Leaving counter thread
2016-08-22 13:25:41 GMT [VER] [11677] Exiting with status code 0

$ aql
Aerospike Query Client
Version 3.9.1
Copyright 2012-2016 Aerospike. All rights reserved.
aql> show indexes backuptest
0 rows in set (0.001 secs)
OK

When I ran asrestore one more time, indexes are restored:

$ asrestore -v --host 127.0.0.1 --directory backuptest -g
2016-08-22 13:27:08 GMT [INF] [11740] Starting restore to 127.0.0.1:3000 (bins: [all], sets: [all]) from backuptest
2016-08-22 13:27:08 GMT [VER] [11740] Connecting to cluster
2016-08-22 13:27:08 GMT [VER] [11740] Adding node BB9A31ECA9D7B50
2016-08-22 13:27:08 GMT [INF] [11740] Processing 1 node(s)
2016-08-22 13:27:08 GMT [VER] [11740] Creating counter thread
2016-08-22 13:27:08 GMT [VER] [11740] Listing backup files in backuptest
2016-08-22 13:27:08 GMT [VER] [11758] Entering counter thread
2016-08-22 13:27:08 GMT [INF] [11740] Found 1 backup file(s) in backuptest
2016-08-22 13:27:08 GMT [VER] [11740] Triaging 1 backup file(s)
2016-08-22 13:27:08 GMT [VER] [11740] Opening backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:27:08 GMT [VER] [11740] Getting file descriptor
2016-08-22 13:27:08 GMT [INF] [11740] Opened backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:27:08 GMT [VER] [11740] Validating backup file version
2016-08-22 13:27:08 GMT [VER] [11740] Found first backup file: backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:27:08 GMT [VER] [11740] Parsing global index and UDF data (line 4)
2016-08-22 13:27:08 GMT [VER] [11740] Parsing index in line 4
2016-08-22 13:27:08 GMT [VER] [11740] Index: idx (on set aaa)
2016-08-22 13:27:08 GMT [VER] [11740] Closing backup file
2016-08-22 13:27:08 GMT [VER] [11740] Closing file descriptor
2016-08-22 13:27:08 GMT [VER] [11740] Pushing 1 exclusive job(s) to job queue
2016-08-22 13:27:08 GMT [INF] [11740] Restoring 0 UDF file(s)
2016-08-22 13:27:08 GMT [INF] [11740] Restoring 1 secondary index(es)
2016-08-22 13:27:08 GMT [VER] [11740] Restoring index backuptest:aaa:idx (PK)
2016-08-22 13:27:08 GMT [INF] [11740] Restoring records
2016-08-22 13:27:08 GMT [VER] [11740] Creating 1 restore thread(s)
2016-08-22 13:27:08 GMT [VER] [11740] Waiting for 1 restore thread(s)
2016-08-22 13:27:08 GMT [VER] [11759] Entering restore thread
2016-08-22 13:27:08 GMT [INF] [11759] Restoring backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:27:08 GMT [VER] [11759] Opening backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:27:08 GMT [VER] [11759] Getting file descriptor
2016-08-22 13:27:08 GMT [INF] [11759] Opened backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:27:08 GMT [VER] [11759] Validating backup file version
2016-08-22 13:27:08 GMT [VER] [11759] Existence policy is default
2016-08-22 13:27:08 GMT [VER] [11759] Generation policy is default
2016-08-22 13:27:08 GMT [VER] [11759] Parsing index in line 4
2016-08-22 13:27:08 GMT [VER] [11759] Index: idx (on set aaa)
2016-08-22 13:27:08 GMT [VER] [11759] Ignoring index block
2016-08-22 13:27:08 GMT [VER] [11759] 1 per-thread record(s) (1 rec/s), read latency: 14 (0) us, store latency: 79 (0) us
2016-08-22 13:27:08 GMT [VER] [11759] Encountered end of file (line 12, col 1)
2016-08-22 13:27:08 GMT [VER] [11759] End of backup file reached
2016-08-22 13:27:08 GMT [VER] [11759] Closing backup file
2016-08-22 13:27:08 GMT [VER] [11759] Closing file descriptor
2016-08-22 13:27:08 GMT [VER] [11759] Job queue is empty
2016-08-22 13:27:08 GMT [VER] [11759] Leaving restore thread
2016-08-22 13:27:08 GMT [VER] [11740] Freeing 1 index(es)
2016-08-22 13:27:08 GMT [VER] [11740] Freeing 0 UDF file(s)
2016-08-22 13:27:08 GMT [VER] [11740] Waiting for counter thread
2016-08-22 13:27:09 GMT [INF] [11758] 0 UDF file(s), 1 secondary index(es), 1 record(s) (0 KiB/s, 1 rec/s, 168 B/rec, backed off: 0)
2016-08-22 13:27:09 GMT [INF] [11758] Expired 0 : skipped 0 : inserted 1 : failed 0 (existed 0, fresher 0)
2016-08-22 13:27:09 GMT [INF] [11758] 100% complete, ~0s remaining
2016-08-22 13:27:09 GMT [VER] [11758] Leaving counter thread
2016-08-22 13:27:09 GMT [VER] [11740] Exiting with status code 0

$ aql
Aerospike Query Client
Version 3.9.1
Copyright 2012-2016 Aerospike. All rights reserved.
aql> show indexes backuptest
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
| ns           | bin  | indextype | set   | state | indexname | path | sync_state | type      |
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
| "backuptest" | "PK" | "NONE"    | "aaa" | "RW"  | "idx"     | "PK" | "synced"   | "NUMERIC" |
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
1 row in set (0.002 secs)
OK

Then I did it several more times and lost indexes again:

$ asrestore -v --host 127.0.0.1 --directory backuptest -g
2016-08-22 13:29:47 GMT [INF] [12426] Starting restore to 127.0.0.1:3000 (bins: [all], sets: [all]) from backuptest
2016-08-22 13:29:47 GMT [VER] [12426] Connecting to cluster
2016-08-22 13:29:47 GMT [VER] [12426] Adding node BB9A31ECA9D7B50
2016-08-22 13:29:47 GMT [INF] [12426] Processing 1 node(s)
2016-08-22 13:29:47 GMT [VER] [12426] Creating counter thread
2016-08-22 13:29:47 GMT [VER] [12426] Listing backup files in backuptest
2016-08-22 13:29:47 GMT [INF] [12426] Found 1 backup file(s) in backuptest
2016-08-22 13:29:47 GMT [VER] [12426] Triaging 1 backup file(s)
2016-08-22 13:29:47 GMT [VER] [12426] Opening backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:29:47 GMT [VER] [12426] Getting file descriptor
2016-08-22 13:29:47 GMT [INF] [12426] Opened backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:29:47 GMT [VER] [12426] Validating backup file version
2016-08-22 13:29:47 GMT [VER] [12426] Found first backup file: backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:29:47 GMT [VER] [12426] Parsing global index and UDF data (line 4)
2016-08-22 13:29:47 GMT [VER] [12426] Parsing index in line 4
2016-08-22 13:29:47 GMT [VER] [12426] Index: idx (on set aaa)
2016-08-22 13:29:47 GMT [VER] [12426] Closing backup file
2016-08-22 13:29:47 GMT [VER] [12426] Closing file descriptor
2016-08-22 13:29:47 GMT [VER] [12426] Pushing 1 exclusive job(s) to job queue
2016-08-22 13:29:47 GMT [INF] [12426] Restoring 0 UDF file(s)
2016-08-22 13:29:47 GMT [INF] [12426] Restoring 1 secondary index(es)
2016-08-22 13:29:47 GMT [VER] [12426] Restoring index backuptest:aaa:idx (PK)
2016-08-22 13:29:47 GMT [INF] [12426] Restoring records
2016-08-22 13:29:47 GMT [VER] [12426] Creating 1 restore thread(s)
2016-08-22 13:29:47 GMT [VER] [12426] Waiting for 1 restore thread(s)
2016-08-22 13:29:47 GMT [VER] [12445] Entering restore thread
2016-08-22 13:29:47 GMT [INF] [12445] Restoring backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:29:47 GMT [VER] [12445] Opening backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:29:47 GMT [VER] [12445] Getting file descriptor
2016-08-22 13:29:47 GMT [INF] [12445] Opened backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:29:47 GMT [VER] [12445] Validating backup file version
2016-08-22 13:29:47 GMT [VER] [12445] Existence policy is default
2016-08-22 13:29:47 GMT [VER] [12445] Generation policy is default
2016-08-22 13:29:47 GMT [VER] [12445] Parsing index in line 4
2016-08-22 13:29:47 GMT [VER] [12445] Index: idx (on set aaa)
2016-08-22 13:29:47 GMT [VER] [12445] Ignoring index block
2016-08-22 13:29:47 GMT [VER] [12444] Entering counter thread
2016-08-22 13:29:47 GMT [VER] [12445] 1 per-thread record(s) (1 rec/s), read latency: 14 (0) us, store latency: 104 (1) us
2016-08-22 13:29:47 GMT [VER] [12445] Encountered end of file (line 12, col 1)
2016-08-22 13:29:47 GMT [VER] [12445] End of backup file reached
2016-08-22 13:29:47 GMT [VER] [12445] Closing backup file
2016-08-22 13:29:47 GMT [VER] [12445] Closing file descriptor
2016-08-22 13:29:47 GMT [VER] [12445] Job queue is empty
2016-08-22 13:29:47 GMT [VER] [12445] Leaving restore thread
2016-08-22 13:29:47 GMT [VER] [12426] Freeing 1 index(es)
2016-08-22 13:29:47 GMT [VER] [12426] Freeing 0 UDF file(s)
2016-08-22 13:29:47 GMT [VER] [12426] Waiting for counter thread
2016-08-22 13:29:48 GMT [INF] [12444] 0 UDF file(s), 1 secondary index(es), 1 record(s) (0 KiB/s, 0 rec/s, 0 B/rec, backed off: 0)
2016-08-22 13:29:48 GMT [INF] [12444] Expired 0 : skipped 0 : inserted 1 : failed 0 (existed 0, fresher 0)
2016-08-22 13:29:48 GMT [VER] [12444] Leaving counter thread
2016-08-22 13:29:48 GMT [VER] [12426] Exiting with status code 0

$ aql
Aerospike Query Client
Version 3.9.1
Copyright 2012-2016 Aerospike. All rights reserved.
aql> show indexes backuptest
0 rows in set (0.001 secs)
OK

You can reproduce problem easily if you try to run asrestore 10 or more times.

reopen issue #2

I would like to reopen this issue: #2
thanks

can we please add flag to compile without s3 support?

same goes for restore

it just adds complexity and build time

and we don't need it at all

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.