alibaba / redisshake Goto Github PK

View Code? Open in Web Editor NEW

3.6K 72.0 647.0 24.94 MB

RedisShake is a Redis data processing and migration tool.

Home Page: https://tair-opensource.github.io/RedisShake/

License: MIT License

Shell 0.37% Go 83.31% Python 16.33%

redis redis-cluster redis-proxy

redisshake's Introduction

RedisShake 4.x: Redis Data Processing & Migration Tool

Overview

RedisShake is a tool designed for processing and migrating Redis data. It offers the following features:

Redis Compatibility: RedisShake is compatible with Redis versions ranging from 2.8 to 7.2, and supports various deployment methods including standalone, master-slave, sentinel, and cluster.
Cloud Service Compatibility: RedisShake works seamlessly with popular Redis-like databases provided by leading cloud service providers, including but not limited to:
Module Compatibility: RedisShake is compatible with TairString, TairZSet, and TairHash modules.
Multiple Export Modes: RedisShake supports PSync, RDB, and Scan export modes.
Data Processing: RedisShake enables data filtering and transformation through custom scripts.

Getting Started

Installation

Download the Binary Package

Download the binary package directly from the Releases page.

Compile from Source

To compile from source, ensure that you have a Golang environment set up on your local machine:

git clone https://github.com/alibaba/RedisShake
cd RedisShake
sh build.sh

Usage

Assume you have two Redis instances:

Instance A: 127.0.0.1:6379
Instance B: 127.0.0.1:6380

Create a new configuration file shake.toml:

[sync_reader]
address = "127.0.0.1:6379"

[redis_writer]
address = "127.0.0.1:6380"

To start RedisShake, run the following command:

./redis-shake shake.toml

For more detailed information, please refer to the documentation:

Contributing

We welcome contributions from the community. For significant changes, please open an issue first to discuss what you would like to change. We are particularly interested in:

Adding support for more modules
Enhancing support for Readers and Writers
Sharing your Lua scripts and best practices

History

RedisShake is a project actively maintained by the Tair team at Alibaba Cloud. Its evolution can be traced back to its initial version, which was forked from redis-port.

During its evolution:

The RedisShake 2.x version brought a series of improvements and updates, enhancing its overall stability and performance.
The RedisShake 3.x version represented a significant milestone where the entire codebase was completely rewritten and optimized, leading to better efficiency and usability.
The current version, RedisShake 4.x, has further enhanced features related to readers, configuration, observability, and functions.

License

RedisShake is open-sourced under the MIT license.

redisshake's People

Contributors

Stargazers

Watchers

Forkers

stutiredboy jackerzhou hulaoan tabdba minghuan2046 yuguangjie zhanglei 307545758 qu823110460 sky-9527-github xiaoshangwu nxsre qsdj isgasho it-xiaoge sdgdsffdsfff gavinljj chyuch-github lcl101 bdjimmy suhuaguo liaokunlin shaobai wangyiyang slimina cosmosyy haizi1996 zbzbzb022 tangyiyong zhaoshiling1017 harriks huangzhiyong keepwalking86 piano-wow dengchao123456 fsyyft-fork xukuitao buyang0911 zhishanlingyun yaoqi runbird ceshihao doovemax zwunix xingxing17 smudkey anlem12 adofsauron loeyae ksharpdabu chakra-coder radtek davidmr001 cocoakekeyu sunshuai2013 gdzy1987 weizai118 callmedba dailai rootzc hoangpq ssxg roarain loojk2008 yochji520 iftechio afghanistanyn linyue515 myocm gridl yangmain bemike789 ted1994g opsfarmer thinking-one-hour-everyday liangchaoboy lifuqiang521 staysun winsomexjs wangnuowa booboowei eleooo bg1011 jiaoyongqing veblenzhang zhangyuexu tom2jack shook2012 wowdba xulibo 8899man csongsy mytxd pingfan108 breakcount pengzihe spade7 cyfanhu terry1504 billy0419

redisshake's Issues

TLS support

Some cloud redis versions support TLS, e.g. ElastiCache

It looks like new version redigo supports TLS.
https://github.com/gomodule/redigo

codis 3.2版本获取版本号报错

codis 3.2版本执行info server命令有问题，codis本身支持info命令，但不支持info server命令，后续可以进行支持。
如果有相同报错的用户请帖一下具体的报错内容。

rump模式，目的端集群版写入1w个key block了

block住了。
目前不能超过4096，这个是开源cluster的队列设置限制。

开启psync，同步报错，[PANIC] invalid psync response, fullsync

2019/04/04 15:55:38 [INFO] redis-shake configuration: {"Id":"redis-shake","LogFile":"","SystemProfile":9310,"HttpProfile":9320,"NCpu":0,"Parallel":4,"InputRdb":"local","OutputRdb":"local_dump","SourceAddress":"192.168.111.93:6379","SourcePasswordRaw":"","SourcePasswordEncoding":"","SourceVersion":7,"SourceAuthType":"auth","TargetAddress":"192.168.111.94:6379","TargetPasswordRaw":"","TargetPasswordEncoding":"","TargetVersion":7,"TargetDB":-1,"TargetAuthType":"auth","FakeTime":"","Rewrite":true,"FilterDB":"","FilterKey":[],"FilterSlot":[],"BigKeyThreshold":524288000,"Psync":true,"Metric":true,"MetricPrintLog":false,"HeartbeatUrl":"","HeartbeatInterval":3,"HeartbeatExternal":"test external","HeartbeatNetworkInterface":"","SenderSize":104857600,"SenderCount":5000,"SenderDelayChannelSize":65535,"KeepAlive":0,"PidPath":"","RedisConnectTTL":0,"ReplaceHashTag":false,"ExtraInfo":false,"SockFileName":"","SockFileSize":0,"HeartbeatIp":"127.0.0.1","ShiftTime":0,"TargetRedisVersion":"4.0.12","TargetReplace":true}
2019/04/04 15:55:38 [INFO] sync from '192.168.111.93:6379' to '192.168.111.94:6379' http '9320'
2019/04/04 15:55:38 [INFO] sync from '192.168.111.93:6379' to '192.168.111.94:6379'
2019/04/04 15:55:38 [PANIC] invalid psync response, fullsync
[error]: bad resp CRLF end
    6   /Users/wangyiyang/Documents/GitHub/RedisShake/src/pkg/redis/decoder.go:179
            pkg/redis.(*Decoder).decodeSingleLineBulkBytesArray
    5   /Users/wangyiyang/Documents/GitHub/RedisShake/src/pkg/redis/decoder.go:97
            pkg/redis.(*Decoder).decodeResp
    4   /Users/wangyiyang/Documents/GitHub/RedisShake/src/pkg/redis/decoder.go:32
            pkg/redis.Decode
    3   /Users/wangyiyang/Documents/GitHub/RedisShake/src/redis-shake/common/utils.go:162
            redis-shake/common.SendPSyncFullsync
    2   /Users/wangyiyang/Documents/GitHub/RedisShake/src/redis-shake/sync.go:180
            redis-shake.(*CmdSync).SendPSyncCmd
    1   /Users/wangyiyang/Documents/GitHub/RedisShake/src/redis-shake/sync.go:119
            redis-shake.(*CmdSync).Main
    0   /Users/wangyiyang/Documents/GitHub/RedisShake/src/redis-shake/main/main.go:115
            main.main
        ... ...
[stack]:
    3   /Users/wangyiyang/Documents/GitHub/RedisShake/src/redis-shake/common/utils.go:164
            redis-shake/common.SendPSyncFullsync
    2   /Users/wangyiyang/Documents/GitHub/RedisShake/src/redis-shake/sync.go:180
            redis-shake.(*CmdSync).SendPSyncCmd
    1   /Users/wangyiyang/Documents/GitHub/RedisShake/src/redis-shake/sync.go:119
            redis-shake.(*CmdSync).Main
    0   /Users/wangyiyang/Documents/GitHub/RedisShake/src/redis-shake/main/main.go:115
            main.main
        ... ...

panic: CROSSSLOT Keys in request don't hash to the same slot

版本：bugfix-1.2.1,f04283013e034acf33bee92ebf836aeb8757f5b2,go1.10.3,2019-03-27_20:11:26
同步源：redis 3.2.8 （codis集群中的slave实例）
同步目标：redis 4.0.10（cluster）
配置：
parallel = 1024
source.version = 7
target.version = 7
big_key_threshold = 52428800

rdb同步完成后，增量同步开始时突然panic：
2019/04/16 17:21:16 [INFO] restore big zset key pgg_user_cash_account_pggsign score 9781932760 member 14681071
2019/04/16 17:21:16 [INFO] restore big zset key pgg_user_cash_account_pggsign score 14625880623 member 133930417
2019/04/16 17:21:16 [INFO] restore big zset key pgg_user_cash_account_pggsign score 16749642548 member 4970085
2019/04/16 17:21:16 [INFO] total=1063796856 - 1063796856 [100%] entry=1160052
2019/04/16 17:21:16 [INFO] sync rdb done
2019/04/16 17:21:16 [INFO] Event:IncrSyncStart Id:redis_shake_3
2019/04/16 17:21:16 [PANIC] Event:ErrorReply Id:redis_shake_3 Command: [unknown] Error: CROSSSLOT Keys in request don't hash to the same slot
[stack]:
0 /home/zhuzhao.cx/redis-shake/src/redis-shake/sync.go:408
redis-shake.(*CmdSync).SyncCommand.func2
... ...

迁移至集群版报错[error]: MOVED

目的端写入集群版出错

rump restore quick list failed

The error reason is quick list data type isn't be handled in restoreBigRdbEntry function called by spit.go/RestoreBigkey.

集群版同步出core

报错：

问题定位：
Put的时候对于事务处理node没有初始化：

此处return的是nil，塞入batch队列没有校验：

导致在RunBatch的时候出core了：

总结：
redis-go-cluster驱动的bug。

redis-shake同步redis5.03报unknown object-type 0f

redis source:5.0.3
redis target:5.0.3
数据类型：zset、string(bitmap)、list、stream、set

"client-output-buffer-limit"
"normal 0 0 0 slave 268435456 67108864 60 pubsub 33554432 8388608 60"

启动redis-shake同步时报错：

2019/03/11 06:07:22 [INFO] redis-shake configuration: {"Id":"redis-shake","LogFile":"","SystemProfile":9310,"HttpProfile":9320,"NCpu":0,"Parallel":4,"InputRdb":"local_dump","OutputRdb":"local_dump","SourceAddress":"xxxxxx:6666","SourcePasswordRaw":"","SourcePasswordEncoding":"","SourceVersion":7,"SourceAuthType":"","TargetAddress":"xxxxxx:6667","TargetPasswordRaw":"","TargetPasswordEncoding":"","TargetVersion":7,"TargetDB":-1,"TargetAuthType":"","FakeTime":"","Rewrite":true,"FilterDB":"","FilterKey":[],"FilterSlot":[],"BigKeyThreshold":524288000,"Psync":false,"Metric":true,"MetricPrintLog":true,"HeartbeatUrl":"","HeartbeatInterval":3,"HeartbeatExternal":"test external","HeartbeatNetworkInterface":"","SenderSize":104857600,"SenderCount":5000,"SenderDelayChannelSize":65535,"ReplaceHashTag":false,"ExtraInfo":false,"SockFileName":"","SockFileSize":0,"HeartbeatIp":"127.0.0.1","ShiftTime":0,"TargetRedisVersion":"5.0.3","TargetReplace":false}
2019/03/11 06:07:22 [INFO] sync from 'xxxxxx:6666' to 'xxxxxx:6667' http '9320'
2019/03/11 06:07:22 [INFO] sync from 'xxxxxx:6666' to 'xxxxxx:6667'
2019/03/11 06:07:22 [INFO] rdb file = 1134
2019/03/11 06:07:22 [INFO] Aux information key:redis-ver value:5.0.3
2019/03/11 06:07:22 [INFO] Aux information key:redis-bits value:64
2019/03/11 06:07:22 [INFO] Aux information key:ctime value:1552298842
2019/03/11 06:07:22 [INFO] Aux information key:used-mem value:1906512
2019/03/11 06:07:22 [INFO] Aux information key:repl-stream-db value:0
2019/03/11 06:07:22 [INFO] Aux information key:repl-id value:7ca1a84deb8c4b7946355938c834fddd40098e61
2019/03/11 06:07:22 [INFO] Aux information key:repl-offset value:6917447
2019/03/11 06:07:22 [INFO] Aux information key:aof-preamble value:0
2019/03/11 06:07:22 [INFO] db_size:26 expire_size:1
2019/03/11 06:07:22 [PANIC] parse rdb entry error, if the err is :EOF, please check that if the src db log has client outout buffer oom, if so set output buffer larger.
[error]: unknown object-type 0f
2 /home/soft/redis-shake/src/pkg/rdb/reader.go:98
pkg/rdb.(*rdbReader).readObjectValue
1 /home/soft/redis-shake/src/pkg/rdb/loader.go:169
pkg/rdb.(*Loader).NextBinEntry
0 /home/soft/redis-shake/src/redis-shake/common/utils.go:763
redis-shake/common.NewRDBLoader.func1
... ...
[stack]:
0 /home/soft/redis-shake/src/redis-shake/common/utils.go:764
redis-shake/common.NewRDBLoader.func1
... ...

源端为集群模式，具体不能开启move slot操作的原因是什么？

某些情况下，同步过程中丢失数据

RedisShake版本：release-v1.2.2-20190403
源redis版本：3.2.8
目的redis版本：4.0.10 cluster
配置：仅增大了 parallel 参数，其余参数均为默认参数
现象：同步过程中，无错误提示，同步完成之后，保持同步进程继续运行。
统计两端key总数，发现目的redis的key数量约为源redis的90%。
使用redis-full-check默认参数进行检查，报错：tair-opensource/RedisFullCheck#36
使用 redis-full-check -m 3 进行检查，发现两端一致，即工具认为key是一致的。

从调试信息中找到了两端不一致的key：uc_basic_data_2852157937_1_28521579370000000000000000000010

源redis：
> type uc_basic_data_2852157937_1_28521579370000000000000000000010
hash
> ttl uc_basic_data_2852157937_1_28521579370000000000000000000010
(integer) 596453

目的redis：
> type uc_basic_data_2852157937_1_28521579370000000000000000000010
none
> ttl uc_basic_data_2852157937_1_28521579370000000000000000000010
(integer) -2

Support Filter by Key in Rump Mode

Current filter by key only supports restore and sync.
It is appreciated if it is supported in rump mode.

https://github.com/alibaba/RedisShake/blob/3fc9515c263db30204d36b22d26d1749c1e46bb1/conf/redis-shake.conf#L111-L116

redis-shake connects slave and sync `evalsha` command error。从slave上拉取，evalsha命令失败

Please check out this question in FAQ in English.
用户反馈问题，4.0.2版本，目的端为标准redis，lua脚本同步失败，从master拉取没问题，slave拉取存在问题。

问题定位中。
相关知识：4.0.4版本开始，redis在全量rdb文件中携带lua脚本信息，4.0.2版本不存在。但是问题不是这个，因为redis目前同步的机制是如果连接建立的一次，那么evalsha命令会转换为eval命令携带lua script一并发送，后续evalsha命令不进行转换，这样就可以找到对应的script。如果用户使用了redis.replicate_commands()，那么也不会报错，因为同步的是事务命令（multi+xxxx+exec），而不是lua脚本

同步数据报错

版本：develop,4a26b1ca10bc9c6849b30fc73004d135c8227063,go1.10.3,2019-03-08_13:43:41
redis版本：
源：redis_version:3.2.4 目标：5.0.2
报错：
2019/04/23 19:49:49 [PANIC] Event:NetErrorWhileReceive Id:redis-shake Error:EOF
[stack]:
0 /home/zhuzhao.cx/redis-shake/src/redis-shake/sync.go:405
redis-shake.(*CmdSync).SyncCommand.func2

Prometheus Metrics Support

Current /metric API response is like json format. e.g.

[{"StartTime":"2019-05-28T07:22:43Z","PullCmdCount":0,"PullCmdCountTotal":69,
"BypassCmdCount":0,"BypassCmdCountTotal":0,"PushCmdCount":0,
"PushCmdCountTotal":69,"SuccessCmdCount":1,"SuccessCmdCountTotal":69,
"FailCmdCount":0,"FailCmdCountTotal":0,"Delay":"268.00 ms","AvgDelay":"284.99 ms",
"NetworkSpeed":0,"NetworkFlowTotal":6607,"FullSyncProgress":100,"Status":"incr",
"SenderBufCount":0,"ProcessingCmdCount":0,"TargetDBOffset":136104580201,
"SourceDBOffset":136104587992,"SourceAddress":"redis_src_address:6379",
"TargetAddress":["redis_target_address:6379"]}]

It is appreciated if it supports Prometheus text-based format.
https://prometheus.io/docs/instrumenting/exposition_formats/

It is better to add a /metrics API in order to not break any existing function.

Support Filter by Key Blacklist

A blacklist key filter can work if current whitelist is not set.

从standalone到cluster同步时，dbSyncer卡在某个进度一直不动

描述

在使用redis-shake进行从standalone到cluster同步时，dbSyncer卡在某个进度一直不动。

redis-shake配置文件参数

id = redis-shake
log.file = ./log/redis-shake.log
log.level = all
pid_path = ./
system_profile = 9310
http_profile = 9320
ncpu = 0
parallel = 32
source.type = standalone
source.address = 
source.password_raw =
source.auth_type = auth
source.tls_enable = false
target.type = cluster
target.address = 
target.password_raw =
target.auth_type = auth
target.db = -1
target.tls_enable = false
rdb.input = local
rdb.output = local_dump
rdb.parallel = 0
rdb.special_cloud = 
fake_time =
rewrite = true
filter.db =
filter.key =
filter.slot =
filter.lua = false
big_key_threshold = 524288000
psync = false
metric = true
metric.print_log = false
heartbeat.url =
heartbeat.interval = 3
heartbeat.external = test external
heartbeat.network_interface =
sender.size = 104857600
sender.count = 5000
sender.delay_channel_size = 65535
keep_alive = 0
scan.key_number = 50
scan.special_cloud =
scan.key_file =
qps = 200000
replace_hash_tag = false
extra = false`

redis-shake 支持双向同步吗？

在redis 3里，配置了双向同步，只有一个能启动成功。问一下shake支持双向同步吗？

希望支持指定db到cluster的迁移

例如源端是：单实例的db2,添加filter.db = 2参数后，
能同步到目的端：cluster的 db0。

断点续传功能支持

考虑支持断点续传功能，由于4.0及以上版本推出的psync2，支持主备切换时候维持master的replid信息，同步replid(4.0以下称runid)+offset信息可以实现断点续传的目的。
当redis-shake断开连接，重新连接后，可以根据这2个信息进行重新拉取，主备切换也能支持。
psync2参考redis主从同步，replication.c/masterTryPartialResynchronization函数。
redis-shake层面本身进行拉取持久化offset位点操作，但目前目的端写入是异步状态，以及源端offset如何获取。这样重启后可以恢复位点，psync进行拉取。
目前该需求的开发支持在讨论中。

RdisShake支持5.0迁移到4.0吗?

metric接口的 Pull和Push的数量对不上

配置了过滤前缀key同步，通过metric接口查看，发现Pull和Push的数量对不上

希望restore rdb done 之后，能自动退出，而不是wait forever。

redis-shake restore 的时候，restore : rdb done了，之后为什么一直wait forever。
感觉done 之后，程序就可以结束了。而现在好像都不会自己结束，而是要自己手动kill。
话说有什么办法，让restore rdb done之后，自动退出的吗？

redisshake能否增加一个守护进程？

我们昨天测试的时候发现，晚上21点左右redisshake的进程断了，日志中也无报错，能否提一个技术上的需求，就是增加一个守护进程，监控redisshake的进程状态，如果进程没有了，自动把redisshake起来，类似mysqld_safe的原理。
日志如下：
2019/04/28 21:10:43 [INFO] {"PullCmdCount":3483,"PullCmdCountTotal":13225521,"BypassCmdCount":0,"BypassCmdCountTotal":0,"PushCmdCount":3483,"
PushCmdCountTotal":13225521,"SuccessCmdCount":3489,"SuccessCmdCountTotal":13225520,"FailCmdCount":0,"FailCmdCountTotal":0,"Delay":"0.09 ms","
AvgDelay":"0.80 ms","NetworkSpeed":521067,"NetworkFlowTotal":2062757702,"FullSyncProgress":100,"Status":"incr","SenderBufCount":0,"Processing
CmdCount":1,"TargetDBOffset":0,"SourceDBOffset":0}
2019/04/28 21:10:43 [INFO] sync: +forward=3339 +nbypass=0 +nbytes=0
2019/04/28 21:10:44 [INFO] sync: +forward=3875 +nbypass=0 +nbytes=0
2019/04/28 21:10:45 [INFO] sync: +forward=3463 +nbypass=0 +nbytes=0
2019/04/28 21:10:46 [INFO] sync: +forward=3656 +nbypass=0 +nbytes=0
2019/04/29 10:15:11 [WARN]

\ \ _ ______ |
\ \ / _-=O'/|O'/|
\ redis-shake, here we go !! _\ / | / )
/ / '/-== _/|/=-| -GM
/ / * \ | |
/ / (o)

restore command error key:xxxxxxxxxx err:ECONNTIMEOUT

使用redis-shake迁移集群时报如下错误：

2019/07/02 16:13:57 [PANIC] restore command error key:xxxxxxxxxx  err:ECONNTIMEOUT
[error]: ECONNTIMEOUT
[stack]: 
    1   /home/zhuzhao.cx/redis-shake/src/redis-shake/common/utils.go:805
            redis-shake/common.RestoreRdbEntry
    0   /home/zhuzhao.cx/redis-shake/src/redis-shake/sync.go:441
            redis-shake.(*dbSyncer).syncRDBFile.func1.1
        ... ...

redis版本：4.0.2

[PANIC] decode redis resp failed

环境：
redis 主从使用 https://github.com/AliyunContainerService/redis-cluster 搭建（修改redis版本：4.0.12）

# Replication
role:master # 172.17.0.2
connected_slaves:2
slave0:ip=172.17.0.3,port=6379,state=online,offset=305992,lag=0
slave1:ip=172.17.0.4,port=6379,state=online,offset=305857,lag=1

RedisShake 源使用从节点，目的地是另外一个docker起的redis单节点
执行redis-cluster内的test.sh(目测是重启master操作)，RedisShake报错停止运行

./bin/redis-shake.linux64 -type=sync -conf=conf/redis-shake.conf
2019/03/27 09:37:46 [WARN]
______________________________
\                             \           _         ______ |
 \                             \        /   \___-=O'/|O'/__|
  \  redis-shake, here we go !! \_______\          / | /    )
  /                             /        '/-==__ _/__|/__=-|  -GM
 /                             /         *             \ | |
/                             /                        (o)
------------------------------
2019/03/27 09:37:46 [INFO] redis-shake configuration: {"Id":"redis-shake","LogFile":"","SystemProfile":9310,"HttpProfile":9320,"NCpu":0,"Parallel":4,"InputRdb":"local","OutputRdb":"local_dump","SourceAddress":"172.17.0.3:6379","SourcePasswordRaw":"","SourcePasswordEncoding":"","SourceVersion":7,"SourceAuthType":"auth","TargetAddress":"172.17.0.8:6379","TargetPasswordRaw":"","TargetPasswordEncoding":"","TargetVersion":7,"TargetDB":-1,"TargetAuthType":"auth","FakeTime":"","Rewrite":true,"FilterDB":"","FilterKey":[],"FilterSlot":[],"BigKeyThreshold":524288000,"Psync":false,"Metric":true,"MetricPrintLog":false,"HeartbeatUrl":"","HeartbeatInterval":3,"HeartbeatExternal":"test external","HeartbeatNetworkInterface":"","SenderSize":104857600,"SenderCount":5000,"SenderDelayChannelSize":65535,"KeepAlive":0,"ReplaceHashTag":false,"ExtraInfo":false,"SockFileName":"","SockFileSize":0,"HeartbeatIp":"127.0.0.1","ShiftTime":0,"TargetRedisVersion":"4.0.12","TargetReplace":true}
2019/03/27 09:37:46 [INFO] sync from '172.17.0.3:6379' to '172.17.0.8:6379' http '9320'
2019/03/27 09:37:46 [INFO] sync from '172.17.0.3:6379' to '172.17.0.8:6379'
2019/03/27 09:37:46 [INFO] rdb file = 179
2019/03/27 09:37:46 [INFO] Aux information key:redis-ver value:4.0.12
2019/03/27 09:37:46 [INFO] Aux information key:redis-bits value:64
2019/03/27 09:37:46 [INFO] Aux information key:ctime value:1553650666
2019/03/27 09:37:46 [INFO] Aux information key:used-mem value:2124672
2019/03/27 09:37:46 [INFO] Aux information key:repl-stream-db value:0
2019/03/27 09:37:46 [INFO] Aux information key:repl-id value:7a3a30b4e88f89ff56b6606d98c2f4c5d5c1ce5d
2019/03/27 09:37:46 [INFO] Aux information key:repl-offset value:314013
2019/03/27 09:37:46 [INFO] Aux information key:aof-preamble value:0
2019/03/27 09:37:46 [INFO] total=179 -          179 [100%]  entry=0
2019/03/27 09:37:46 [INFO] sync rdb done
2019/03/27 09:37:46 [INFO] Event:IncrSyncStart	Id:redis-shake
2019/03/27 09:37:47 [INFO] sync:  +forward=1      +nbypass=0      +nbytes=0
2019/03/27 09:37:48 [INFO] sync:  +forward=2      +nbypass=0      +nbytes=0
2019/03/27 09:37:49 [INFO] sync:  +forward=1      +nbypass=0      +nbytes=0
2019/03/27 09:37:50 [INFO] sync:  +forward=2      +nbypass=0      +nbytes=0
2019/03/27 09:37:51 [INFO] sync:  +forward=1      +nbypass=0      +nbytes=0
2019/03/27 09:37:52 [INFO] sync:  +forward=2      +nbypass=0      +nbytes=0
2019/03/27 09:37:53 [INFO] sync:  +forward=2      +nbypass=0      +nbytes=0
2019/03/27 09:37:54 [INFO] sync:  +forward=2      +nbypass=0      +nbytes=0
2019/03/27 09:37:55 [INFO] sync:  +forward=1      +nbypass=0      +nbytes=0
2019/03/27 09:37:56 [WARN] Event:GetFakeSlaveOffsetFail	Id:redis-shake	Warn:OffsetNotFoundInInfo
2019/03/27 09:37:56 [INFO] sync:  +forward=2      +nbypass=0      +nbytes=0
2019/03/27 09:37:57 [INFO] sync:  +forward=1      +nbypass=0      +nbytes=0
2019/03/27 09:37:58 [INFO] sync:  +forward=2      +nbypass=0      +nbytes=0
2019/03/27 09:37:59 [INFO] sync:  +forward=1      +nbypass=0      +nbytes=0
2019/03/27 09:38:00 [INFO] sync:  +forward=0      +nbypass=0      +nbytes=0
2019/03/27 09:38:01 [INFO] sync:  +forward=0      +nbypass=0      +nbytes=0
2019/03/27 09:38:02 [INFO] sync:  +forward=0      +nbypass=0      +nbytes=0
2019/03/27 09:38:03 [INFO] sync:  +forward=0      +nbypass=0      +nbytes=0
2019/03/27 09:38:04 [INFO] sync:  +forward=0      +nbypass=0      +nbytes=0
2019/03/27 09:38:05 [INFO] sync:  +forward=0      +nbypass=0      +nbytes=0
2019/03/27 09:38:06 [PANIC] decode redis resp failed
[error]: EOF
    3   /home/zhuzhao.cx/redis-shake/src/pkg/redis/decoder.go:103
            pkg/redis.(*Decoder).decodeType
    2   /home/zhuzhao.cx/redis-shake/src/pkg/redis/decoder.go:65
            pkg/redis.(*Decoder).decodeResp
    1   /home/zhuzhao.cx/redis-shake/src/pkg/redis/decoder.go:36
            pkg/redis.MustDecodeOpt
    0   /home/zhuzhao.cx/redis-shake/src/redis-shake/sync.go:452
            redis-shake.(*CmdSync).SyncCommand.func3
        ... ...
[stack]:
    1   /home/zhuzhao.cx/redis-shake/src/pkg/redis/decoder.go:38
            pkg/redis.MustDecodeOpt
    0   /home/zhuzhao.cx/redis-shake/src/redis-shake/sync.go:452
            redis-shake.(*CmdSync).SyncCommand.func3

在看到redis-shake报错的时候停止redis-cluster test的执行

redis-cluster git:(master) ✗ ./test.sh
Redis master: 172.17.0.2
Redis Slave: 172.17.0.3
------------------------------------------------
Initial status of sentinel
------------------------------------------------
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=172.17.0.2:6379,slaves=4,sentinels=3
Current master is
172.17.0.2
6379
------------------------------------------------
Stop redis master
redis-cluster_master_1
Wait for 10 seconds
^C

跨云迁移支持

由于部分云环境，不支持sync/psync命令，采用scan命令进行全量扫描，然后restore写入到目的端，实现迁移的需求。类似rump功能。
该功能只支持全量迁移，不支持增量。
计划开发v1.4。

希望能够在restore模式下，支持metric信息

add ci

need adding ci

rump模式迁移报错DUMP payload version or checksump are wrong

疑似key number过多导致

rump模式迁移集群select报错

2019/07/25 09:48:20 [PANIC] dbRumper[0] executor[0] restore key[user:passwordUpdateStatus:32a9d72c51cf4c83b644c594170f3890] error[ERR SELECT is not allowed in cluster mode]: pttl[13012576], value length[13], rdb version[8], checksum[14166943956650907039], check error[<nil>]

同步出错

redis 2.8 到5.0
2019/04/19 20:24:45 [WARN] Event:GetFakeSlaveOffsetFail Id:redis-shake Warn:OffsetNotFoundInInfo
2019/04/19 20:24:45 [INFO] sync: +forward=825 +nbypass=0 +nbytes=0
2019/04/19 20:24:46 [INFO] sync: +forward=748 +nbypass=0 +nbytes=0
2019/04/19 20:24:47 [INFO] sync: +forward=641 +nbypass=0 +nbytes=0
2019/04/19 20:24:48 [INFO] sync: +forward=688 +nbypass=0 +nbytes=0
2019/04/19 20:24:49 [INFO] sync: +forward=678 +nbypass=0 +nbytes=0
2019/04/19 20:24:50 [INFO] sync: +forward=598 +nbypass=0 +nbytes=0
2019/04/19 20:24:51 [INFO] sync: +forward=689 +nbypass=0 +nbytes=0
2019/04/19 20:24:52 [INFO] sync: +forward=846 +nbypass=0 +nbytes=0
2019/04/19 20:24:53 [INFO] sync: +forward=698 +nbypass=0 +nbytes=0
2019/04/19 20:24:54 [INFO] sync: +forward=749 +nbypass=0 +nbytes=0
2019/04/19 20:24:55 [WARN] Event:GetFakeSlaveOffsetFail Id:redis-shake Warn:OffsetNotFoundInInfo
2019/04/19 20:24:55 [INFO] sync: +forward=693 +nbypass=0 +nbytes=0
2019/04/19 20:24:56 [INFO] sync: +forward=854 +nbypass=0 +nbytes=0
2019/04/19 20:24:57 [INFO] sync: +forward=700 +nbypass=0 +nbytes=0
2019/04/19 20:24:58 [INFO] sync: +forward=676 +nbypass=0 +nbytes=0
2019/04/19 20:24:59 [INFO] sync: +forward=627 +nbypass=0 +nbytes=0
2019/04/19 20:25:00 [INFO] sync: +forward=781 +nbypass=0 +nbytes=0
2019/04/19 20:25:01 [INFO] sync: +forward=673 +nbypass=0 +nbytes=0
2019/04/19 20:25:02 [PANIC] Event:ErrorReply Id:redis-shake Command: [unknown] Error: MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.
[stack]:
0 /home/zhuzhao.cx/redis-shake/src/redis-shake/sync.go:415
redis-shake.(*CmdSync).SyncCommand.func2
... ...

standalone sync 数据到 cluster报错

环境：
源端redis版本：2.8.23
目的端redis版本：4.0.14
shake版本：v1.6.14
报错日志如下：

2019/08/07 11:38:39 [INFO] dbSyncer[0] total=1352383372 -   1269785219 [ 93%]  entry=11639996
2019/08/07 11:38:40 [INFO] dbSyncer[0] total=1352383372 -   1277721210 [ 94%]  entry=11715744
2019/08/07 11:38:41 [INFO] dbSyncer[0] total=1352383372 -   1284887683 [ 95%]  entry=11787117
2019/08/07 11:38:42 [INFO] dbSyncer[0] total=1352383372 -   1291825175 [ 95%]  entry=11856256
2019/08/07 11:38:43 [INFO] dbSyncer[0] total=1352383372 -   1299130996 [ 96%]  entry=11926784
2019/08/07 11:38:44 [INFO] dbSyncer[0] total=1352383372 -   1307006281 [ 96%]  entry=11986998
2019/08/07 11:38:45 [INFO] dbSyncer[0] total=1352383372 -   1315537352 [ 97%]  entry=12047234
2019/08/07 11:38:46 [INFO] dbSyncer[0] total=1352383372 -   1322866812 [ 97%]  entry=12117705
2019/08/07 11:38:47 [INFO] dbSyncer[0] total=1352383372 -   1332603869 [ 98%]  entry=12180760
2019/08/07 11:38:48 [INFO] dbSyncer[0] total=1352383372 -   1340029408 [ 99%]  entry=12243857
2019/08/07 11:38:49 [INFO] dbSyncer[0] total=1352383372 -   1347629960 [ 99%]  entry=12316338
2019/08/07 11:38:49 [INFO] dbSyncer[0] total=1352383372 -   1352383372 [100%]  entry=12350830
2019/08/07 11:38:49 [INFO] dbSyncer[0] sync rdb done
2019/08/07 11:38:49 [INFO] dbSyncer[0] Event:IncrSyncStart	Id:redis-shake
2019/08/07 11:38:49 [WARN] dbSyncer[0] GetFakeSlaveOffset not enable when psync == false
2019/08/07 11:38:49 [PANIC] dbSyncer[0] Event:SendToTargetFail	Id:redis-shake	Error:transaction command[setex] key[game_msg_user_msg_7_2_15400_202917_1_zset_empty] not hashed in the same slot
[stack]:
    0   /home/zhuzhao.cx/redis-shake/src/redis-shake/sync.go:674
            redis-shake.(*dbSyncer).syncCommand.func4

Can RedisShake set delay for syncer and schedule to run?

1 . Can set delay in sync function ?

2019/07/11 16:33:12 [INFO] dbSyncer[0] sync: +forwardCommands=3 +filterCommands=0 +writeBytes=262

Can set schedule for run sync command ?

thank you

redis-shake 5.0 stream大key支持

目前，对于stream类型目前统一restore处理，这对于stream比较小的情况没有问题，但是stream如果value过大，将可能导致以下2个风险：

目的端redis有限流策略将会引发限流
目的端redis由于load需要消耗内存，有引发oom的风险。
超过700MB大小直接restore失败

后续考虑进行优化拆分，不用restore处理，xrange解析源端的拉取命令，挨个xadd写入目的端。对于group consumer类型，还需要适配PEL队列。
这个处理有点繁琐，暂时先搁置。

rump模式不完善

rump模式目前只能拉取db=0的库，别的库没办法拉取。

Can RedisShake skip sync all data when start but continue do "forwardCommands" ?

In this case my source redis have key about 1 Million and I have start redis shake with sync command. It takes a while for sync data.
Later redisshake was terminated and i want to start redisshake again then redisshake have sync all key in source redis again and print log all the same key have sync

eg.

2019/07/12 17:35:40 [INFO] warning, rewrite key: {$key}
2019/07/12 17:35:40 [INFO] warning, rewrite key: {$key}
....
...
...
(key per line. if i have 1 million key it will print 1 million line )

I want to skip sync all data and do "fowardCommands" only.

What is your recommend way?

pls forgive my english skill, thank you very much

filter 规则不生效

源版本：codis-server 3.2.4
目标版本： redis_version:5.0.2
share版本：improve-1.6.9,1737b9045681eb923830add7743606075c71b748,go1.10.3,2019-06-24_16:14:53
过来策略：
filter.db = 0
filter.key =mms*;aisp*

结果：
目标redis一个key都没有同步过去

v1.6.4全量同步报错

2019/06/12 10:35:05 [INFO] the target redis type is cluster, only pass db0
2019/06/12 10:35:05 [INFO] input password is empty, skip auth address[XX.XX.20.81:port] with type[auth].
2019/06/12 10:35:05 [INFO] input password is empty, skip auth address[XX.XX.8.48:port] with type[auth].
2019/06/12 10:35:05 [INFO] input password is empty, skip auth address[xx.xx.118.56:port] with type[auth].
2019/06/12 10:35:05 [INFO] input password is empty, skip auth address[XX.XX.6.60:port] with type[auth].
2019/06/12 10:35:05 [INFO] input password is empty, skip auth address[xx.xx.25.48:port] with type[auth].
2019/06/12 10:35:05 [INFO] input password is empty, skip auth address[xx.xx.120.88:port] with type[auth].
2019/06/12 10:35:05 [WARN]
______________________________
\                             \           _         ______ |
 \                             \        /   \___-=O'/|O'/__|
  \  redis-shake, here we go !! \_______\          / | /    )
  /                             /        '/-==__ _/__|/__=-|  -GM
 /                             /         *             \ | |
/                             /                        (o)
------------------------------
if you have any problem, please visit https://github.com/alibaba/RedisShake/wiki/FAQ

2019/06/12 10:35:05 [INFO] redis-shake configuration: {"Id":"redis-shake","LogFile":"./redis-shake.log","LogLevel":"info","SystemProfile":9310,"HttpProfile":9320,"NCpu":0,"Parallel":32,"SourceType":"standalone","SourceAddress":"cc.cc.cc.cc:port","SourcePasswordRaw":"","SourcePasswordEncoding":"","SourceVersion":0,"SourceAuthType":"auth","SourceParallel":1,"SourceTLSEnable":false,"TargetAddress":"@xx.xx.120.88:port","TargetPasswordRaw":"","TargetPasswordEncoding":"","TargetVersion":0,"TargetDB":-1,"TargetAuthType":"auth","TargetType":"cluster","TargetTLSEnable":false,"RdbInput":["local"],"RdbOutput":"local_dump","RdbParallel":1,"RdbSpecialCloud":"","FakeTime":"","Rewrite":true,"FilterDB":"","FilterKey":[],"FilterSlot":[],"BigKeyThreshold":524288000,"Psync":false,"Metric":true,"MetricPrintLog":false,"HeartbeatUrl":"","HeartbeatInterval":3,"HeartbeatExternal":"test external","HeartbeatNetworkInterface":"","SenderSize":104857600,"SenderCount":5000,"SenderDelayChannelSize":65535,"KeepAlive":0,"PidPath":"","ScanKeyNumber":50,"ScanSpecialCloud":"","ScanKeyFile":"","ReplaceHashTag":false,"ExtraInfo":false,"SockFileName":"","SockFileSize":0,"SourceAddressList":["cc.cc.cc.cc:port"],"TargetAddressList":["XX.XX.20.81:port","XX.XX.8.48:port","xx.xx.118.56:port","XX.XX.6.60:port","xx.xx.25.48:port","xx.xx.120.88:port"],"HeartbeatIp":"127.0.0.1","ShiftTime":0,"TargetRedisVersion":"4.0.14","TargetReplace":true}
2019/06/12 10:35:05 [INFO] routine[0] starts syncing data from cc.cc.cc.cc:port to [XX.XX.20.81:port XX.XX.8.48:port xx.xx.118.56:port XX.XX.6.60:port xx.xx.25.48:port xx.xx.120.88:port] with http[9321]
2019/06/12 10:35:05 [INFO] input password is empty, skip auth address[cc.cc.cc.cc:port] with type[auth].
2019/06/12 10:35:06 [INFO] dbSyncer[0] +
2019/06/12 10:35:07 [INFO] dbSyncer[0] -
2019/06/12 10:35:07 [INFO] dbSyncer[0] +
2019/06/12 10:35:08 [INFO] dbSyncer[0] -
2019/06/12 10:35:08 [INFO] dbSyncer[0] +
2019/06/12 10:35:09 [INFO] dbSyncer[0] -
2019/06/12 10:35:09 [INFO] dbSyncer[0] +
2019/06/12 10:35:10 [INFO] dbSyncer[0] +
2019/06/12 10:35:10 [INFO] dbSyncer[0] rdb file size = 131183832
2019/06/12 10:35:11 [INFO] dbSyncer[0] total=131183832 -      5936230 [  4%]  entry=73776
2019/06/12 10:35:12 [INFO] dbSyncer[0] total=131183832 -     12509968 [  9%]  entry=166533
2019/06/12 10:35:13 [INFO] dbSyncer[0] total=131183832 -     19167220 [ 14%]  entry=259863
2019/06/12 10:35:14 [INFO] dbSyncer[0] total=131183832 -     25711606 [ 19%]  entry=342123
2019/06/12 10:35:15 [INFO] dbSyncer[0] total=131183832 -     31863873 [ 24%]  entry=407146
2019/06/12 10:35:16 [INFO] dbSyncer[0] total=131183832 -     36975071 [ 28%]  entry=437695
2019/06/12 10:35:17 [INFO] dbSyncer[0] total=131183832 -     42981252 [ 32%]  entry=504336
2019/06/12 10:35:18 [INFO] dbSyncer[0] total=131183832 -     49077243 [ 37%]  entry=572174
2019/06/12 10:35:19 [INFO] dbSyncer[0] total=131183832 -     55879778 [ 42%]  entry=664604
2019/06/12 10:35:20 [INFO] dbSyncer[0] total=131183832 -     61482482 [ 46%]  entry=710679
2019/06/12 10:35:21 [INFO] dbSyncer[0] total=131183832 -     68403047 [ 52%]  entry=800163
2019/06/12 10:35:22 [INFO] dbSyncer[0] total=131183832 -     75020589 [ 57%]  entry=859849
2019/06/12 10:35:23 [INFO] dbSyncer[0] total=131183832 -     79977101 [ 60%]  entry=929511
2019/06/12 10:35:24 [INFO] dbSyncer[0] total=131183832 -     85255016 [ 64%]  entry=962036
2019/06/12 10:35:25 [INFO] dbSyncer[0] total=131183832 -     91682982 [ 69%]  entry=1041265
2019/06/12 10:35:26 [INFO] dbSyncer[0] total=131183832 -     97855745 [ 74%]  entry=1081210
2019/06/12 10:35:27 [INFO] dbSyncer[0] total=131183832 -    102759902 [ 78%]  entry=1133170
2019/06/12 10:35:28 [INFO] dbSyncer[0] total=131183832 -    109733064 [ 83%]  entry=1191292
2019/06/12 10:35:29 [INFO] dbSyncer[0] total=131183832 -    115457745 [ 88%]  entry=1239640
2019/06/12 10:35:30 [INFO] dbSyncer[0] total=131183832 -    121370115 [ 92%]  entry=1277261
2019/06/12 10:35:31 [INFO] dbSyncer[0] total=131183832 -    126519409 [ 96%]  entry=1320204
2019/06/12 10:35:32 [INFO] dbSyncer[0] total=131183832 -    131183832 [100%]  entry=1379513
2019/06/12 10:35:32 [INFO] dbSyncer[0] sync rdb done
2019/06/12 10:35:32 [WARN] dbSyncer[0] GetFakeSlaveOffset not enable when psync == false
2019/06/12 10:35:32 [INFO] dbSyncer[0] Event:IncrSyncStart	Id:redis-shake
2019/06/12 10:35:32 [PANIC] dbSyncer[0] Event:SendToTargetFail	Id:redis-shake	Error:Put: no key found in args
[stack]:
    0   /home/zhuzhao.cx/redis-shake/src/redis-shake/sync.go:678
            redis-shake.(*dbSyncer).syncCommand.func4
        ... ...

同步出错

阿里云redis实例之间的同步出如下错误：
版本：2.8->4.0
2019/06/18 11:24:09 [INFO] routine[0] starts syncing data from 192.168.6.2:6379 to [r-wz914ed7bd8be154506.redis.rds.aliyuncs.com:6379] wit
h http[9321]2019/06/18 11:24:09 [PANIC] invalid sync response, rsp = '-ERR command sync not support for normal user
'
[stack]:
0 /home/zhuzhao.cx/redis-shake/src/redis-shake/common/utils.go:190
redis-shake/common.waitRdbDump.func1

报错信息不明显

这个原因是用的redigo直接处理返回，但是在类型是redis.Error，redigo驱动应该抛出具体的错误，但是其实并没有：

dumps, err := redis.Strings(dre.sourceClient.Do(""))
if err != nil && err != redis.ErrNil {
	return fmt.Errorf("do dump with failed[%v]", err)
}

修改后的代码，抛出具体的错误类型。目前redis-shake里面还有类似的问题，由于比较多，暂时先不统一修改了。这里最好是由驱动层进行抛出错误，当type是redis.Error的时候：redis.Strings。

reply, err := dre.sourceClient.Do("")
dumps, err := redis.Strings(reply, err)
if err != nil && err != redis.ErrNil {
	return fmt.Errorf("do dump with failed[%v], reply[%v]", err, reply)
}

计划release在v1.6.17版本

dump模式rdb.parallel没有限制住

设置rdb.parallel=4，但并发都在下载。
问题出在dump模式控制并发的参数不是rdb.paralle，而是source.parallel，source.parallel参数当初应该是改了一半，没加在配置文件里面。此处命令统一，表示对源端RDB/全量同步的限制，统一都改为source.parallel，适用模式：restore, dump, sync。
另外，rdb.xxx进行重命名：

rdb.input 重命名为 source.rdb.input，同时向上兼容，rdb.input依旧有效。
rdb.output重命名为target.rdb.output，同时向上兼容，rdb.output依旧有效。
rdb.parallel，如上所示，改名为source.parallel，同时向上兼容，rdb.parallel依旧有效。
rdb.special_cloud，重命名为source.rdb.special_cloud，同时向上兼容，rdb.special_cloud依旧有效。

此部分功能修改完，将会在wiki添加说明和例子。

希望加入Sentinel的支持

希望能够加入Sentinel的支持，毕竟单节点的Redis在生产环境是很少用的，希望RedisShake能够订阅Sentinel，当Sentinel发生角色切换的时候，能够切换自己的源（期望是某个从节点）

不定期全量卡住

源端是2.8.7的redis，同步的时候出现了全量同步到一定比例卡住的现象。每次卡住的比例也不一样，有时候也不卡，顺利进入增量。
tcpdump抓包和redis-shake堆栈都是源端block在发送上，没有继续发送后续的数据。
问题暂时不明，疑似redis本身的问题

全量同步过程中出错

Redis版本：4.0.12
Shake版本：develop,4435cae466f42ccaf11b2c43e8754c0ae9603a09,go1.12.1,2019-05-30_22:12:46
环境：源2C8G 目的2C8G
OS: CentOS 7
步骤：

源已有数据30W条数据总大小5.2G，数据类型数据长度随机生成
开启Shake
源缓冲区大小100M，开启脚本向源的写入，计划15W条，预计2.6G

如果步骤2、3调换则会引起 #26

Shake日志如下：

2019/06/06 15:14:56 [INFO] dbSyncer[0] psync connect '192.168.111.232:6379' with auth type[auth] OK!
2019/06/06 15:14:56 [INFO] dbSyncer[0] psync send listening port[9320] OK!
2019/06/06 15:14:56 [INFO] dbSyncer[0] try to send 'psync' command
2019/06/06 15:14:56 [INFO] Event:FullSyncStart	Id:redis-shake
2019/06/06 15:14:56 [INFO] dbSyncer[0] psync runid = f7a516fe11d71060c45459b3796bd5138276d1ca offset = 76551343, fullsync
2019/06/06 15:14:56 [INFO] dbSyncer[0] +
2019/06/06 15:14:57 [INFO] dbSyncer[0] -
2019/06/06 15:14:57 [INFO] dbSyncer[0] +
2019/06/06 15:14:58 [INFO] dbSyncer[0] -
2019/06/06 15:14:58 [INFO] dbSyncer[0] +
2019/06/06 15:14:59 [INFO] dbSyncer[0] -
2019/06/06 15:14:59 [INFO] dbSyncer[0] +
2019/06/06 15:15:00 [INFO] dbSyncer[0] -
2019/06/06 15:15:00 [INFO] dbSyncer[0] +
2019/06/06 15:15:01 [INFO] dbSyncer[0] -
2019/06/06 15:15:01 [INFO] dbSyncer[0] +
2019/06/06 15:15:02 [INFO] dbSyncer[0] +
2019/06/06 15:15:03 [INFO] dbSyncer[0] -
2019/06/06 15:15:03 [INFO] dbSyncer[0] +
2019/06/06 15:15:04 [INFO] dbSyncer[0] -
2019/06/06 15:15:04 [INFO] dbSyncer[0] +
2019/06/06 15:15:05 [INFO] dbSyncer[0] -
2019/06/06 15:15:05 [INFO] dbSyncer[0] +
2019/06/06 15:15:06 [INFO] dbSyncer[0] +
2019/06/06 15:15:07 [INFO] dbSyncer[0] +
2019/06/06 15:15:08 [INFO] dbSyncer[0] +
2019/06/06 15:15:09 [INFO] dbSyncer[0] -
2019/06/06 15:15:09 [INFO] dbSyncer[0] +
2019/06/06 15:15:10 [INFO] dbSyncer[0] -
2019/06/06 15:15:10 [INFO] dbSyncer[0] +
2019/06/06 15:15:11 [INFO] dbSyncer[0] -
2019/06/06 15:15:11 [INFO] dbSyncer[0] +
2019/06/06 15:15:12 [INFO] dbSyncer[0] -
2019/06/06 15:15:12 [INFO] dbSyncer[0] +
2019/06/06 15:15:13 [INFO] dbSyncer[0] -
2019/06/06 15:15:13 [INFO] dbSyncer[0] +
2019/06/06 15:15:14 [INFO] dbSyncer[0] -
2019/06/06 15:15:14 [INFO] dbSyncer[0] +
2019/06/06 15:15:15 [INFO] dbSyncer[0] -
2019/06/06 15:15:15 [INFO] dbSyncer[0] +
2019/06/06 15:15:16 [INFO] dbSyncer[0] -
2019/06/06 15:15:16 [INFO] dbSyncer[0] +
2019/06/06 15:15:17 [INFO] dbSyncer[0] +
2019/06/06 15:15:18 [INFO] dbSyncer[0] -
2019/06/06 15:15:18 [INFO] dbSyncer[0] +
2019/06/06 15:15:19 [INFO] dbSyncer[0] -
2019/06/06 15:15:19 [INFO] dbSyncer[0] +
2019/06/06 15:15:20 [INFO] dbSyncer[0] -
2019/06/06 15:15:20 [INFO] dbSyncer[0] +
2019/06/06 15:15:21 [INFO] dbSyncer[0] +
2019/06/06 15:15:22 [INFO] dbSyncer[0] +
2019/06/06 15:15:23 [INFO] dbSyncer[0] -
2019/06/06 15:15:23 [INFO] dbSyncer[0] +
2019/06/06 15:15:24 [INFO] dbSyncer[0] -
2019/06/06 15:15:24 [INFO] dbSyncer[0] +
2019/06/06 15:15:25 [INFO] dbSyncer[0] +
2019/06/06 15:15:26 [INFO] dbSyncer[0] -
2019/06/06 15:15:26 [INFO] dbSyncer[0] +
2019/06/06 15:15:27 [INFO] dbSyncer[0] +
2019/06/06 15:15:28 [INFO] dbSyncer[0] -
2019/06/06 15:15:28 [INFO] dbSyncer[0] +
2019/06/06 15:15:29 [INFO] dbSyncer[0] -
2019/06/06 15:15:29 [INFO] dbSyncer[0] +
2019/06/06 15:15:30 [INFO] dbSyncer[0] -
2019/06/06 15:15:30 [INFO] dbSyncer[0] +
2019/06/06 15:15:31 [INFO] dbSyncer[0] -
2019/06/06 15:15:31 [INFO] dbSyncer[0] +
2019/06/06 15:15:32 [INFO] dbSyncer[0] +
2019/06/06 15:15:33 [INFO] dbSyncer[0] -
2019/06/06 15:15:33 [INFO] dbSyncer[0] +
2019/06/06 15:15:34 [INFO] dbSyncer[0] -
2019/06/06 15:15:34 [INFO] dbSyncer[0] +
2019/06/06 15:15:35 [INFO] dbSyncer[0] -
2019/06/06 15:15:35 [INFO] dbSyncer[0] +
2019/06/06 15:15:36 [INFO] dbSyncer[0] -
2019/06/06 15:15:36 [INFO] dbSyncer[0] +
2019/06/06 15:15:37 [INFO] dbSyncer[0] -
2019/06/06 15:15:37 [INFO] dbSyncer[0] +
2019/06/06 15:15:38 [INFO] dbSyncer[0] -
2019/06/06 15:15:38 [INFO] dbSyncer[0] +
2019/06/06 15:15:39 [INFO] dbSyncer[0] -
2019/06/06 15:15:39 [INFO] dbSyncer[0] +
2019/06/06 15:15:40 [INFO] dbSyncer[0] -
2019/06/06 15:15:40 [INFO] dbSyncer[0] +
2019/06/06 15:15:40 [INFO] dbSyncer[0] rdb file size = 924836132
2019/06/06 15:15:40 [INFO] Aux information key:redis-ver value:4.0.12
2019/06/06 15:15:40 [INFO] Aux information key:redis-bits value:64
2019/06/06 15:15:40 [INFO] Aux information key:ctime value:1559805284
2019/06/06 15:15:40 [INFO] Aux information key:used-mem value:5634337848
2019/06/06 15:15:40 [INFO] Aux information key:repl-stream-db value:0
2019/06/06 15:15:40 [INFO] Aux information key:repl-id value:f7a516fe11d71060c45459b3796bd5138276d1ca
2019/06/06 15:15:40 [INFO] Aux information key:repl-offset value:76551343
2019/06/06 15:15:40 [INFO] Aux information key:aof-preamble value:0
2019/06/06 15:15:40 [INFO] db_size:312951 expire_size:0
2019/06/06 15:15:41 [INFO] dbSyncer[0] total=924836132 -      9155943 [  0%]  entry=2109
2019/06/06 15:15:42 [INFO] dbSyncer[0] total=924836132 -     16107663 [  1%]  entry=4411
2019/06/06 15:15:43 [INFO] dbSyncer[0] total=924836132 -     22914262 [  2%]  entry=6750
2019/06/06 15:15:44 [INFO] dbSyncer[0] total=924836132 -     29707595 [  3%]  entry=9060
2019/06/06 15:15:45 [INFO] dbSyncer[0] total=924836132 -     35741354 [  3%]  entry=11067
2019/06/06 15:15:46 [INFO] dbSyncer[0] total=924836132 -     42911547 [  4%]  entry=13480
2019/06/06 15:15:47 [INFO] dbSyncer[0] total=924836132 -     49512746 [  5%]  entry=15664
2019/06/06 15:15:48 [INFO] dbSyncer[0] total=924836132 -     56104645 [  6%]  entry=17910
2019/06/06 15:15:49 [INFO] dbSyncer[0] total=924836132 -     63045141 [  6%]  entry=20186
2019/06/06 15:15:50 [INFO] dbSyncer[0] total=924836132 -     69949205 [  7%]  entry=22559
2019/06/06 15:15:51 [INFO] dbSyncer[0] total=924836132 -     77117063 [  8%]  entry=24968
2019/06/06 15:15:52 [INFO] dbSyncer[0] total=924836132 -     84282189 [  9%]  entry=27369
2019/06/06 15:15:53 [INFO] dbSyncer[0] total=924836132 -     90989718 [  9%]  entry=29628
2019/06/06 15:15:54 [INFO] dbSyncer[0] total=924836132 -     97729739 [ 10%]  entry=31946
2019/06/06 15:15:55 [INFO] dbSyncer[0] total=924836132 -    104256188 [ 11%]  entry=34115
2019/06/06 15:15:56 [INFO] dbSyncer[0] total=924836132 -    111033890 [ 12%]  entry=36399
2019/06/06 15:15:57 [INFO] dbSyncer[0] total=924836132 -    118535751 [ 12%]  entry=38911
2019/06/06 15:15:58 [INFO] dbSyncer[0] total=924836132 -    126189233 [ 13%]  entry=41523
2019/06/06 15:15:59 [INFO] dbSyncer[0] total=924836132 -    133360750 [ 14%]  entry=44066
2019/06/06 15:16:00 [INFO] dbSyncer[0] total=924836132 -    140788499 [ 15%]  entry=46560
2019/06/06 15:16:01 [INFO] dbSyncer[0] total=924836132 -    147116545 [ 15%]  entry=48702
2019/06/06 15:16:02 [INFO] dbSyncer[0] total=924836132 -    153627524 [ 16%]  entry=51076
2019/06/06 15:16:03 [INFO] dbSyncer[0] total=924836132 -    160951290 [ 17%]  entry=53512
2019/06/06 15:16:04 [INFO] dbSyncer[0] total=924836132 -    167516368 [ 18%]  entry=55650
2019/06/06 15:16:05 [INFO] dbSyncer[0] total=924836132 -    174250116 [ 18%]  entry=57910
2019/06/06 15:16:06 [INFO] dbSyncer[0] total=924836132 -    180506205 [ 19%]  entry=59992
2019/06/06 15:16:07 [INFO] dbSyncer[0] total=924836132 -    187490501 [ 20%]  entry=62331
2019/06/06 15:16:08 [INFO] dbSyncer[0] total=924836132 -    194030871 [ 20%]  entry=64546
2019/06/06 15:16:09 [INFO] dbSyncer[0] total=924836132 -    199593625 [ 21%]  entry=66462
2019/06/06 15:16:10 [INFO] dbSyncer[0] total=924836132 -    206292732 [ 22%]  entry=68713
2019/06/06 15:16:11 [INFO] dbSyncer[0] total=924836132 -    212929736 [ 23%]  entry=70948
2019/06/06 15:16:12 [INFO] dbSyncer[0] total=924836132 -    219194011 [ 23%]  entry=73043
2019/06/06 15:16:13 [INFO] dbSyncer[0] total=924836132 -    225388180 [ 24%]  entry=75130
2019/06/06 15:16:14 [INFO] dbSyncer[0] total=924836132 -    231766226 [ 25%]  entry=77299
2019/06/06 15:16:15 [INFO] dbSyncer[0] total=924836132 -    238580544 [ 25%]  entry=79658
2019/06/06 15:16:16 [INFO] dbSyncer[0] total=924836132 -    244616746 [ 26%]  entry=81763
2019/06/06 15:16:17 [INFO] dbSyncer[0] total=924836132 -    250529352 [ 27%]  entry=83773
2019/06/06 15:16:18 [INFO] dbSyncer[0] total=924836132 -    256156609 [ 27%]  entry=85713
2019/06/06 15:16:19 [INFO] dbSyncer[0] total=924836132 -    262837718 [ 28%]  entry=88046
2019/06/06 15:16:20 [INFO] dbSyncer[0] total=924836132 -    269365289 [ 29%]  entry=90278
2019/06/06 15:16:21 [INFO] dbSyncer[0] total=924836132 -    276758962 [ 29%]  entry=92645
2019/06/06 15:16:22 [INFO] dbSyncer[0] total=924836132 -    283908533 [ 30%]  entry=95059
2019/06/06 15:16:23 [INFO] dbSyncer[0] total=924836132 -    290261639 [ 31%]  entry=97181
2019/06/06 15:16:24 [INFO] dbSyncer[0] total=924836132 -    296866958 [ 32%]  entry=99416
2019/06/06 15:16:25 [INFO] dbSyncer[0] total=924836132 -    302758123 [ 32%]  entry=101461
2019/06/06 15:16:26 [INFO] dbSyncer[0] total=924836132 -    309017140 [ 33%]  entry=103532
2019/06/06 15:16:27 [INFO] dbSyncer[0] total=924836132 -    315946797 [ 34%]  entry=105912
2019/06/06 15:16:28 [INFO] dbSyncer[0] total=924836132 -    322180663 [ 34%]  entry=108061
2019/06/06 15:16:29 [INFO] dbSyncer[0] total=924836132 -    329206470 [ 35%]  entry=110423
2019/06/06 15:16:30 [INFO] dbSyncer[0] total=924836132 -    336995059 [ 36%]  entry=113063
2019/06/06 15:16:31 [INFO] dbSyncer[0] total=924836132 -    343731725 [ 37%]  entry=115326
2019/06/06 15:16:32 [PANIC] read error
[error]: EOF
[stack]:
    1   /home/vagrant/projects/RedisShake/src/redis-shake/common/utils.go:818
            redis-shake/common.Iocopy
    0   /home/vagrant/projects/RedisShake/src/redis-shake/sync.go:317
            redis-shake.(*dbSyncer).sendPSyncCmd.func1
        ... ...

迁移过程报oom，但是服务器内存只使用了10%左右

迁移redis中，报oom错误，但是整个过程中服务器内存使用量一直在250M左右，服务器内存为16GB，详细报错日志如下：
2019/04/26 18:14:44 [INFO] total=2745316203 - 61479452 [ 2%] entry=308680
2019/04/26 18:14:45 [INFO] total=2745316203 - 61870915 [ 2%] entry=310614
2019/04/26 18:14:46 [PANIC] parse rdb entry error, if the err is :EOF, please check that if the src db log has client outout buffer oom, if so set output buffer larger.
[error]: EOF
10 /data/RedisShake-develop/src/pkg/rdb/reader.go:87
pkg/rdb.(*rdbReader).Read
9 /data/go/src/io/io.go:535
io.(*teeReader).Read
8 /data/RedisShake-develop/src/pkg/rdb/reader.go:85
pkg/rdb.(*rdbReader).Read
7 /data/go/src/io/io.go:310
io.ReadAtLeast
6 /data/go/src/io/io.go:329
io.ReadFull
5 /data/RedisShake-develop/src/pkg/rdb/reader.go:407
pkg/rdb.(*rdbReader).readFull
4 /data/RedisShake-develop/src/pkg/rdb/reader.go:413
pkg/rdb.(*rdbReader).ReadBytes
3 /data/RedisShake-develop/src/pkg/rdb/reader.go:324
pkg/rdb.(*rdbReader).ReadString
2 /data/RedisShake-develop/src/pkg/rdb/reader.go:117
pkg/rdb.(*rdbReader).readObjectValue
1 /data/RedisShake-develop/src/pkg/rdb/loader.go:170
pkg/rdb.(*Loader).NextBinEntry
0 /data/RedisShake-develop/src/redis-shake/common/utils.go:772
redis-shake/common.NewRDBLoader.func1
... ...
[stack]:
0 /data/RedisShake-develop/src/redis-shake/common/utils.go:773
redis-shake/common.NewRDBLoader.func1
... ...

请教下同步中断的问题

hi,您好：
我在测试redis cluster的时候，使用redis shake从源端的3.0单实例往目标端4.0.14的cluster同步。全同步的时候，总是会中断卡住。但是cluster是正常的。操作系统版本为centOS7.4
报错如下：
2019/07/16 16:29:31 [PANIC] restore command error key:¬첵content_type_564827a1a71d802acc44f0c5 err:CLUSTERDOWN The cluster is down
[error]: CLUSTERDOWN The cluster is down
[stack]:
1 /home/zhuzhao.cx/redis-shake/src/redis-shake/common/utils.go:808
redis-shake/common.RestoreRdbEntry
0 /home/zhuzhao.cx/redis-shake/src/redis-shake/sync.go:441
redis-shake.(*dbSyncer).syncRDBFile.func1.1

RedisTypeCluster 模式下，优化地址填写一个即可

对于源端是 cluster类型的，比如说有30个node,也要填30个地址？
是否可以考虑，填写1个地址，然后通过 cluster nodes命令获取集群拓扑信息，这样用户不用填30个地址了~

redis 4.0 cluster 同步到 redis 3.2.9 主从结构

程序没有使用4.0相关的命令，都是基础命令。
报错如下：
2019/05/21 16:20:01 [PANIC] target key name is busy:%!(EXTRA string=-^P<9C>K~m)
[stack]:
1 /home/zhuzhao.cx/redis-shake/src/redis-shake/common/utils.go:756
redis-shake/common.RestoreRdbEntry
0 /home/zhuzhao.cx/redis-shake/src/redis-shake/sync.go:435
redis-shake.(*dbSyncer).syncRDBFile.func1.1
... ...