Giter Club home page Giter Club logo

Comments (4)

suxb201 avatar suxb201 commented on August 24, 2024
  1. 使用最新版本,优化了内存占用
  2. key 丢失不应该,可以翻翻日志看看为什么
  3. 速度慢是预期内的,想要速度快,可以多起几个 shake,每个db 一个 shake 这样不会慢

from redisshake.

tentosleep avatar tentosleep commented on August 24, 2024

对比.pdf
1.这边使用的是最新的Redisshake4.0.5版本redis-shake-linux-amd64.tar.gz,但是可以很明显地观察到迁移相同的数据到相同规格的集群,内存消耗远大于2版本;
2.日志就是上面列举的,在小数据量的情况下,src-1、src0和src-2三个分片都会显示同步进度类似size=[123 MiB/1.3 GiB]",最后同步都能顺利完成;大数据量的情况下src-1,会一直卡在hand shaking阶段,可能就是因为这个导致丢数据,请问有解决方法吗
3.源目Redis只用了db0这一个库

Keyspace

db0:keys=27020349,expires=0,avg_ttl=0

from redisshake.

suxb201 avatar suxb201 commented on August 24, 2024

@tentosleep 一种缓解方法是,如果源端有 3 个分片,那么启动 3 个 redis-shake。其中 reader 分别配置为三个源端,writer 配置为目的端集群。可以解决同步慢的问题,内存膨胀问题难解决,现在应该不会很严重,你可以给些数据看看,比如源端内存使用量,shake内存使用量,是否用大 hash、set、list 等结构。

from redisshake.

tentosleep avatar tentosleep commented on August 24, 2024

@suxb201 好的多谢,内存慢的问题我试一下尝试开三个进程,但这样和老版本想比更为繁琐;
但是缺数据的情况是因为数据量过大的问题吗,当前源端总内存是48个G左右,每对主从16G,集群都是string类型的散key,最大的key就几kb左右,SHAKE的内存使用量上面截图中有,4.0.5Redisshake进行同步会消耗大概10G左右的内存;
现在比较在意的是用2版本进行数据同步没有这个缺数据的情况,而且内存消耗也远小于4版本,性能上是否2版本略胜一筹
以下是key的总体扫描情况
-------- summary -------
Sampled 27010993 keys in the keyspace!
Total key length in bytes is 297120990 (avg len 11.00)

Biggest string found '"188683"' has 1219 bytes
27010993 strings with 12782063401 bytes (100.00% of keys, avg size 473.22)

from redisshake.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.