Giter Club home page Giter Club logo

comparison-of-elasticache-migration-tools.'s Introduction

AWS-elasticache migration

aws elasticache 有限制一些指令的使用,想做跨雲的異地同步上限制較多,沒辦法使用 slaveof 或是psync 的方式來達成migration。

https://docs.aws.amazon.com/zh_tw/AmazonElastiCache/latest/red-ug/RestrictedCommands.html

使用 RedisShake 工具來達成異地的同步 https://github.com/stickermule/rump

RedisShake 是阿里雲所開發的開源工具 有支援 sync / restore / scan 三種資料同步的方式

SCAN 非阻塞式的獲取keys 能避免lock的問題

但SCAN 和DUMP 指令會對佔用soure redis instance 較多的CPU 資源。

image

RedisShake使用v3 版本時,有使用lua script 把大key給過濾掉,但v3 相對v4 執行時所使用的多執行緒會短時間上升很多 造成 cmds 與new connection 上升太快。

image

image

導致後面新的connection 連不進來,前台報503.504 的錯誤。

v4 版本相對v3 在執行時,不會有command 與 new connection 衝太高的問題,但有遇到另一個問題是記憶體使用量太多導致OOM,腳本就直接kill 了。

要migration 到本地的redis 使用 13.x GB , 預計要抓快30GB ram 才夠

image

image

提aws support 工單請他們協助開啟psync 的功能,但會需要對同步的參數做調優。

repl-backlog-size

repl-backlog-ttl

min-replicas-to-write 和 min-replicas-max-lag

repl-timeout

image

使用 rdb-tool 工具來還原每日備份的rdb檔案 https://github.com/sripathikrishnan/redis-rdb-tools

若要過濾掉一些大key 可以搭配 --not-key 指令

例如要把這幾個大key 給排除 core_user_token_list / core_charm_fans_count: / core_charm_follow_count: / core_game_tbtguil

指令如下:

rdb -c protocol --db 1 --not-key '^core_user_token_list$|^core_charm_fans_count:$|^core_charm_follow_count:$|^core_game_tbtguil$' redis-db-0919-0001.rdb | redis-cli -p 6380 -n 1 --pipe

執行上不影響source redis instance 的資源。

comparison-of-elasticache-migration-tools.'s People

Contributors

cyleedba avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.