terry-mao / mysql-syncer Goto Github PK
View Code? Open in Web Editor NEWMySQL Syncer is a project which parse mysql binlog and sync to other datases, such as redis, mongodb and any other databases..
mysql-syncer's Introduction
# README.zh FOR Chinses # README.en FOR English
mysql-syncer's People
Contributors
Stargazers
Watchers
Forkers
lkunemail ttgive phoenixluo wtmmac guoyunsky gitssie yahveh fengabe xucheng zhanglei tan-zz-gx liujianping askyer chu888chu888 wbteve linwaterbin mrxiaoz chamsiin1982 lemonhall zhangjingpu pgy8288 huokedu koolhazz no2key jerrychen2008 jerrysearch hzrandd miuc hilerchyn lewis-ing evilsmile mr8 sdgdsffdsfff jucehrx xushan duzhanyuan chefxu gaizai zhimin711 chaohona kangzhenkang biggerbang delkyd ruckfull cybrobin zhangf911 yizhilong28 dikoufu xjyh jiesensun yucz trentsky zsmj513 jinrenlab hand-code meizhuhanxiang fengshao0907 tinyhome askskype chenbk85 4ecolbyte liucyao jxctx88 frlk bradbann windofthesky zacklin923 liuxiaozhen xiongxj wangcr2015 qiailin ycaihua jiangkevin brian110 yuanfeng0905 willenchou subbu6502 yy1117 kwokhunglee sandysuehe leonliu315 lb1980 xiaomaixiaomai jixiaod cluo jackerzhou dede8385 wfxiang08 liumorgan mzcaptainyang callmedba yuanddqiao yihaohuoche hubertwng beebol hyb1234hi maxshuang wacr2008 tennessine xiaolongn100mysql-syncer's Issues
Slave 启动日志报错，rs_read() failed, error or empty
启动master 后，接着启动slave 打印如下：
[root@localhost MySQL-Syncer]# ./rs.sh slave start
2015-03-23 11:47:07 [INFO] <thread : 140001125885760> add conf key cwd, type 3
2015-03-23 11:47:07 [INFO] <thread : 140001125885760> add conf key user, type 3
2015-03-23 11:47:07 [INFO] <thread : 140001125885760> add conf key pid, type 3
2015-03-23 11:47:07 [INFO] <thread : 140001125885760> add conf key log, type 3
2015-03-23 11:47:07 [INFO] <thread : 140001125885760> add conf key log.level, type 2
2015-03-23 11:47:07 [INFO] <thread : 140001125885760> add conf key debug.level, type 3
2015-03-23 11:47:07 [INFO] <thread : 140001125885760> add conf key daemon, type 2
2015-03-23 11:47:07 [INFO] <thread : 140001125885760> conf file = /home/xin/code/mysqlredis/MySQL-Syncer/etc/slave.cf, module = core
/tmp/slave.info
/tmp/目录下没有 rs_slave.pid ，但有 rs_slave.log
如下：
2015-03-23 11:47:07 [ERROR] <thread : 140001125885760> rs_read() failed, error or empty
这大概是什么问题，请指教
以上问题是因为 /tmp/slave.info 文件不正确而导致，已修改
现rs_master.log 报错 ：
2015-03-23 12:18:31 [ERROR] <thread : 139785948059392> get dumpcmd failed
2015-03-23 12:18:41 [ERROR] <thread : 139785948059392> get dumpcmd failed
2015-03-23 12:18:51 [ERROR] <thread : 139785948059392> get dumpcmd failed
2015-03-23 12:19:01 [ERROR] <thread : 139785948059392> get dumpcmd failed
.....
而rs_slave.log 则是：
2015-03-23 12:18:31 [ERROR] <thread : 139817795663616> retry connect to master
2015-03-23 12:18:41 [ERROR] <thread : 139817795663616> retry connect to master
2015-03-23 12:18:51 [ERROR] <thread : 139817795663616> retry connect to master
2015-03-23 12:19:01 [ERROR] <thread : 139817795663616> retry connect to master
2015-03-23 12:19:11 [ERROR] <thread : 139817795663616> retry connect to master
2015-03-23 12:19:21 [ERROR] <thread : 139817795663616> retry connect to master
.....
getgrnam("root ") failed
修改 master.cf 中
execute user group
user terry terry
为
user root root
启动就报 getgrnam("root ") failed 跟用户名有关？
master，slave 都正常启动了，但是redis没有数据
==================rs_master.log
cmd = /var/log/mysql/mysql-bin.000002,107
,test.test ,
dump_file = /var/log/mysql/mysql-bin.000002
dump_num = 2
dump_pos = 107
filter_tables = ,test.test ,
ringbuf_susec = 10000
2014-03-16 21:03:17 [INFO] <thread : 140292578621184> open a new binlog = /var/log/mysql/mysql-bin.000002
2014-03-16 21:06:31 [DEBUG] <thread : 140292578621184>
========== event header =============
server id : 1
event type : 2
event length : 68
event next position : 175
dump position : 175
data length : 49
2014-03-16 21:06:31 [DEBUG] <thread : 140292578621184>
========== QUERY_EVENT ==============
2014-03-16 21:06:31 [DEBUG] <thread : 140292578621184>
database name : test
query sql : BEGIN
next position : 175
2014-03-16 21:06:31 [DEBUG] <thread : 140292578621184>
========== event header =============
server id : 1
event type : 19
event length : 46
event next position : 221
dump position : 221
data length : 27
2014-03-16 21:06:31 [DEBUG] <thread : 140292578621184>
========== TABLE_MAP_EVENT ==============
2014-03-16 21:06:31 [DEBUG] <thread : 140292578621184>
database : test
table : test
column num : 2
skip : 1
2014-03-16 21:06:31 [DEBUG] <thread : 140292578621184>
========== event header =============
server id : 1
event type : 23
event length : 40
event next position : 261
dump position : 261
data length : 21
2014-03-16 21:06:31 [DEBUG] <thread : 140292578621184>
========== WRITE_ROWS_EVENT ==============
2014-03-16 21:06:31 [DEBUG] <thread : 140292578621184>
========== event header =============
server id : 1
event type : 16
event length : 27
event next position : 288
dump position : 288
data length : 8
2014-03-16 21:06:31 [DEBUG] <thread : 140292578621184>
========== XID_EVENT ==============
2014-03-16 21:06:31 [DEBUG] <thread : 140292578621184>
tran : 0
tran id : 114
==================rs_slave.log
2014-03-16 21:03:17 [DEBUG] <thread : 140333885789952> slab class id= 7, chunk size = 1840, num = 2
2014-03-16 21:03:17 [DEBUG] <thread : 140333885789952> pool align init_size 2760
2014-03-16 21:03:17 [DEBUG] <thread : 140333885789952> slab class id= 8, chunk size = 2760, num = 1
2014-03-16 21:03:17 [DEBUG] <thread : 140333885789952> pool class size 40, clsid 0
2014-03-16 21:03:17 [DEBUG] <thread : 140333885789952> palloc free chunk num = 4
2014-03-16 21:03:17 [DEBUG] <thread : 140333885789952> pool class size 40, clsid 0
2014-03-16 21:03:17 [DEBUG] <thread : 140333885789952> palloc free chunk num = 3
2014-03-16 21:03:17 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
2014-03-16 21:03:47 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
2014-03-16 21:04:17 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
2014-03-16 21:04:47 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
2014-03-16 21:05:17 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
2014-03-16 21:05:47 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
2014-03-16 21:06:17 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
2014-03-16 21:06:47 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
2014-03-16 21:07:17 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
2014-03-16 21:07:47 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
2014-03-16 21:08:17 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
2014-03-16 21:08:47 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
2014-03-16 21:09:17 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
2014-03-16 21:09:47 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
2014-03-16 21:10:17 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
2014-03-16 21:10:47 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
2014-03-16 21:11:17 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
2014-03-16 21:11:47 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
2014-03-16 21:12:17 [INFO] <thread : 140333832079104> flush slave.info /var/log/mysql/mysql-bin.000002,107
对比了下前
#5
提交的问题，为什么我这的master日志：
query sql : BEGIN
我看他打印的日志是SQL，有哪里配置错了？
mysql> show global variables like '%binlog%';
+-----------------------------------------+----------------------+
| Variable_name | Value |
+-----------------------------------------+----------------------+
| binlog_cache_size | 32768 |
| binlog_direct_non_transactional_updates | OFF |
| binlog_format | ROW |
| binlog_stmt_cache_size | 32768 |
| innodb_locks_unsafe_for_binlog | OFF |
| max_binlog_cache_size | 18446744073709547520 |
| max_binlog_size | 104857600 |
| max_binlog_stmt_cache_size | 18446744073709547520 |
| sync_binlog | 0 |
+-----------------------------------------+----------------------+
是binlog_format 没有使用row？
发现个问题，字段长度必须固定？
如：
mysql> insert into test(col) values ('1234567890');
Query OK, 1 row affected (0.03 sec)
mysql> insert into test(col) values ('12');
Query OK, 1 row affected (0.07 sec)
在redis里面看到的数据是这样的：
127.0.0.1:6379> get test_1
"1234567890"
127.0.0.1:6379> get test_2
"1234567890"
test_2 的值为什么不是12 ，而是 "1234567890" ?
日志打印出来的是：
========== test ==========
id : 2
col : 12
还在维护更新的吗？
stack smashing detected
[ERROR] <thread : 140408291006272> rs_read() bin:x:1:1::/:/sbin/nologin
接收不到 master 的数据
slave 日志显示在不停地 retry connect to master
master 日志显示 get dumpcmd failed
slave 程序在 rs_io_thread.c
78 if(rs_recv_tmpbuf(si->recv_buf, si->svr_fd, &pack_len, 4) != RS_OK) {
79 goto retry;
80 }
收不到数据。
我的 slave 配置文件：
[core]
this is a master config file
user group
user root root
pid
pid /tmp/rs_slave.pid
daemon
daemon 1
log path
log /tmp/rs_slave.log
log debug level (3 DEBUG : 2, INFO : 1, ERR : 0)
log.level 3
debug level
debug.level BINLOG|ALLOC
[slave]
listen addr
listen.addr 127.0.0.1
listen port
listen.port 1919
slave.info path
slave.info /home/server/MySQL-Syncer/etc/slave.info
redis addr
redis.addr 127.0.0.1
redis port
redis.port 6379
slab init size
pool.initsize 100
slab max memory size (10MB)
pool.memsize 10485760
slab slab class grow factor
pool.factor 1.5
ring buffer num
ringbuf.num 10000
binlog save
binlog.save 10
binlog savesec
binlog.savesec 30
filter tables
filter.tables test.test
recv buffer length
recvbuf.size 10485760
server ring buffer empty sleep usec
svr.ringbuf.esusec 10000
client ring buffer empty sleep usec
cli.ringbuf.esusec 10000
slave.info 文件
/var/log/mysql/mysql-bin.000001,287
master 配置文件
[core]
this is a master config file
execute user group
user server server
pid file
pid /tmp/rs_master.pid
daemon
daemon 1
log path
log /tmp/rs_master.log
log error level (3 DEBUG : 2 INFO : 1 ERR : 0)
log.level 3
log debug level (CORE|ALLOC)
debug.level BINLOG
[master]
listen addr
listen.addr 127.0.0.1
listen port
listen.port 1919
binlog index path
binlog.index /var/log/mysql/mysql-bin.index
pool init size
pool.initsize 100
pool max memory size (10MB)
pool.memsize 10485760
pool slab class grow factor
pool.factor 1.5
ring buffer num
ringbuf.num 8000
send buf size
sendbuf.size 1024000
io buf size (128MB)
iobuf.size 10485760
#iobuf.size 1024000
current mysql server_id
server.id 123
max dump thread
dump.thread 36
请问这是哪里出问题了？
2014-03-05 16:43:17 [ERROR] <thread : 47362632485184> unknown event, 30
========== event header =============
server id : 1
event type : 30
event length : 57
event next position : 1329
dump position : 1329
data length : 38
2014-03-05 16:43:17 [ERROR] <thread : 47362632485184> unknown event, 30
2014-03-05 16:43:17 [INFO] <thread : 47362632485184> open a new binlog = /data/mysql/mysql-bin.000022
2014-03-05 16:43:17 [DEBUG] <thread : 47362632485184>
src/slave/rs_slave_info.c中n = rs_read(si->info_fd, si->dump_info, RS_SLAVE_INFO_STR_LEN);的si->dump_info在哪里初始化的？
如题，这里的si->dump_info我没有找到它的初始化的地方，该rs_read函数总是报错！！！
rs_master 和 rs_slave启动成功后，redis没有被同步
您好：
所有配置按照README.zh.md文档配置以及之前的issus解决出现的问题。
所有软件都在一台虚拟机上，包括mysql，redis，mysql-syncer
在成功启动了master 和 slave 后，进行了mysql数据库的test表的insert工作。
进入redis-cli查看
redis 127.0.0.1:6379> keys *
(empty list or set)
以下是日志
master日志如下：
2013-09-24 14:44:15 [DEBUG] <thread : 140176434669312>
========== XID_EVENT ==============
2013-09-24 14:44:15 [DEBUG] <thread : 140176434669312>
tran : 0
tran id : 24
2013-09-24 14:46:23 [DEBUG] <thread : 140176434669312>
========== event header =============
server id : 1
event type : 2
event length : 68
event next position : 1065
dump position : 1065
data length : 49
2013-09-24 14:46:23 [DEBUG] <thread : 140176434669312>
========== QUERY_EVENT ==============
2013-09-24 14:46:23 [DEBUG] <thread : 140176434669312>
database name : test
query sql : BEGIN
next position : 1065
2013-09-24 14:46:23 [DEBUG] <thread : 140176434669312>
========== event header =============
server id : 1
event type : 5
event length : 28
event next position : 1093
dump position : 1093
data length : 9
2013-09-24 14:46:23 [DEBUG] <thread : 140176434669312>
========== INTVAR_EVNET ==============
2013-09-24 14:46:23 [DEBUG] <thread : 140176434669312>
increment id : 8
2013-09-24 14:46:23 [DEBUG] <thread : 140176434669312>
========== event header =============
server id : 1
event type : 2
event length : 102
event next position : 1195
dump position : 1195
data length : 83
2013-09-24 14:46:23 [DEBUG] <thread : 140176434669312>
========== QUERY_EVENT ==============
2013-09-24 14:46:23 [DEBUG] <thread : 140176434669312>
database name : test
query sql : INSERT INTO test(col) VALUES('Mytest1')
next position : 1195
2013-09-24 14:46:23 [DEBUG] <thread : 140176434669312>
========== event header =============
server id : 1
event type : 16
event length : 27
event next position : 1222
dump position : 1222
data length : 8
2013-09-24 14:46:23 [DEBUG] <thread : 140176434669312>
========== XID_EVENT ==============
2013-09-24 14:46:23 [DEBUG] <thread : 140176434669312>
tran : 0
tran id : 56
slave日志提示一直flush slave.info。而位置就是最开始写入的位置。
日志如下：
2013-09-24 14:46:45 [INFO] <thread : 140119431694080> flush slave.info /var/lib/mysql/master_binlog.000002,106
2013-09-24 14:47:15 [INFO] <thread : 140119431694080> flush slave.info /var/lib/mysql/master_binlog.000002,106
2013-09-24 14:47:45 [INFO] <thread : 140119431694080> flush slave.info /var/lib/mysql/master_binlog.000002,106
2013-09-24 14:48:15 [INFO] <thread : 140119431694080> flush slave.info /var/lib/mysql/master_binlog.000002,106
2013-09-24 14:48:45 [INFO] <thread : 140119431694080> flush slave.info /var/lib/mysql/master_binlog.000002,106
2013-09-24 14:49:15 [INFO] <thread : 140119431694080> flush slave.info /var/lib/mysql/master_binlog.000002,106
2013-09-24 14:49:45 [INFO] <thread : 140119431694080> flush slave.info /var/lib/mysql/master_binlog.000002,106
2013-09-24 14:50:15 [INFO] <thread : 140119431694080> flush slave.info /var/lib/mysql/master_binlog.000002,106
2013-09-24 14:50:45 [INFO] <thread : 140119431694080> flush slave.info /var/lib/mysql/master_binlog.000002,106
2013-09-24 14:51:15 [INFO] <thread : 140119431694080> flush slave.info /var/lib/mysql/master_binlog.000002,106
2013-09-24 14:51:45 [INFO] <thread : 140119431694080> flush slave.info /var/lib/mysql/master_binlog.000002,106
2013-09-24 14:52:15 [INFO] <thread : 140119431694080> flush slave.info /var/lib/mysql/master_binlog.000002,106
2013-09-24 14:52:45 [INFO] <thread : 140119431694080> flush slave.info /var/lib/mysql/master_binlog.000002,106
烦请帮忙解决一下，谢谢
License
Kudos on the great work. What license is MYSQL-Syncer available under: MIT/GPL/BSD?
发现几个bug
1.event type我5.6的版本有30和31，但是和23，24相对应这个已经解决了
2.src/slave/rs_binlog_row.c:536,虽然我不知道为什么会触发这里，但是这里goto next_col;
明显会导致core dump
因为596行cmp += meta->meta_len;的meta还并没有赋值
暂时没法帮你改进，请作者修复下
配置master和slave
两个配置实在同一台机子上，还是需要主从服务器啊？目前在一台机子上试了，有master进程，但是slave进程start之后，没有进程相关信息，小白，谢谢
retry connect to master
mast slave 启动成功后，slave 日志里一直显示:
<thread : 140337872852736> retry connect to master
有两个地方不太清楚：
- 我看到slave和master的配置里都是监听1919端口，这是为什么？应该是不一样的吗，如果不一样的话，slave 又是从哪里知道master的端口。
- 本地telnet 127.0.0.1 1919 端口成功，但是master log一直显示：
[ERROR] <thread : 139641752966912> get dumpcmd failed
不太清楚他们之间发生了什么。。
为什么master.cf中监听端口为1919而不是mysql的3306端口？
启动时执行./rs.sh slave start 失败
我的操作系统 Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic x86_64) ，是台虚拟机。
Mysql是二进制安装5.5.16版本，
MySQL-Syncer的安装目录是/data/tool/MySQL-Syncer-master
执行./rs.sh master start 成功
执行./rs.sh slave start 失败
错误提示./rs.sh: line 23: /data/tool/MySQL-Syncer-master/objs/rs_slave: No such file or directory
在objs目录中，我只找到rs_master这个文件，没找到rs_slave，如何解决？
slave端si->dump_info初始值总是为空，请问si->dump_info初始值是什么？如何获得该初始值？
只能过滤到表？
只能支持到表？假如现在只需要某张表的几个字段数据复制过去呢？
2014-03-05 15:39:04 [ERROR] <thread : 47661750942016> get dumpcmd failed
2014-03-05 15:39:04 [ERROR] <thread : 47661750942016> get dumpcmd failed
这个是什么问题呢?
对于decimal类型 能支持同步吗
看了源码貌似没有现实 decimal 数据类型的转换,能否解析一下?
第一次全量导入redis
有全量把数据和结构从msyql导入redis的功能不
或者有没有别的方法，在第一次的时候全量导入redis，之后就syncer来同步
slave端rs_read() failed, error or empty报错问题
master端socket创建成功，处于listen状态，但是slave端总是报错--rs_read() failed, error or empty。
问题出在si->dump_info为空，从而导致n = rs_read(si->info_fd, si->dump_info, RS_SLAVE_INFO_STR_LEN);读取失败。
我想应该是master端没有获取到mysql的binlog，但我找不到master端拉取mysql binlog的程序，请问在哪里？？？
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.