go-mysql-org / go-mysql Goto Github PK

View Code? Open in Web Editor NEW

4.5K 4.5K 959.0 2.57 MB

a powerful mysql toolset with Go

License: MIT License

Makefile 0.24% Go 99.68% Shell 0.08%

go-mysql's People

Contributors

Stargazers

Watchers

Forkers

heartshare tobz cw2018 perepechaev erikstmartin no2key chenfeii huangzhiyong ikarishinjieva duzhanyuan jack1582 goeswind justintung fusionrsrch eddyzhou hank4work blalor yhyleo2011 wfxiang08 eaglechen maxid piaohai zhanglei linlams liuqian1990 holys tcbeliever ningoo windrainw is00hcw jameswei shlomi-noach ninefive gdey karmakaze puper gooops zsmj513 medvednikov ehalpern shuc324 wskuner shenhua19891109 lzpfmh hand-code dingjun84 wuli07101 uguarder hafeez3000 jmccarthy xuweili hanhui870 hinike ltd-beget yuanfeng0905 rubyfrea eddytrex 3manuek d0xin davars sputnik13 yodamaster marknewmail zhapuyu peterdeka mr8 xurenlu forestkeeper jekywong zengnotes yangql flybird119 wojtekzw takatoshiono gs02048 ericmort shengxianye yy1117 github shanmuganandh picago glycerine mixadior tyouno1 mixadior-casafari mission008 masteryconnect edric-shen tylerkovacs thinkdb zhq651 xxxxxheng iyidan fiefdx qizexi zhouruisong apokalyptik maple603 wtysksk zjh1943

go-mysql's Issues

gtid replication, how to tell the master our GTID_PURGED

Thanks for this nice library. It is super helpful.

I'm trying to start gtid based binlog replication, but my server is closing the connection before my (Go) slave can get started:

binlogstreamer.go:47: [error] close sync with err: ERROR 1236 (HY000): 
The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, 
but the master has purged binary logs containing GTIDs that the slave requires.

According to https://www.percona.com/blog/2013/02/08/how-to-createrestore-a-slave-using-gtid-replication-in-mysql-5-6/ I simply need to set GTID_PURGED on the (Go) slave, so that it gets communicated to the server.

I don't see any place in the Go api of go-mysql to set the GTID_PURGED value. Is there one? I'm not sure how the master is asking the slave; or maybe the slave is just passing along that info in some initialization message?

Why these channels buffered?

Is there are reason these channels are buffered?
https://github.com/siddontang/go-mysql/blob/master/replication/binlogstreamer.go#L68-L69

support travis

JSON deserialization

Hi. I've noticed you added JSON support. mysql-binlog-connector-java@master is missing a number of changes that were included in 0.5.2 that you might be interested in. This one in particular (shyiko/mysql-binlog-connector-java@881e9c7). Take a look. It might save you from a couple of bug reports. Cheers!

Extraneous parameter (or possible bug) in row_event.go decodeDecimal()

https://github.com/siddontang/go-mysql/blob/master/replication/row_event.go#L549
Sprintf format only has one value placeholder, seems meant for value not compFractional

use juju errors

Support for GTID-mode replication

using UTC for timestamp type

refer github/gh-ost#182

/cc @qiuyesuifeng

Use distributed key-value store for master info storage.

Maybe etcd or pd?

support DATETIME subsecond fraction in row_event

As per https://github.com/siddontang/go-mysql/blob/e1dd629909a09927b9d237cf91831ad82ae39146/replication/row_event.go#L658-L659, the replication library ignores the DATETIME fractional value. I'm wondering what is the reason for that, and whether this should not be supported.

Related: github/gh-ost#300

Add Mysql string type in schema

Please add string schema type
I mean

ta.Columns[index].MysqlType = columnType

in https://github.com/siddontang/go-mysql/blob/master/schema/schema.go#L59

BinlogSyncer.Close() will be blocked if BinlogStreamer exists

BinlogSyncer.wg will be Add(1) when creating BinlogStreamer. And BinlogSyncer.wg only will be decrease when an error comes to BinlogSyncer.onStream.
BinlogSyncer.Close() was waiting on wg.
So if I call BinlogSyncer.Close() when there is no error in BinlogSyncer.onStream, process will be blocked.

improve binlog backup, support resuming from last position

'invalid ok packet' error

Received when connecting using user with 'replication slave' privilege. The user was already being used in master -> slave1 replication. This lib was being used for slave1 -> other datastore replication. Creating a different user with replication slave privilege resolved the issue. Unclear why, but resolved.

server端预处理执行的接口问题

对于代理服务器，往往需要转发客户端请求给服务端，在预处理的执行函数HandleStmtExecute(context interface{}, query string, args []interface{}) (_Result, error)中，args的类型为interface的数组，在client端，预处理执行函数func (s *Stmt) Execute(args ...interface{}) (_Result, error)，args表示不定参数，用户在实现HandleStmtExecute时，很容易会将args []interface{}传递给args ...interface{}，这很容易引起接口的误用

Replication RowsEvent support

Hello. I have a use case where I want to read and process RowsEvents as a slave to a master MySQL DB. I had a project going that does this by tailing the binlog on the DB server itself, however, I would like to do it via the replication protocol instead. I noticed, in playing around with the replication package in the repository, that RowsEvents are just deserialized as GenericEvents. It only works with one version of MySQL currently, but I do have this library https://github.com/Granicus/mysql-binlog-go, which is capable of deserializing RowsEvents.

I want to fork this repository and add support for RowsEvents deserialization. My question to you is this: is it preferred that I create a more generic interface for my deserialization library and integrate it as a dependency here, or would it be better to attempt to fold the deserialization logic from the library into this library?

reading the binlog, is the gtid on the record available?

I can read from the binlog starting at a given point. I could count forward from there, but I think that gaps are allowed. So I think the records from the binlog must themselves be stamped with a GTID. But when I print out these records, I only see the old offset from non-gtid replication.

 ev.Header = &replication.EventHeader{Timestamp:0x58abc314, EventType:0x1e, ServerID:0x98352c, EventSize:0x673, LogPos:0x10081, Flags:0x0}

 ev = '=== WriteRowsEventV2 ===
Date: 2017-02-20 22:33:24
Log position: 65665
Event size: 1651
TableID: 72
Flags: 1
Column count: 58
Values:
--
...

Is the GTID hiding somewhere in the binary data, just not yet decoded?

wrap database/sql

StreamEvents encountered unexpected error: Err: runtime error: index out of range

Issue description

I try to migrate table using the tool gh-ost that uses go-sql-driver/mysql but each time I get the same error unexpected error: Err: runtime error: index out of range. Do you have any idea how to fix the issue?

Thank you

There is ticket in gh-ost

Error log

2016-10-24 00:43:18 INFO StreamEvents encountered unexpected error: Err: runtime error: index out of range
 Stack: goroutine 16 [running]:
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql.Pstack(0x0, 0x0)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql/util.go:18 +0x7b
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.(*BinlogSyncer).onStream.func1(0xc82000bf60, 0xc8200d80c0)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication/binlogsyncer.go:488 +0x66
panic(0x798ea0, 0xc82000c080)
        /usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:443 +0x4e9
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.decodeDecimal(0xc82042385e, 0x3, 0x5a2, 0x0, 0xa, 0x13, 0x5, 0x0, 0x0)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication/row_event.go:513 +0xcc7
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.(*RowsEvent).decodeValue(0xc82007dcc0, 0xc82042385e, 0x3, 0x5a2, 0xa00f6, 0x0,

2016-10-23 20:31:10 INFO StreamEvents encountered unexpected error: Err: runtime error: index out of range
 Stack: goroutine 13 [running]:
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql.Pstack(0x0, 0x0)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql/util.go:18 +0x7b
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.(*BinlogSyncer).onStream.func1(0xc82000b720, 0xc820112480)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication/binlogsyncer.go:488 +0x66
panic(0x798ea0, 0xc82000c080)
        /usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:443 +0x4e9
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.decodeDecimal(0xc82022a05e, 0x3, 0x5a2, 0x0, 0xa, 0x13, 0x5, 0x0, 0x0)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication/row_event.go:513 +0xcc7
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.(*RowsEvent).decodeValue(0xc8201c6d20, 0xc82022a05e, 0x3, 0x5a2, 0xa00f6, 0x0,

2016-10-14 22:36:32 INFO StreamEvents encountered unexpected error: Err: runtime error: index out of range
 Stack: goroutine 20 [running]:
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql.Pstack(0x0, 0x0)
  /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql/util.go:18 +0x7b
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.(*BinlogSyncer).onStream.func1(0xc820134700, 0xc820112480)
  /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication/binlogsyncer.go:488 +0x66
panic(0x798ea0, 0xc82000c090)
  /usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:443 +0x4e9
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.decodeDecimal(0xc82037d85e, 0x3, 0x5a2, 0x0, 0xa, 0x13, 0x5, 0x0, 0x0)
  /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication/row_event.go:513 +0xcc7
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.(*RowsEvent).decodeValue(0xc8201b6aa0, 0xc82037d85e, 0x3, 0x5a2, 0xa00f6, 0x0,

2016-10-20 04:30:17 INFO StreamEvents encountered unexpected error: Err: runtime error: index out of range
 Stack: goroutine 16 [running]:
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql.Pstack(0x0, 0x0)
  /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql/util.go:18 +0x7b
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.(*BinlogSyncer).onStream.func1(0xc820152100, 0xc8200d60c0)
  /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication/binlogsyncer.go:488 +0x66
panic(0x798ea0, 0xc82000c080)
  /usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:443 +0x4e9
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.decodeDecimal(0xc82030cc5e, 0x3, 0x5a2, 0x0, 0xa, 0x13, 0x5, 0x0, 0x0)
  /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication/row_event.go:513 +0xcc7
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.(*RowsEvent).decodeValue(0xc8201a59a0, 0xc82030cc5e, 0x3, 0x5a2, 0xa00f6, 0x0,

2016-10-13 20:13:24 INFO StreamEvents encountered unexpected error: Err: runtime error: index out of range
 Stack: goroutine 20 [running]:
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql.Pstack(0x0, 0x0)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql/util.go:18 +0x7b
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.(*BinlogSyncer).onStream.func1(0xc8200fd420, 0xc82013a000)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication/binlogsyncer.go:488 +0x66
panic(0x798ea0, 0xc82000c090)
        /usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:443 +0x4e9
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.decodeDecimal(0xc8203d78c7, 0xaa, 0x539, 0x0, 0xa, 0x13, 0x5, 0x0, 0x0)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication/row_event.go:513 +0xcc7
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.(*RowsEvent).decodeValue(0xc82022b180, 0xc8203d78c7, 0xaa, 0x539, 0xa00f6, 0x0

2016-10-13 08:36:37 INFO StreamEvents encountered unexpected error: Err: runtime error: index out of range
 Stack: goroutine 137735 [running]:
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql.Pstack(0x0, 0x0)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/mysql/util.go:18 +0x7b
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.(*BinlogSyncer).onStream.func1(0xc820195d80, 0xc820230840)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication/binlogsyncer.go:488 +0x66
panic(0x797b60, 0xc82000c090)
        /usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:443 +0x4e9
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.decodeDecimal(0xc82019e05e, 0x3, 0x5a2, 0x0, 0xa, 0x13, 0x5, 0x0, 0x0)
        /Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication/row_event.go:513 +0xcc7
github.com/github/gh-ost/vendor/github.com/siddontang/go-mysql/replication.(*RowsEvent).decodeValue(0xc8201cd2c0, 0xc82019e05e, 0x3, 0x5a2, 0xa00f6, 0

Configuration

MySQL 5.7.11 on AWS RDS
Ubuntu 14.04

sync.go:50 [Error] handle rows event error connection was bad

When handleRowsEvent() return error, it caused startSyncBinlog() to stop running. Can we ignore the error?

select column meta for table map event

I think this can improve #28.

@karmakaze

When registering as a slave we should be using the host's hostname, not the server's hostname.

We have a bug in writeRegisterSlaveCommand (https://github.com/siddontang/go-mysql/blob/master/replication/binlogsyncer.go#L414-L417) where we are using the host name of the server we are connecting to instead of the host we are on.

This makes is confusing when looking at the show slave status output on the master.

We should just use os.Hostname() here.

Support for POINT GIS datatype

Hi,
Could you please add support for POINT datatype (very much needed by users of GIS features of mysql).

Thank you.

invalid table id %d, no correspond table map event

As TableMapEvent and WriteRowsEventV2 are non-atomic, how can program work when TableMapEvent is processed but program restart, which makes goroutine named "onStream()" exit?

canal什么时候能支持alter table format at runtime.

[ask] How to get Mysql position?

In previous version, we have syncer.ExecuteSql("SHOW MASTER STATUS") to get Mysql position.
How to do that now?

use vendor or glide instead of origin godep

panic decoding packet

I'm in pretty deep over my head here, but in attempting to write a simple program that dumps replication events from one of my MySQL 5.6.20 servers (based on the example in the README), I'm getting a panic in parser.go. Here's a test that reproduces the problem, but I had to build up parser.format and parser.tables from the parser state just before my program crashes. It's possible that earlier events are the true cause of the problem, but I do feel like an index bounds check in RowsEvent.decodeRows would help avoid the panic.

Here's parser_test.go:

package replication

import (
    "testing"
)

func TestIndexOutOfRange(t *testing.T) {
    parser := NewBinlogParser()

    parser.format = &FormatDescriptionEvent{
        Version:                0x4,
        ServerVersion:          []uint8{0x35, 0x2e, 0x36, 0x2e, 0x32, 0x30, 0x2d, 0x6c, 0x6f, 0x67, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
        CreateTimestamp:        0x0,
        EventHeaderLength:      0x13,
        EventTypeHeaderLengths: []uint8{0x38, 0xd, 0x0, 0x8, 0x0, 0x12, 0x0, 0x4, 0x4, 0x4, 0x4, 0x12, 0x0, 0x0, 0x5c, 0x0, 0x4, 0x1a, 0x8, 0x0, 0x0, 0x0, 0x8, 0x8, 0x8, 0x2, 0x0, 0x0, 0x0, 0xa, 0xa, 0xa, 0x19, 0x19, 0x0},
        ChecksumAlgorithm:      0x1,
    }

    parser.tables = map[uint64]*TableMapEvent{
        0x3043b:  &TableMapEvent{tableIDSize:6, TableID:0x3043b, Flags:0x1, Schema:[]uint8{0x73, 0x65, 0x69, 0x75, 0x6d, 0x61, 0x73, 0x74, 0x65, 0x72}, Table:[]uint8{0x61, 0x70, 0x70, 0x5f, 0x63, 0x72, 0x6f, 0x6e}, ColumnCount:0x15, ColumnType:[]uint8{0x3, 0xf, 0xc, 0xc, 0xf, 0x3, 0xc, 0x3, 0xfc, 0xf, 0x1, 0xfe, 0x2, 0xc, 0xf, 0xf, 0xc, 0xf, 0xf, 0x3, 0xf}, ColumnMeta:[]uint16{0x0, 0x180, 0x0, 0x0, 0x2fd, 0x0, 0x0, 0x0, 0x2, 0x180, 0x0, 0xfe78, 0x0, 0x0, 0x180, 0x180, 0x0, 0x180, 0x180, 0x0, 0x2fd}, NullBitmap:[]uint8{0xf8, 0xfb, 0x17}},
        0x30453:  &TableMapEvent{tableIDSize:6, TableID:0x30453, Flags:0x1, Schema:[]uint8{0x73, 0x65, 0x69, 0x75, 0x6d, 0x61, 0x73, 0x74, 0x65, 0x72}, Table:[]uint8{0x73, 0x74, 0x67, 0x5f, 0x73, 0x69, 0x67, 0x6e, 0x75, 0x70}, ColumnCount:0x36, ColumnType:[]uint8{0x3, 0x3, 0x3, 0x3, 0x3, 0xf, 0xf, 0x8, 0x3, 0x3, 0x3, 0xf, 0xf, 0x1, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf, 0xfe, 0x12, 0xf, 0xf, 0xf, 0xf6, 0x1, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf, 0xfe, 0xf6, 0x12, 0x3, 0xf, 0xf, 0x1, 0x1, 0x12, 0xf, 0xf, 0xf, 0xf, 0x3, 0xf, 0x3}, ColumnMeta:[]uint16{0x0, 0x0, 0x0, 0x0, 0x0, 0x2fd, 0x12c, 0x0, 0x0, 0x0, 0x0, 0x180, 0x180, 0x0, 0x30, 0x180, 0x180, 0x180, 0x30, 0xc0, 0xfe03, 0x0, 0x180, 0x180, 0x180, 0xc02, 0x0, 0x5a, 0x5a, 0x5a, 0x5a, 0x2fd, 0x2fd, 0x2fd, 0xc0, 0x12c, 0x30, 0xc, 0xfe06, 0xb02, 0x0, 0x0, 0x180, 0x180, 0x0, 0x0, 0x0, 0x180, 0x180, 0x2d, 0x2fd, 0x0, 0x2fd, 0x0}, NullBitmap:[]uint8{0xee, 0xdf, 0xff, 0xff, 0xff, 0xff, 0x17}},
        0x30504:  &TableMapEvent{tableIDSize:6, TableID:0x30504, Flags:0x1, Schema:[]uint8{0x73, 0x65, 0x69, 0x75, 0x6d, 0x61, 0x73, 0x74, 0x65, 0x72}, Table:[]uint8{0x6c, 0x6f, 0x67, 0x5f, 0x73, 0x74, 0x67, 0x5f, 0x73, 0x69, 0x67, 0x6e, 0x75, 0x70}, ColumnCount:0x13, ColumnType:[]uint8{0x3, 0x3, 0x3, 0x3, 0x3, 0x3, 0x3, 0x3, 0x3, 0x3, 0x3, 0xf, 0xc, 0xc, 0xc, 0xf, 0xf, 0x3, 0xf}, ColumnMeta:[]uint16{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x180, 0x0, 0x0, 0x0, 0x180, 0x180, 0x0, 0x2fd}, NullBitmap:[]uint8{0x6, 0xfb, 0x5}},
        0x30450:  &TableMapEvent{tableIDSize:6, TableID:0x30450, Flags:0x1, Schema:[]uint8{0x73, 0x65, 0x69, 0x75, 0x6d, 0x61, 0x73, 0x74, 0x65, 0x72}, Table:[]uint8{0x73, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74}, ColumnCount:0x16, ColumnType:[]uint8{0x3, 0xfc, 0xc, 0x3, 0xc, 0xf, 0x3, 0xf, 0xc, 0xf, 0xf, 0xf, 0xf, 0x3, 0xc, 0xf, 0xf, 0xf, 0xf, 0x3, 0x3, 0xf}, ColumnMeta:[]uint16{0x0, 0x2, 0x0, 0x0, 0x0, 0x2d, 0x0, 0x180, 0x0, 0x180, 0x180, 0x2fd, 0x2d, 0x0, 0x0, 0x180, 0x180, 0x2fd, 0x2d, 0x0, 0x0, 0x2fd}, NullBitmap:[]uint8{0xfe, 0xff, 0x2f}},
        0x305bb:  &TableMapEvent{tableIDSize:6, TableID:0x305bb, Flags:0x1, Schema:[]uint8{0x79, 0x6d, 0x63, 0x61, 0x63, 0x68, 0x67, 0x6f}, Table:[]uint8{0x72, 0x65, 0x64, 0x69, 0x72, 0x65, 0x63, 0x74, 0x5f, 0x6c, 0x6f, 0x67}, ColumnCount:0x11, ColumnType:[]uint8{0x3, 0x3, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf, 0xc, 0xf, 0xf, 0xc, 0xf, 0xf, 0x3, 0xf}, ColumnMeta:[]uint16{0x0, 0x0, 0x2fd, 0x12c, 0x2fd, 0x2fd, 0x2d, 0x12c, 0x2fd, 0x0, 0x180, 0x180, 0x0, 0x180, 0x180, 0x0, 0x2fd}, NullBitmap:[]uint8{0xfe, 0x7f, 0x1}},
        0x16c36b: &TableMapEvent{tableIDSize:6, TableID:0x16c36b, Flags:0x1, Schema:[]uint8{0x61, 0x63, 0x70}, Table:[]uint8{0x73, 0x74, 0x67, 0x5f, 0x6d, 0x61, 0x69, 0x6c, 0x69, 0x6e, 0x67, 0x5f, 0x72, 0x65, 0x63, 0x69, 0x70, 0x69, 0x65, 0x6e, 0x74, 0x5f, 0x63, 0x6c, 0x69, 0x63, 0x6b, 0x32}, ColumnCount:0xe, ColumnType:[]uint8{0x8, 0x8, 0x3, 0x3, 0x2, 0x2, 0xf, 0x12, 0xf, 0xf, 0x12, 0xf, 0xf, 0xf}, ColumnMeta:[]uint16{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2d, 0x0, 0x180, 0x180, 0x0, 0x180, 0x180, 0x2fd}, NullBitmap:[]uint8{0xba, 0x3f}},
        0x16c368: &TableMapEvent{tableIDSize:6, TableID:0x16c368, Flags:0x1, Schema:[]uint8{0x73, 0x65, 0x69, 0x75, 0x6d, 0x61, 0x73, 0x74, 0x65, 0x72}, Table:[]uint8{0x73, 0x74, 0x67, 0x5f, 0x6d, 0x61, 0x69, 0x6c, 0x69, 0x6e, 0x67, 0x5f, 0x72, 0x65, 0x63, 0x69, 0x70, 0x69, 0x65, 0x6e, 0x74, 0x5f, 0x63, 0x6c, 0x69, 0x63, 0x6b, 0x32}, ColumnCount:0xe, ColumnType:[]uint8{0x8, 0x8, 0x3, 0x3, 0x2, 0x2, 0xf, 0x12, 0xf, 0xf, 0x12, 0xf, 0xf, 0xf}, ColumnMeta:[]uint16{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2d, 0x0, 0x180, 0x180, 0x0, 0x180, 0x180, 0x2fd}, NullBitmap:[]uint8{0xba, 0x3f}},
        0x3045a:  &TableMapEvent{tableIDSize:6, TableID:0x3045a, Flags:0x1, Schema:[]uint8{0x73, 0x65, 0x69, 0x75, 0x6d, 0x61, 0x73, 0x74, 0x65, 0x72}, Table:[]uint8{0x63, 0x6f, 0x6e, 0x73}, ColumnCount:0x1e, ColumnType:[]uint8{0x3, 0x3, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf, 0xfe, 0x12, 0xf, 0xf, 0xf, 0xf6, 0xf, 0xf, 0xf, 0xf, 0x1, 0x1, 0x1, 0x12, 0xf, 0xf, 0x12, 0xf, 0xf, 0x3, 0xf, 0x1}, ColumnMeta:[]uint16{0x0, 0x0, 0x30, 0x180, 0x180, 0x180, 0x30, 0xc0, 0xfe03, 0x0, 0x180, 0x180, 0x180, 0xc02, 0x180, 0x180, 0x180, 0x180, 0x0, 0x0, 0x0, 0x0, 0x180, 0x180, 0x0, 0x180, 0x180, 0x0, 0x2fd, 0x0}, NullBitmap:[]uint8{0xfc, 0xff, 0xe3, 0x37}},
        0x3045f:  &TableMapEvent{tableIDSize:6, TableID:0x3045f, Flags:0x1, Schema:[]uint8{0x73, 0x65, 0x69, 0x75, 0x6d, 0x61, 0x73, 0x74, 0x65, 0x72}, Table:[]uint8{0x63, 0x6f, 0x6e, 0x73, 0x5f, 0x61, 0x64, 0x64, 0x72}, ColumnCount:0x19, ColumnType:[]uint8{0x3, 0x3, 0x3, 0x1, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf, 0xfe, 0x3, 0xc, 0x1, 0xc, 0xf, 0xf, 0xc, 0xf, 0xf, 0x3, 0xf, 0x4, 0x4}, ColumnMeta:[]uint16{0x0, 0x0, 0x0, 0x0, 0x2fd, 0x2fd, 0x2fd, 0xc0, 0x12c, 0x30, 0xc, 0xfe06, 0x0, 0x0, 0x0, 0x0, 0x180, 0x180, 0x0, 0x180, 0x180, 0x0, 0x2fd, 0x4, 0x4}, NullBitmap:[]uint8{0xf0, 0xef, 0x5f, 0x0}},
        0x3065f:  &TableMapEvent{tableIDSize:6, TableID:0x3065f, Flags:0x1, Schema:[]uint8{0x73, 0x65, 0x69, 0x75, 0x6d, 0x61, 0x73, 0x74, 0x65, 0x72}, Table:[]uint8{0x63, 0x6f, 0x6e, 0x73, 0x5f, 0x61, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x73, 0x70, 0x65, 0x61, 0x6b, 0x6f, 0x75, 0x74, 0x5f, 0x6c, 0x65, 0x74, 0x74, 0x65, 0x72}, ColumnCount:0xd, ColumnType:[]uint8{0x3, 0x3, 0x3, 0x3, 0x1, 0x12, 0xf, 0xf, 0x12, 0xf, 0xf, 0x3, 0xf}, ColumnMeta:[]uint16{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x180, 0x180, 0x0, 0x180, 0x180, 0x0, 0x2fd}, NullBitmap:[]uint8{0xe0, 0x17}},
    }

    _, err := parser.parse([]byte{
        /* 0x00, */ 0xc1, 0x86, 0x8e, 0x55, 0x1e, 0xa5, 0x14, 0x80, 0xa, 0x55, 0x0, 0x0, 0x0, 0x7, 0xc,
        0xbf, 0xe, 0x0, 0x0, 0x5f, 0x6, 0x3, 0x0, 0x0, 0x0, 0x1, 0x0, 0x2, 0x0, 0xd, 0xff,
        0x0, 0x0, 0x19, 0x63, 0x7, 0x0, 0xca, 0x61, 0x5, 0x0, 0x5e, 0xf7, 0xc, 0x0, 0xf5, 0x7,
        0x0, 0x0, 0x1, 0x99, 0x96, 0x76, 0x74, 0xdd, 0x10, 0x0, 0x73, 0x69, 0x67, 0x6e, 0x75, 0x70,
        0x5f, 0x64, 0x62, 0x5f, 0x73, 0x63, 0x72, 0x69, 0x70, 0x74, 0x6, 0x0, 0x73, 0x79, 0x73, 0x74,
        0x65, 0x6d, 0xb1, 0x3c, 0x38, 0xcb,
    })

    if err != nil {
        t.Errorf("error: %+v", err)
    }
}

And the test output:

=== RUN TestIndexOutOfRange
--- FAIL: TestIndexOutOfRange (0.00s)
panic: runtime error: index out of range [recovered]
    panic: runtime error: index out of range

goroutine 5 [running]:
testing.func·006()
    /usr/local/Cellar/go/1.4.2/libexec/src/testing/testing.go:441 +0x181
github.com/siddontang/go-mysql/replication.(*RowsEvent).decodeRows(0xc208090140, 0xc208052320, 0x31, 0x35, 0xc2080900a0, 0xc20805231e, 0x2, 0x37, 0xc208090140, 0x0, ...)
    /tmp/gopathtmp/src/github.com/siddontang/go-mysql/replication/row_event.go:292 +0x60c
github.com/siddontang/go-mysql/replication.(*RowsEvent).Decode(0xc208090140, 0xc208052313, 0x3e, 0x42, 0x0, 0x0)
    /tmp/gopathtmp/src/github.com/siddontang/go-mysql/replication/row_event.go:261 +0x6a6
github.com/siddontang/go-mysql/replication.(*BinlogParser).parseEvent(0xc20801e260, 0xc20801e400, 0xc208052313, 0x3e, 0x42, 0x0, 0x0, 0x0, 0x0)
    /tmp/gopathtmp/src/github.com/siddontang/go-mysql/replication/parser.go:188 +0x12f
github.com/siddontang/go-mysql/replication.(*BinlogParser).parse(0xc20801e260, 0xc208052313, 0x42, 0x42, 0x0, 0x0, 0x0)
    /tmp/gopathtmp/src/github.com/siddontang/go-mysql/replication/parser.go:223 +0x33b
github.com/siddontang/go-mysql/replication.TestIndexOutOfRange(0xc20808e000)
    /tmp/gopathtmp/src/github.com/siddontang/go-mysql/replication/parser_test.go:42 +0x2434
testing.tRunner(0xc20808e000, 0x4350a0)
    /usr/local/Cellar/go/1.4.2/libexec/src/testing/testing.go:447 +0xbf
created by testing.RunTests
    /usr/local/Cellar/go/1.4.2/libexec/src/testing/testing.go:555 +0xa8b

[Error] canal start sync binlog err: invalid stream header [fe]

canal.go:145 canal start sync binlog err: invalid stream header [fe]

then，canal sync is stop

date time fields are treated as strings

when pushing data in to elastic search it's helpful to know what fields are date time type so they can be inserted as date fields

add syncer lib

Abstract some go-mysql-elasticsearch codes to build up a common syncer lib, not only for elasticsearch but also for others, like redis.

find a decimal package to support mysql decimal type really.

Replication may be broken when meets corrupted binlog

Handling 'invalid data size' errors

Getting this error, unsure what the event is: invalid data size 57 in event WriteRowsEventV0, less event length 3215523821

If this is a protocol error, what's the best way to capture the data that's causing it to fail? I got there with #12, but that was painful. :-) How do I continue from this or any other error where I'd like to skip to the next event? There appears to be no way to retrieve the current position.

support json type

Refer github/gh-ost#268
or #83

Can add a handshake method for Server handler Interface

github.com/siddontang/go-mysql/server

Now only support a single user.
Can determine the user name and password authentication in the handshake phase to make multi user connectio

replication block when network down

when the network is down (such as unplug the network cable) and recover after, the replication client will block at ReadPacket, this is the info of pprof:

goroutine 19 [IO wait, 14 minutes]:
net.runtime_pollWait(0x7fe48c176b70, 0x72, 0xc82177e000)
    /home/cdyf/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82176a760, 0x72, 0x0, 0x0)
    /home/cdyf/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82176a760, 0x0, 0x0)
    /home/cdyf/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82176a700, 0xc82177e000, 0x1000, 0x1000, 0x0, 0x7fe48c171050, 0xc82000a0f8)
    /home/cdyf/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc821768120, 0xc82177e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/cdyf/go/src/net/net.go:172 +0xe4
bufio.(*Reader).fill(0xc8217783c0)
    /home/cdyf/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc8217783c0, 0xc821c66148, 0x4, 0x4, 0x8, 0x0, 0x0)
    /home/cdyf/go/src/bufio/bufio.go:207 +0x260
io.ReadAtLeast(0x7fe48c131078, 0xc8217783c0, 0xc821c66148, 0x4, 0x4, 0x4, 0x0, 0x0, 0x0)
    /home/cdyf/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fe48c131078, 0xc8217783c0, 0xc821c66148, 0x4, 0x4, 0x20, 0x0, 0x0)
    /home/cdyf/go/src/io/io.go:315 +0x62
mysql_byroad/vendor/github.com/siddontang/go-mysql/packet.(*Conn).ReadPacketTo(0xc82101e8c0, 0x7fe48c175d78, 0xc821a388c0, 0x0, 0x0)
    /home/cdyf/goworkspace/src/mysql_byroad/vendor/github.com/siddontang/go-mysql/packet/conn.go:81 +0xf1
mysql_byroad/vendor/github.com/siddontang/go-mysql/packet.(*Conn).ReadPacket(0xc82101e8c0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/cdyf/goworkspace/src/mysql_byroad/vendor/github.com/siddontang/go-mysql/packet/conn.go:35 +0x92
mysql_byroad/vendor/github.com/siddontang/go-mysql/replication.(*BinlogSyncer).onStream(0xc821717480, 0xc82101eca0)
    /home/cdyf/goworkspace/src/mysql_byroad/vendor/github.com/siddontang/go-mysql/replication/binlogsyncer.go:479 +0xa8
created by mysql_byroad/vendor/github.com/siddontang/go-mysql/replication.(*BinlogSyncer).startDumpStream
    /home/cdyf/goworkspace/src/mysql_byroad/vendor/github.com/siddontang/go-mysql/replication/binlogsyncer.go:220 +0x111

I think this is because mysql server closed the connection and replication client didn't receive any close data because of bad network, so it blocked at reading data from this connection. Is there any solution for this condition?

add event handler instead of only row handler.

Maybe like this

type EventHandler interface {
    // Handle Event, if return ErrHandleInterrupted, canal will
    // stop the sync
    Do(e *Event) error
    String() string
}

Replication EOF handling

Greetings - it appears that EOF handling is missing from the replication code. https://github.com/siddontang/go-mysql/blob/master/replication/binlogsyncer.go#L500 will close the stream with an error when a 0xFE packet header is detected. Are there any plans to add support for EOF handling? What should a client do when the syncer reaches the end of the stream, but wants to keep reading? Reopen the stream?

add ping routine cause panic

I add a new routine to check connection like this:

panic info:

2016/11/28 20:19:02 binlogsyncer.go:514: [error] invalid sequence 1 != 2
panic: runtime error: slice bounds out of range

goroutine 54 [running]:
panic(0x3ce180, 0xc42000c130)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
bufio.(*Reader).Read(0xc4211fe7e0, 0xc4204bc394, 0x4, 0x4, 0x4, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:216 +0x2a8
io.ReadAtLeast(0x611c40, 0xc4211fe7e0, 0xc4204bc394, 0x4, 0x4, 0x4, 0x3b4200, 0x1, 0xc4204bc394)
/usr/local/go/src/io/io.go:307 +0xa4
io.ReadFull(0x611c40, 0xc4211fe7e0, 0xc4204bc394, 0x4, 0x4, 0xc4201be800, 0x715000, 0x0)
/usr/local/go/src/io/io.go:325 +0x58
canal/libs/github.com/siddontang/go-mysql/packet.(*Conn).ReadPacketTo(0xc42278d0c0, 0x611d00, 0xc42171fc00, 0x5, 0x0)
/Users/liangkai/Code/gopath/src/canal/libs/github.com/siddontang/go-mysql/packet/conn.go:81 +0x89
canal/libs/github.com/siddontang/go-mysql/packet.(*Conn).ReadPacket(0xc42278d0c0, 0x0, 0x0, 0xc42003bee0, 0x2c60e4, 0xc4201d0020)
/Users/liangkai/Code/gopath/src/canal/libs/github.com/siddontang/go-mysql/packet/conn.go:35 +0x7b
canal/libs/github.com/siddontang/go-mysql/client.(*Conn).readOK(0xc42285c070, 0xc42003bf0e, 0x0, 0x0)
/Users/liangkai/Code/gopath/src/canal/libs/github.com/siddontang/go-mysql/client/resp.go:85 +0x5b
canal/libs/github.com/siddontang/go-mysql/client.(*Conn).Ping(0xc42285c070, 0x14, 0xc42003bf88)
/Users/liangkai/Code/gopath/src/canal/libs/github.com/siddontang/go-mysql/client/conn.go:97 +0x7d
canal/libs/github.com/siddontang/go-mysql/replication.(*BinlogSyncer).ping(0xc4201d4880)
/Users/liangkai/Code/gopath/src/canal/libs/github.com/siddontang/go-mysql/replication/binlogsyncer.go:250 +0x4e
created by canal/libs/github.com/siddontang/go-mysql/replication.(*BinlogSyncer).startDumpStream
/Users/liangkai/Code/gopath/src/canal/libs/github.com/siddontang/go-mysql/replication/binlogsyncer.go:242 +0x11c

Is there something wrong?

use go-fuzz for binlog data parsing

Selects with 0 rows result in a crash

Hi,

When I run a select that doesn't return any rows, I get a panic:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x693de]

goroutine 12 [running]:
github.com/siddontang/go-mysql/mysql.(*Field).Dump(0x0, 0x0, 0x0, 0x0)
    /gocode/src/github.com/siddontang/go-mysql/mysql/field.go:121 +0x4e

Authentication handler

Authentication should be exposed as a handler that way more than a single username/password combination could be supported.

Canal output file `master.info` will lost binlog file name.

full bin_name was 'mysqld-bin.001957'

use docopt to parse flag easily

https://github.com/docopt/docopt.go

replication:index out of range error

在运行的过程中，对于某一binlog位置，会报index out of range 错误，之前有邮件询问过，经过分析是发现RowEvent数据中最后多了一个字节，之后看了python-replication的代码，发现在https://github.com/noplay/python-mysql-replication/blob/master/pymysqlreplication/row_event.py#L391 判断是否结束是通过 while self.packet.read_bytes + 1 < self.event_size 来判断的，而你的代码https://github.com/siddontang/go-mysql/blob/master/replication/row_event.go#L260 是for pos < len(data) 来判断是否结束，因此如果最后多了一个字节，会使得循环继续下去而报错，在改为for pos+1 < len(data) 之后能够顺利的解析，不知道这样是否正确，希望能够解答下。

Replication, binlogreader: "connection was bad"

While using binlogreader, as in:

... = binlogReader.binlogSyncer.RegisterSlave(connectionConfig.Key.Hostname, uint16(connectionConfig.Key.Port), connectionConfig.User, connectionConfig.Password)
...
<time passes>
...
ev, err := this.binlogStreamer.GetEvent()

I get a non nil err, which says "connection was bad"

This happens when enough time passes for the connection to invalidate. I'm trying to figure out if there is support for the equivalent MASTER_CONNECT_RETRY, MASTER_RETRY_COUNT, such that I would be able to configure the connection to retry as much as I want, with as small interval that I want.

I haven't figured this out yet -- is there such support?

Kind regards

Getting unsigned values for INT column

Set up to use row-format replication from MySQL 5.6.5, I'm getting large positive values in place of small negative ones for a column defined as INT which should be 32-bit signed. Here's a part of the debug log:

2016/01/07 20:04:27 [DEBUG] === UpdateRowsEventV2 ===
Date: 2016-01-07 15:48:58
Log position: 1007
Event size: 72
TableID: 74
Flags: 1
Column count: 4
Values:
--
0:566
1:2379
2:4294967295
3:"2016-01-07 15:44:32"
--
0:566
1:2379
2:4294967294
3:"2016-01-07 15:44:32"

The column '2' value was changed from -1 (received as 4294967295) to -2 (received as 4294967294).
I looked at the TableMapEvent for this table and didn't see anywhere that an unsigned 'meta' bit was set.

I even changed the schema of column '1' to be unsigned and I still didn't see any unsigned 'meta' bit in the debug output. There's a lot of binary data and it's possible I missed it, but the first case of an INT field -1 => -2 being reported as 4294967295 => 4294967294 stands.

add configuration to ignore mysqldump.

Dump fails if field value contains INSERT INTO statement

Dump fails if field value contains INSERT INTO statement, like INSERT INTO `table` VALUES ('');

2016/11/18 11:23:50 dump.go:32: [error] get schema.task` VALUES ('2QhUaRNzRUBP6yUH','TRaScZha',1474954092309183,3,'2QhUaRNz','2QhUaRNz','','','2QhUaRNznpZYD3jt',0,1,'','INSERT INTO `table information err: ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'table' at line 1

valuesExp in dump/parser.go should use non greedy capture

// origin
valuesExp = regexp.MustCompile("^INSERT INTO `(.+)` VALUES \\((.+)\\);$")

// fixed
valuesExp = regexp.MustCompile("^INSERT INTO `(.+?)` VALUES \\((.+)\\);$")