文档介绍，说在这里的
https://segmentfault.com/a/1190000008169097

Syncer version:
Release Version: v1.0.0-71-ge8f6868
Git Commit Hash: e8f68685fbcb79fb4f9a1b4c271b9aa8cbdcc728
Git Branch: master
UTC Build Time: 2019-08-09 10:08:53
Go Version: go version go1.12 linux/amd64

出现错误：
2019/08/17 19:49:40 syncer.go:489: [fatal] [error query event][error type]execution error [sql]USE drp; ALTER TABLE drp.bill_index MODIFY COLUM N print_flags bit(1) DEFAULT b'0' NOT NULL COMMENT '未用。配送打印'; [schema]drp [table]bill_index [current pos](logbin.000086, 1454952) [next pos] (logbin.000086, 1455180) [current gtid set] [next gtid set] [error message]/home/jenkins/workspace/build_tidb_enterprise_tools_master/go/src/git hub.com/pingcap/tidb-enterprise-tools/syncer/db.go:122: exec sqls[[USE drp; ALTER TABLE drp.bill_index MODIFY COLUMN print_flags bit(1) DEFAU LT b'0' NOT NULL COMMENT '未用。配送打印';]] failed, err:Error 1105: unsupported modify column unsigned false not match origin true

[tidb@localhost ent]$ cat syncer.meta
binlog-name = "logbin.000086"
binlog-pos = 1365129
binlog-gtid = ""

请问这种情况下我要如何恢复？
如果meta中的位置是1454952，那我只要手工执行DDL操作，然后将meta位置调整到DDL后面，也就是1455180。但现在我这么做的话，前面的一些更新就会丢失了吧？

Feature Request

Is your feature request related to a problem? Please describe:

DDL succeed to execute when longtext is contained using loader.
TiDB-Lightning can succed to execute when longtext is contained, but exit with 
"Error: db schema failed to init : Error 1074: Column length too big for column 'body' (max = 16383); use BLOB or TEXT instead"

Describe the feature you'd like:

Both of these tools has the same result for longtext type.

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
   If possible, provide a recipe for reproducing the error.

【pump问题】
部署了3个pump，但是tidb有的时候就突然挂了，并提示如下错误，实际上3个pump运行良好

2019/01/16 09:24:14.614 2pc.go:681: [error] failed to write binlog: [global:3]critical error no avaliable pump to write binlog
2019/01/16 09:24:34.517 server.go:271: [error] listener stopped, waiting for manual kill.

【drainer问题】
drainer启动过程中报如下错误并退出，但是tidb里并没有sbtest5这个表，之前是存在的，后来删掉了

2019/01/15 23:39:31 schema.go:373: ESC[0;37m[info] get unknown ddl type add indexESC[0m
2019/01/15 23:39:31 server.go:248: ESC[0;31m[error] syncer exited, error /home/jenkins/workspace/build_tidb_binlog_2.1/go/src/github.com/pingcap/tidb-binlog/drainer/schema.go:178: table sbtest5(157) not found
/home/jenkins/workspace/build_tidb_binlog_2.1/go/src/github.com/pingcap/tidb-binlog/drainer/schema.go:390: 
/home/jenkins/workspace/build_tidb_binlog_2.1/go/src/github.com/pingcap/tidb-binlog/drainer/schema.go:233: 
/home/jenkins/workspace/build_tidb_binlog_2.1/go/src/github.com/pingcap/tidb-binlog/drainer/syncer.go:468: 
/home/jenkins/workspace/build_tidb_binlog_2.1/go/src/github.com/pingcap/tidb-binlog/drainer/syncer.go:100: ESC[0m
2019/01/15 23:39:31 server.go:242: ESC[0;37m[info] syncer goroutine exited ESC[0m
2019/01/15 23:39:31 server.go:437: ESC[0;37m[info] begin to close drainer server ESC[0m

2. What did you expect to see?

3. What did you see instead?

4. What version of TiDB are you using (tidb-server -V or run select tidb_version(); on TiDB)?

Release Version: v2.1.2-1-g8ba8096
Git Commit Hash: 8ba809622668a1287e9f7152bfd34ad6be220304
Git Branch: release-2.1
UTC Build Time: 2018-12-21 03:45:55
GoVersion: go version go1.11.2 linux/amd64
Race Enabled: false
TiKV Min Version: 2.1.0-alpha.1-ff3dd160846b7d1aed9079c389fc188f7f5ea13e
Check Table Before Drop: false

5. which tool are you using?
   tidb-binlog

6. what versionof tool are you using (pump -V or tidb-lightning -V or syncer -V)?

./pump -V
2019/01/16 13:07:13 version.go:18: [info] Git Commit Hash: ba468df227f8a9ba69cc1e34f403a3d9bf7344a0
2019/01/16 13:07:13 version.go:19: [info] Build TS: 2018-12-21 12:47:38
2019/01/16 13:07:13 version.go:20: [info] Go Version: go1.11.2
2019/01/16 13:07:13 version.go:21: [info] Go OS/Arch: linuxamd64

./drainer -V
2019/01/16 13:07:44 version.go:18: [info] Git Commit Hash: ba468df227f8a9ba69cc1e34f403a3d9bf7344a0
2019/01/16 13:07:44 version.go:19: [info] Build TS: 2018-12-21 12:48:01
2019/01/16 13:07:44 version.go:20: [info] Go Version: go1.11.2
2019/01/16 13:07:44 version.go:21: [info] Go OS/Arch: linuxamd64

谢谢

I tried from several DBs, all failed for same error

2018/07/16 16:51:19 main.go:72: [fatal] create database connection failed: Error 1115: Unknown character set: 'utf8mb4'

When I check with a previous version downloaded from June, it completed successfully.

Feature Request

链接：[https://github.com/pingcap/docs-cn/issues/1171]

./syncer -V
Release Version: v1.0.0-51-gd2ff8d6
Git Commit Hash: d2ff8d6639c29b1bcd7eafd7231e5707a864465d
Git Branch: master
UTC Build Time: 2018-12-12 06:27:14
Go Version: go version go1.11.2 linux/amd64

分表同步的场景下，上游mysql表名为user_dynamic_0000到user_dynamic_0031，同步到tidb中合并为user_dynamic一个表，上游各个表加字段的时间不同步，但是tidb中已经同步好了增加字段的sql，导致报错
gen update sqls failed: update columns and data mismatch in length: 33 vs 32, schema: user_circle, table: user_dynamic

new-binlog Support

Please describe your problem here:

new-binlog 部署后，pump 的 dir 下面没有数据生成
用户使用 master 版本搭建 new-binlog 环境
pump 和 drainer 通过 binlogctl 状态看起来正常
TiDB 有数据写入

Additionally, please provide the following info before submitting your issue. Thanks!

1. Versions of the tools

   • binlogctl version (run syncer -V):

       binlogctl: v1.0.0~rc2+git.0a51ecb
       Git Commit Hash: 0a51ecb01dbd670d719e740bb3f2d3f5505ca821
       UTC Build Time: 2018-10-31 08:22:17
       Go Version: go1.11

- [ ] TiDB cluster version (execute `SELECT tidb_version();` in a MySQL client):

MySQL [(none)]> select tidb_version()\G;
*************************** 1. row ***************************
Release Version: v2.1.0-rc.3-151-gbb6d0a9
Git Commit Hash: bb6d0a935d7002a2e2803842171ed8f23a271751
Git Branch: master
UTC Build Time: 2018-11-06 09:31:22
GoVersion: go version go1.11 linux/amd64
Race Enabled: false
TiKV Min Version: 2.1.0-alpha.1-ff3dd160846b7d1aed9079c389fc188f7f5ea13e
Check Table Before Drop: false

- [ ] How did you deploy pump&drainer?

    ```

yes
![24581541650440_ pic_hd](https://user-ima
ges.githubusercontent.com/43946384/48179591-e0084400-e35a-11e8-92ca-94a5a78e9f0e.jpg)
```

- [ ] Other interesting information (system version, hardware config, etc):

    >[pump.log](https://github.com/pingcap/tidb-tools/files/2560440/pump.log)
    >[drainer.log](https://github.com/pingcap/tidb-tools/files/2560441/drainer.log)
    >[pump.txt](https://github.com/pingcap/tidb-tools/files/2560451/pump.txt)
    >[drainer.txt](https://github.com/pingcap/tidb-tools/files/2560452/drainer.txt)
    >

2. Operation logs

3. Common issues

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?

DDL:

/*!40101 SET NAMES binary*/;
/*!40014 SET FOREIGN_KEY_CHECKS=0*/;
CREATE TABLE  exam_detail (
  resourceid varchar(50) COLLATE utf8mb4_unicode_ci NOT NULL,
   level_score_statistic json DEFAULT NULL,
  PRIMARY KEY (resourceid),
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

sql:

/*!40101 SET NAMES binary*/;
/*!40014 SET FOREIGN_KEY_CHECKS=0*/;
INSERT INTO exam_detail VALUES
("00004aedb3584bxxxx4254e482dd92d2",CONVERT("{}" USING UTF8MB4)),
("0000b8930bb24eyyyy2ce4ccb29dd91e",CONVERT("{\"52\": 1, \"54\": 1, \"68\": 1, \"71\": 1, \"72\": 2, \"75\": 1, \"77\": 1, \"78\": 1, \"79\": 1, \"82\": 1, \"85\": 1, \"86\": 2, \"89\": 3, \"91\": 1, \"93\": 2, \"95\": 1, \"98\": 2, \"100\": 3, \"101\": 2, \"102\": 1, \"103\": 1, \"104\": 3, \"105\": 1, \"106\": 1, \"107\": 1, \"108\": 2, \"109\": 2, \"110\": 3, \"111\": 1, \"112\": 2, \"113\": 1, \"114\": 1, \"115\": 1, \"117\": 1, \"126\": 1}" USING UTF8MB4));

2. What did you expect to see?

tidb-lightning can load these sql into tidb successfully.

3. What did you see instead?

2018/12/16 03:47:17.054 main.go:52: [error] tidb lightning encountered error:Syntax error
github.com/pingcap/tidb-lightning/lightning/mydump.(*ChunkParser).lex
        /home/jenkins/workspace/build_tidb_lightning_2.1/go/src/github.com/pingcap/tidb-lightning/lightning/mydump/parser_generated.go:1283
github.com/pingcap/tidb-lightning/lightning/mydump.(*ChunkParser).ReadRow
        /home/jenkins/workspace/build_tidb_lightning_2.1/go/src/github.com/pingcap/tidb-lightning/lightning/mydump/parser.go:128
github.com/pingcap/tidb-lightning/lightning/mydump.(*ChunkParser).ReadChunks
        /home/jenkins/workspace/build_tidb_lightning_2.1/go/src/github.com/pingcap/tidb-lightning/lightning/mydump/parser.go:178
github.com/pingcap/tidb-lightning/lightning/mydump.splitExactChunks
        /home/jenkins/workspace/build_tidb_lightning_2.1/go/src/github.com/pingcap/tidb-lightning/lightning/mydump/region.go:145
github.com/pingcap/tidb-lightning/lightning/mydump.(*RegionFounder).MakeTableRegions.func1
        /home/jenkins/workspace/build_tidb_lightning_2.1/go/src/github.com/pingcap/tidb-lightning/lightning/mydump/region.go:100
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1333
/data/tidb/mydumper/exam_detail.sql

4. What version of TiDB are you using (tidb-server -V or run select tidb_version(); on TiDB)?

v2.1.1

5. which tool are you using?

./bin/mydumper  -V
mydumper 0.9.5, built against MySQL 5.7.23

6. what versionof tool are you using (pump -V or tidb-lightning -V or syncer -V)?

./bin/tidb-lightning -V
Release Version: v2.1.0-4-g510f71e
Git Commit Hash: 510f71e1fc653e2baf9b8539c02010e16a4f8977
Git Branch: release-2.1
UTC Build Time: 2018-12-12 05:15:00
Go Version: go version go1.11.2 linux/amd64

json data doesn't replicate with correct order using syncer.

• show create table

CREATE TABLE `dynamic` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT COMMENT '主键',
  `uid` varchar(32) NOT NULL COMMENT '业务uid',
  `type` int(11) NOT NULL DEFAULT '1' ,
  `title` varchar(256) NOT NULL DEFAULT '' COMMENT '标题',
  `content` varchar(8192) DEFAULT NULL,
  `picture_url` varchar(2048) NOT NULL DEFAULT '' COMMENT '图片地址字符串，多个地址用","分隔',
  `audio_url` varchar(128) NOT NULL DEFAULT '' COMMENT '音频url',
  `audio_duration` bigint(20) unsigned NOT NULL DEFAULT '0' COMMENT '音频时长',
  `web_content` varchar(1024) NOT NULL DEFAULT '' COMMENT '外链内容',
  `need_reply` tinyint(1) unsigned NOT NULL DEFAULT '0' COMMENT '是否需要回复，0是不需要，1是需要',
  `reply_expired_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '回复截止时间',
  `show_role` tinyint(2) unsigned NOT NULL DEFAULT '0' COMMENT '可见的角色',
  `creator_uid` varchar(32) NOT NULL DEFAULT '' COMMENT '创建者uid',
  `created_at` datetime NOT NULL COMMENT '创建时间',
  `updated_at` datetime NOT NULL COMMENT '更新时间',
  `is_deleted` tinyint(1) NOT NULL DEFAULT '0' COMMENT '删除标识',
  `like_count` bigint(20) unsigned NOT NULL DEFAULT '0' COMMENT '点赞数',
  `audios` json DEFAULT NULL COMMENT '存储音频列表',
  `videos` json DEFAULT NULL COMMENT '存储视频列表',
  PRIMARY KEY (`id`),
  UNIQUE KEY `idx_uid` (`uid`) USING BTREE,
  KEY `idx_type` (`type`)
) ENGINE=InnoDB AUTO_INCREMENT=5542058 DEFAULT CHARSET=utf8;

• checking failed data due to column audios

• source from mysql

id: 5541955
               uid: 3b047b3fe8604026ac12084770e2a609
              type: 1
             title: 111
           content: 555
       picture_url:
         audio_url:
    audio_duration: 0
       web_content:
        need_reply: 1
reply_expired_time: 2018-11-16 11:03:46
         show_role: 1
       creator_uid: ced40d39cc0a4d23a28f47c4d218e5a0
        created_at: 2018-11-16 11:03:24
        updated_at: 2018-11-16 11:03:24
        is_deleted: 0
        like_count: 0
            audios: [{"audioUrl": "111", "audioDuration": 111}]
            videos: NULL

• target from tidb

id: 5541955
               uid: 3b047b3fe8604026ac12084770e2a609
              type: 1
             title: 111
           content: 555
       picture_url:
         audio_url:
    audio_duration: 0
       web_content:
        need_reply: 1
reply_expired_time: 2018-11-16 11:03:46
         show_role: 1
       creator_uid: ced40d39cc0a4d23a28f47c4d218e5a0
        created_at: 2018-11-16 11:03:24
        updated_at: 2018-11-16 11:03:24
        is_deleted: 0
        like_count: 0
            audios: [{"audioDuration": 111, "audioUrl": "111"}]
            videos: NULL

• MySQL master version (percona server) :

Connection id:		73439
Current database:	xxx
Current user:		[email protected]
SSL:			Not in use
Current pager:		stdout
Using outfile:		''
Using delimiter:	;
Server version:		5.7.19-17-log Source distribution
Protocol version:	10
Connection:		10.10.12.116 via TCP/IP
Server characterset:	utf8mb4
Db     characterset:	utf8mb4
Client characterset:	utf8mb4
Conn.  characterset:	utf8mb4
TCP port:		3306
Uptime:			50 days 15 hours 13 min 39 sec

• tidb_version:

Release Version: v2.0.7
Git Commit Hash: 29ec059
Git Branch: release-2.0
UTC Build Time: 2018-09-07 12:36:02
GoVersion: go version go1.11 linux/amd64
TiKV Min Version: 2.0.0-rc.4.1

Release Version: v1.0.0-48-g4ac1cfc
Git Commit Hash: 4ac1cfc7c1f9ac256c37733d831ce7dd1643b3fa
Git Branch: master
UTC Build Time: 2018-11-15 09:15:13
Go Version: go version go1.11 linux/amd64

在官方文档：https://pingcap.com/docs-cn/tools/syncer/#syncer-%E4%BD%BF%E7%94%A8%E6%96%87%E6%A1%A3 里面的“配置 Syncer 监控与告警”段落

能否详细解释一下这个expr表达式是什么意思？谢谢！
IF syncer_binlog_file{node='master'} - ON(instance, job) syncer_binlog_file{node='syncer'} > 1

Hi,

I have 2 clusters TiDB and use TiDB-Binlog to sync between them. My question is: How do I record the latency between the upstream and downstream?

2019/01/24 11:19:50 diff.go:258: [error] table: test_user, range: (uid >= ? AND uid < ? AND ctime >= '2017-05-09 10:39:37' AND ctime < '2019-01-23 17:40:00'), args: [151262720433299085 151753833161977213], checksum is not equal, one is 1422990440, another is 2838034901
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x68 pc=0xc5fcfe]

goroutine 29 [running]:
github.com/pingcap/tidb-tools/pkg/dbutil.SelectUniqueOrderKey(0x0, 0xed0060, 0x11, 0xc000147b18, 0x4c6816, 0xc0001ee840, 0x0)
/home/jenkins/workspace/build_tidb_tools_master/go/src/github.com/pingcap/tidb-tools/pkg/dbutil/index.go:165 +0xae
github.com/pingcap/tidb-tools/pkg/diff.getChunkRows(0xfcda60, 0xc0000ac010, 0xc0001ee9c0, 0xc0002dbb00, 0xf, 0xc0002dbb40, 0x10, 0x0, 0xc00043cd20, 0x67, ...)
/home/jenkins/workspace/build_tidb_tools_master/go/src/github.com/pingcap/tidb-tools/pkg/diff/diff.go:541 +0x43
github.com/pingcap/tidb-tools/pkg/diff.(*TableDiff).checkChunkDataEqual(0xc0000da480, 0xfcda60, 0xc0000ac010, 0xc0004f8000, 0x419, 0x1063, 0x0, 0x0, 0x0)
/home/jenkins/workspace/build_tidb_tools_master/go/src/github.com/pingcap/tidb-tools/pkg/diff/diff.go:264 +0x19f
github.com/pingcap/tidb-tools/pkg/diff.(*TableDiff).EqualTableData.func1(0xc0000da480, 0xfcda60, 0xc0000ac010, 0xc0004cf3b0, 0xc0004f8000, 0x419, 0x1063)
/home/jenkins/workspace/build_tidb_tools_master/go/src/github.com/pingcap/tidb-tools/pkg/diff/diff.go:192 +0x73
created by github.com/pingcap/tidb-tools/pkg/diff.(*TableDiff).EqualTableData
/home/jenkins/workspace/build_tidb_tools_master/go/src/github.com/pingcap/tidb-tools/pkg/diff/diff.go:191 +0x3e4

1、upstream mysql： Server version: 5.7.16-log MySQL Community Server (GPL)
2、downstr mysql： Server version: 5.6.16-log Source distribution
3、./bin/sync_diff_inspector -V
INFO[0000] version:
sync_diff_inspector: v1.0.0~rc2+git.883b07a
Git Commit Hash: 883b07a
UTC Build Time: 2019-01-04 03:41:22
Go Version: go1.11.2

2018/10/10 11:44:49 syncer.go:691: [info] [query]delete from database.table where xdy_id = 0 and order_push = '' LIMIT 10000 [current pos](mysql-bin.000342, 45411735) [next pos](mysql-bin.000342, 45412052) [current gtid set]6c8a10ed-ed0b-11e4-91eb-00163ec546ca:474850265-545675226,b5d40430-394b-11e7-8dcf-1866daeaae14:1-10,b5d40430-394b-11e7-8dcf-2381daeaae14:1-200688052 [next gtid set]6c8a10ed-ed0b-11e4-91eb-00163ec546ca:474850265-545675226,b5d40430-394b-11e7-8dcf-1866daeaae14:1-10,b5d40430-394b-11e7-8dcf-2381daeaae14:1-200688038
2018/10/10 11:44:49 syncer.go:697: [error] fail to be parsed, error query delete from database.table where xdy_id = 0 and order_push = '' LIMIT 10000: unexpected dml statement found in query event
2018/10/10 11:44:49 syncer.go:962: [info] flush all jobs meta = syncer-binlog = (mysql-bin.000342, 45411735), syncer-binlog-gtid = b5d40430-394b-11e7-8dcf-1866daeaae14:1-10,b5d40430-394b-11e7-8dcf-2381daeaae14:1-200688057,6c8a10ed-ed0b-11e4-91eb-00163ec546ca:474850265-545675226
2018/10/10 11:44:49 meta.go:137: [info] save position to file, binlog-name:mysql-bin.000342 binlog-pos:45411735 binlog-gtid:6c8a10ed-ed0b-11e4-91eb-00163ec546ca:474850265-545675226,b5d40430-394b-11e7-8dcf-1866daeaae14:1-10,b5d40430-394b-11e7-8dcf-2381daeaae14:1-200688037
2018/10/10 11:44:49 main.go:79: [error] /home/jenkins/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/ddl.go:29: unexpected dml statement found in query event
/home/jenkins/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/ddl.go:76: query delete from database.table where xdy_id = 0 and order_push = '' LIMIT 10000
/home/jenkins/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/syncer.go:698: 
/home/jenkins/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/syncer.go:179:  
2018/10/10 11:44:49 syncer.go:872: [info] print status exits, err:context canceled

the purpose of this issue is only requestting feature that syncer can recognize and skip statement format binlogs. thanks.

syncer Support

环境信息
使用Syncer 同步上游 阿里云 RDS 数据库失败
上游表sec 为没有主键表, 有索引的数据表, 使用syncer 同步时, 似乎一碰到有删除的syncer 就直接同步失败且自动停止

请问有没有什么解决方案?

提示信息:
2019/06/04 14:09:07 syncer.go:633: [info] rotate binlog to (mysql-bin.002591, 372809559)
2019/06/04 14:09:07 syncer.go:987: [info] flush all jobs meta = syncer-binlog = (mysql-bin.002591, 372809559), syncer-binlog-gtid =
2019/06/04 14:09:07 meta.go:137: [info] save position to file, binlog-name:mysql-bin.002591 binlog-pos:372809559 binlog-gtid:
2019/06/04 14:09:07 main.go:79: [error] /home/jenkins/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/syncer.go:692: gen delete sqls failed: delete columns and data mismatch in length: 6 vs 7, schema: reg, table: sec

syncer没有开源吗

There is a table has a set field region defined as

`region` set('1','1001','2001','3001','4001','5001','6001','7001','8001','8100','9001')

And for a record with 5001, mysql binlog shows

### SET
...
###   @6=b'0010000000000000' /* SET(2 bytes) meta=63490 nullable=0 is_null=0 */

syncer tried to insert 8192 (b'0010000000000000') directly into the record, where it should decode the bit to '5001' first.

tidb failed for

"REPLACE INTO `u`.`t_common` (`id`,`bit`,`purpose`,`is_valid`,`is_basic`,`region`) VALUES (126,90,'XXX',1,0,8192);"
/home/jenkins/workspace/build_tidb_2.0/go/src/github.com/pingcap/tidb/types/datum.go:1554: cannot convert datum from tinyint to type set.

参考官方文档部署了DM，同步上游mysql数据到下游tidb时，进行到全量备份时，发现 mydumper的日志伴随如下错误，而且备份出的一部分表数据为空
2019-01-24 18:07:56 [ERROR] - Error dumping schemas (dbname.tbname): MySQL server has gone away
同样的命令，替换为原生的mydumper再去备份，没有发生这种情况，相关版本信息如下
./dm-master -V
Release Version: v1.0.0-alpha-1-g6b61fa7
Git Commit Hash: 6b61fa79d72d0e388c7e2a8c630a063e47056df0
Git Branch: master
UTC Build Time: 2019-01-21 10:04:57
Go Version: go version go1.11.2 linux/amd64

./dm-worker -V
Release Version: v1.0.0-alpha-1-g6b61fa7
Git Commit Hash: 6b61fa79d72d0e388c7e2a8c630a063e47056df0
Git Branch: master
UTC Build Time: 2019-01-21 10:04:54
Go Version: go version go1.11.2 linux/amd64

./mydumper -V
mydumper 0.9.5, built against MySQL 5.7.24

备份没问题的原生mydumper版本如下
mydumper 0.9.1, built against MySQL 5.7.23-23

谢谢

checker.go:202: [error] query sql[show tables;] failed dial tcp 127.0.0.1:3306: connect: connection refused
总是显示这句，我能十分肯定我的mysql服务可以使用root用户正常连接。

feature page

• document of shard table merge @csuzhangxc
• heartbeat @GregoryIan

improve

• detail description of query-status @amyangfei
• improve troubleshoot @GregoryIan
• exception handling @csuzhangxc
• relay log @amyangfei
• how to enable gtid and set relay log position @GregoryIan
• usage scene example @GregoryIan
• improve index @amyangfei

使用 pump跟drainer 将tidb binlog 同步到kafka ...

有几个问题：
1、仅往一个partition 写数据。
2、有可能单条记录很大，request超过限制大小，一条日志可以解析出多条操作日志。

3、官网文档没有比较详细的解释，kafka的参数太少
4、源码没有找到drainer的相关代码

您好：
我的环境是：mysql_1 -> syncer -> mysql_2

当我在mysql_1上创建表tt_11时，成功，但是syncer却报如下错误，谢谢！

2018/10/29 10:37:49 db.go:130: [warning] [exec][sql]USE test; create table tt_11(id int not null AUTO_INCREMENT primary key, name VARCHAR (30));[args][][error]Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'create table tt_11(id int not null AUTO_INCREMENT primary key, name VARCHAR (30' at line 1

显然语法是正确的，不知道syncer是否需要作出什么调整还是不兼容multi sql，谢谢

分别测试了mydumper 0.9.2, built against MySQL 5.5.53与mydumper 0.9.5, built against MySQL 5.7.24两个版本，如果填写一个错误的密码，会出现Access denied for user xxx的错误，说明是可以正确解析域名并尝试连接的，但是填写正确密码后，0.9.5版本直接报Segmentation fault错误，0.9.2版本报CRITICAL **: Error connecting to database: Unknown MySQL server host 'xxxxx'
直接写ip的话是可以成功dump数据的，所以应该不是后端mysql的问题

使用syncer从mysql5.7同步数据到tidb 3.0，同步过程中报错：
2019/02/22 04:00:19 ddl.go:57: [error] 2019/02/22 04:00:19 syncer.go:739: [error] fail to be parsed, error error while parsing sql: ALTER TABLE zichan360bi_dw.data_analy_agent_history_case TRUNCATE PARTITION p20190221, err:line 1 column 65 near " PARTITION p20190221" (total length 85) 2019/02/22 04:00:19 main.go:79: [error]

tidb 3.0 是支持删除表分区的，而且测试手动删除可以成功，但通过syncer同步时就报错。

怀疑syncer没有对删除分区就行解析。

我使用的版本：tidb v3.0 syncer v1.0 .

单库test，下面分为多个库t_0000~t_0031，不同分表之间dynamic_id存在主键冲突
使用配置样例如下
column-mappings:
rule-1:
​ schema-pattern: "test"
​ table-pattern: "t_*"
​ expression: "partition id"
​ source-column: "dynamic_id"
​ target-column: "dynamic_id"
​ arguments: ["1", "test", "t_00"]
出现错误"msg": "test is not the prefix of test not valid
文档里面都是分库分表的形式，求教单库的这种情况这块配置该怎么写？

Feature Request

Is your feature request related to a problem? Please describe:

The provided mydumper from the TiDB enterprise toolset downloadable tarball is probably statically linked with MySQL 5.7 which is using YaSSL and makes it very slow on SSL/TLS enabled MySQL servers

Describe alternatives you've considered:

suggest to build/link mydumper with MySQL 8, which uses openssl instead Or add a flag to disable ssl in mydumper.

hello ,
sorry ,i'm comming again.

first, i gots re-deployed .
but,when i deploy and start_pump ,it's no problem,
and i deploy drainer it's also success,,then
i get start drainer ,it's fails:

TASK [start drainer by systemd] ************************************************************************************************************************************************************
ok: [mysqltest]

TASK [wait until the drainer port is up] ***************************************************************************************************************************************************
fatal: [mysqltest]: FAILED! => {"changed": false, "elapsed": 300, "msg": "the drainer port 8249 is not up"}
to retry, use: --limit @/home/tidb/tidb-ansible-01/retry_files/start_drainer.retry

PLAY RECAP *********************************************************************************************************************************************************************************
172.19.243.118 : ok=3 changed=0 unreachable=0 failed=0
172.19.243.119 : ok=3 changed=0 unreachable=0 failed=0
172.19.243.126 : ok=3 changed=0 unreachable=0 failed=0
172.19.30.127 : ok=3 changed=0 unreachable=0 failed=0
172.19.30.128 : ok=3 changed=0 unreachable=0 failed=0
172.19.30.129 : ok=3 changed=0 unreachable=0 failed=0
172.19.30.130 : ok=3 changed=0 unreachable=0 failed=0
172.19.30.131 : ok=3 changed=0 unreachable=0 failed=0
172.19.30.132 : ok=3 changed=0 unreachable=0 failed=0
localhost : ok=1 changed=0 unreachable=0 failed=0
mysqltest : ok=4 changed=0 unreachable=0 failed=1
pump1 : ok=3 changed=0 unreachable=0 failed=0

ERROR MESSAGE SUMMARY **********************************************************************************************************************************************************************
[mysqltest]: Ansible FAILED! => playbook: start_drainer.yml; TASK: wait until the drainer port is up; message: {"changed": false, "elapsed": 300, "msg": "the drainer port 8249 is not up"}

Ask for help:
Contact us: [email protected]

it's not print log on drainer machine ,

and pump log message:
.....
LevelDurations:[]}
2019/03/19 16:11:12 server.go:496: [info] writeBinlogCount: 0, alivePullerCount: 0, maxCommitTS: 407105193868787713
.....

no more other message .

Feature Request

Is your feature request related to a problem? Please describe:
dm-worker have warnings describing binlog files is failed to pull.

Describe the feature you'd like:
it will be nice to have an alarm if mysql-master is migrated, and corresponding config for dm-worker failed to be corrected in time.

./bin/ddl_checker --host 127.0.0.1 --port 3306 --user root --password 123456
执行完也没报错 也没检查结果 是修改了使用规则还是?

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
   dumping data from mysql5.7 and load into tidb

2. What did you expect to see?
   all data are dumped into tidb

3. What did you see instead?
   invalid mydumper files for there are no -schema-create.sql files found

4. What version of TiDB are you using (tidb-server -V or run select tidb_version(); on TiDB)?
   Release Version: v3.0.0-rc.1-383-g18724b950
   Git

5. which tool are you using?
   tidb-enterprise-tools

6. what versionof tool are you using (pump -V or tidb-lightning -V or syncer -V)?
   tidb-enterprise-tools-latest-linux-amd64.tar.gz

• MySQL master version (percona server) :

Connection id:		73439
Current database:	xxx
Current user:		[email protected]
SSL:			Not in use
Current pager:		stdout
Using outfile:		''
Using delimiter:	;
Server version:		5.7.19-17-log Source distribution
Protocol version:	10
Connection:		10.10.12.116 via TCP/IP
Server characterset:	utf8mb4
Db     characterset:	utf8mb4
Client characterset:	utf8mb4
Conn.  characterset:	utf8mb4
TCP port:		3306
Uptime:			50 days 15 hours 13 min 39 sec

• syncer ERROR LOG:

2018/11/13 15:09:25 syncer.go:720: [info] [query]ALTER TABLE schemadb.ep_edu_course_message_auto_reply MODIFY answer JSON COMMENT '回复的内容-格式为list，有两个字段："answerType"：//''发送客服消息类型：1-文本消息，2-图片，3-图文链接''；  answer：回复内容' [current pos](bin-log-mysqld1.000119, 81773118) [next pos](bin-log-mysqld1.000119, 81773517) [current gtid set]a2c131c8-2729-11e8-b0fd-005056a2d719:1-115434626 [next gtid set]a2c131c8-2729-11e8-b0fd-005056a2d719:1-115434627
2018/11/13 15:09:25 syncer.go:982: [info] flush all jobs meta = syncer-binlog = (bin-log-mysqld1.000119, 81773118), syncer-binlog-gtid = a2c131c8-2729-11e8-b0fd-005056a2d719:1-115434626
2018/11/13 15:09:25 meta.go:137: [info] save position to file, binlog-name:bin-log-mysqld1.000119 binlog-pos:81773118 binlog-gtid:a2c131c8-2729-11e8-b0fd-005056a2d719:1-115434626


2018/11/13 15:09:25 main.go:79: [error] line 1 column 142 near "-文本消息，2-图片，3-图文链接'；  answer：回复内容'" (total length 261)
github.com/pingcap/tidb/parser/yy_parser.go:106:
github.com/pingcap/tidb/parser/yy_parser.go:119:
/home/jenkins/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/ddl.go:280:
/home/jenkins/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/syncer.go:1040:
/home/jenkins/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/syncer.go:737:
/home/jenkins/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/syncer.go:182:
2018/11/13 15:09:25 syncer.go:892: [info] print status exits, err:context canceled

• tidb_version:

Release Version: v2.0.7
Git Commit Hash: 29ec059
Git Branch: release-2.0
UTC Build Time: 2018-09-07 12:36:02
GoVersion: go version go1.11 linux/amd64
TiKV Min Version: 2.0.0-rc.4.1

syncer  --V
Release Version: v1.0.0-39-gd68cb83
Git Commit Hash: d68cb83af5628aca4a98a84b4ef83ccd65e84d2b
Git Branch: master
UTC Build Time: 2018-10-17 09:03:40
Go Version: go version go1.11 linux/amd64

2019/03/21 15:55:27 ddl.go:57: [error]
encountered incompatible DDL in TiDB: -- 计算类休假表 延期日期
DROP PROCEDURE IF EXISTS schema_change
please confirm your DDL statement is correct and needed.
for TiDB compatible DDL, please see the docs:
English version: https://github.com/pingcap/docs/blob/master/sql/ddl.md
Chinese version: https://github.com/pingcap/docs-cn/blob/master/sql/ddl.md
if the DDL is not needed, you can modify the meta file and restart syncer to skip it.

2019/03/21 15:55:27 syncer.go:713: [info] [query]-- 计算类休假表 延期日期
DROP PROCEDURE IF EXISTS schema_change [current pos](mysql-bin.002132, 262673070) [next pos](mysql-bin.002132, 262673285) [current gtid set] [next gtid set]
2019/03/21 15:55:27 syncer.go:719: [error] fail to be parsed, error error while parsing sql: -- 计算类休假表 延期日期
DROP PROCEDURE IF EXISTS schema_change, err:line 2 column 15 near " IF EXISTS schema_change" (total length 74)
2019/03/21 15:55:27 syncer.go:987: [info] flush all jobs meta = syncer-binlog = (mysql-bin.002132, 262673070), syncer-binlog-gtid =
2019/03/21 15:55:27 meta.go:137: [info] save position to file, binlog-name:mysql-bin.002132 binlog-pos:262673070 binlog-gtid:
2019/03/21 15:55:27 main.go:79: [error] /home/jenkins/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/ddl.go:67: error while parsing sql: -- 计算类休假表 延期日期
DROP PROCEDURE IF EXISTS schema_change, err:line 2 column 15 near " IF EXISTS schema_change" (total length 74)
/home/jenkins/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/syncer.go:720:
/home/jenkins/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/syncer.go:187:
2019/03/21 15:55:27 syncer.go:897: [info] print status exits, err:context canceled
[2019/03/21 15:55:27] [info] binlogsyncer.go:163 syncer is closing...
[2019/03/21 15:55:27] [error] binlogstreamer.go:57 close sync with err: sync is been closing...
[2019/03/21 15:55:27] [info] binlogsyncer.go:178 syncer is closed

Feature Request

Is your feature request related to a problem? Please describe:

support online diff check; sync-diff-inspector check data which are produced before specific time in source and target side.
Describe the feature you'd like:

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

checker dosn't support IPv6 host address at present, it's required by my project.

Not sure why my previous issue got deleted, but I would like to request this again.

Pls let me know why when you delete this request, thanks!

i want to access https://github.com/pingcap/tidb-tools/blob/docs/docs/dm/zh_CN/get-started.md

but, it's unaccessable now..
can you give me the same?

部署测试TiDB监控时发现几个疑问：

1. 如果部署多套tidb集群，相应的监控也会部署多套（Prometheus+grafana）,是否可以将多套监控整合在一起？
2. DM的监控是否也可以和tidb的监控整合在一起，而不需要单独的再部署一套DM监控的Prometheus+grafana？

base on tidb-binlog tools.

1. i'm installed success from https://pingcap.github.io/docs-cn/tools/tidb-binlog-cluster/

   when i 'ansible-playbook stop_drainer.yml' ,then 'rm -rf drainer.toml'

   now i'm running 'ansible-playbook start_drainer.yml' ,but it's don't work!.

2. and ,i'm running 'ansible-playbook stop.yml --tags=pump' then

   but i'm running 'ansible-playbook start.yml' then,it's does't work too.

so, i'm getting the pump status from pd-server ,thant's 'State: paused'

.i don't how to get start them, can you give me start method ,please..

thank you, very much.

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?

DM 连接的源数据库使用的账户如果没有 RELOAD 权限，DM 会报错，虽然 DM 有 ignore-checking-items 选项可以忽略掉 dump_privilege 的检查，但这个选项对应 RELOAD + SELECT，会也跳过 SELECT 的检查。

2. What did you expect to see?

对 RELOAD 权限跟文档描述的应该一致，是可选的权限，如果没有，输出警告即可。

3. What did you see instead?

没有 RELOAD 时 DM task 启动失败。

4. What version of TiDB are you using (tidb-server -V or run select tidb_version(); on TiDB)?

2.1.15。

5. which tool are you using?

TiDB DM.

6. what versionof tool are you using (pump -V or tidb-lightning -V or syncer -V)?

2.1.15。

• target table info from:
  

  tidb-tools/pkg/diff/diff.go

  Line 124 in d57120b

  tableInfo, err := dbutil.GetTableInfoWithRowID(ctx, t.TargetTable.Conn, t.TargetTable.Schema, t.TargetTable.Table, t.UseRowID)

  
  *source table info from:
  

  tidb-tools/pkg/diff/diff.go

  Line 131 in d57120b

  tableInfo, err := dbutil.GetTableInfoWithRowID(ctx, t.TargetTable.Conn, t.TargetTable.Schema, t.TargetTable.Table, t.UseRowID)

Source and target table generated by the same connection, schema ?

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
   I use syncer to sync data from mysql to tidb

2. What did you expect to see?
   syncer is always running

3. What did you see instead?
   syncer stuck in one positon, and under supervisor ,it keeps restarting and crashing.

[2018/11/01 18:54:05] [info] binlogsyncer.go:715 rotate to (mysql-bin.000003, 927030)
[2018/11/01 18:54:08] [info] binlogsyncer.go:132 create BinlogSyncer with config {2708 mysql 192.168.252.52 3316 data_strategy false false false UTC true 0 30s 1m0s 0 false}
[2018/11/01 18:54:08] [info] binlogsyncer.go:346 begin to sync binlog from position (mysql-bin.000003, 927030)
[2018/11/01 18:54:08] [info] binlogsyncer.go:195 register slave for master server 192.168.252.52:3316
[2018/11/01 18:54:08] [info] binlogsyncer.go:715 rotate to (mysql-bin.000003, 927030)
[2018/11/01 18:54:11] [info] binlogsyncer.go:132 create BinlogSyncer with config {2708 mysql 192.168.252.52 3316 data_strategy false false false UTC true 0 30s 1m0s 0 false}
[2018/11/01 18:54:11] [info] binlogsyncer.go:346 begin to sync binlog from position (mysql-bin.000003, 927030)
[2018/11/01 18:54:11] [info] binlogsyncer.go:195 register slave for master server 192.168.252.52:3316
[2018/11/01 18:54:11] [info] binlogsyncer.go:715 rotate to (mysql-bin.000003, 927030)

4. What version of TiDB are you using (tidb-server -V or run select tidb_version(); on TiDB)?
   Release Version: v1.0.8-1-gaacba4a\nGit Commit Hash: aacba4a427f6c1dd64cd581e1f888e00aa64e948\nGit Branch: release-1.0\nUTC Build Time: 2018-02-11 12:21:59

5. which tool are you using?

6. what versionof tool are you using (pump -V or tidb-lightning -V or syncer -V)?
   Release Version: v1.0.0-35-g8c16e63
   Git Commit Hash: 8c16e6370bbb0168ea809de39f9c0fd77255fa7a
   Git Branch: master
   UTC Build Time: 2018-09-25 06:38:03
   Go Version: go version go1.11 linux/amd64

2019/05/24 15:22:06 syncer.go:654: [info] rotate binlog to (mysql-bin.000479, 97533469)
2019/05/24 15:22:06 syncer.go:744: [error] [error query event][schema]circle [unsupport statement event][sql]UPDATE scores SET score = score - 5, updated_at = '2019-05-24 10:40:31'  WHERE user_id = 9678281 [error message]format of some binlog events are not `ROW`, syncer only support `ROW` format binlog (unexpected DML found in query event) [current pos](mysql-bin.000479, 97533469) [next pos](mysql-bin.000479, 97533726) [current gtid set] [next gtid set]<nil>
2019/05/24 15:22:06 syncer.go:1013: [info] flush all jobs meta = syncer-binlog = (mysql-bin.000479, 97533469), syncer-binlog-gtid =
2019/05/24 15:22:06 meta.go:135: [info] save position to file, binlog-name:mysql-bin.000479 binlog-pos:97533469 binlog-gtid:
2019/05/24 15:22:06 main.go:79: [error] /home/jenkins/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/ddl.go:86: [unsupport statement event][sql]UPDATE scores SET score = score - 5, updated_at = '2019-05-24 10:40:31'  WHERE user_id = 9678281 [error message]format of some binlog events are not `ROW`, syncer only support `ROW` format binlog (unexpected DML found in query event)

Additionally, please provide the following info before submitting your issue. Thanks!

1. Versions of the tools

   • syncer version (run syncer -V):

     Release Version: v1.0.0-65-geb481eb
     Git Commit Hash: eb481ebd78cbcbf85cde9161f92fa4c7c488a50a
     Git Branch: master
     UTC Build Time: 2019-05-07 03:35:01
     Go Version: go version go1.12 linux/amd64
     

   • Upstream MySQL server version (execute SELECT @@version; in a MySQL client):

      5.6.25-log
     

   • Downstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

     Release Version: v2.1.1
     Git Commit Hash: 990f859384b834fe5edf1410d2a4142def5c3fa9
     Git Branch: release-2.1
     UTC Build Time: 2018-12-12 10:57:44
     GoVersion: go version go1.11.2 linux/amd64
     Race Enabled: false
     TiKV Min Version: 2.1.0-alpha.1-ff3dd160846b7d1aed9079c389fc188f7f5ea13e
     Check Table Before Drop: false
     

   • Upstream MySQL sql_mode (execute SELECT @@sql_mode; in a MySQL client):

      STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION
     

   • Downstream TiDB sql_mode (execute SELECT @@sql_mode; in a MySQL client):

     STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION 
     

   • How did you deploy syncer?

      run on mysql instance machine.
     

     show global variables like 'binlog_format';
     +---------------+-------+
     | Variable_name | Value |
     +---------------+-------+
     | binlog_format | ROW   |
     +---------------+-------+
     

离线部署 ansible-playbook bootstrap.yml 到check_system_static 时遇到错误 ：

`TASK [check_system_static : get facts] *****************************************************************************************
2018-11-29 13:55:29,879 p=28716 u=tidb | ERROR! Unexpected Exception, this is probably a bug: update expected at most 1 arguments, got 2
2018-11-29 13:55:29,880 p=28716 u=tidb | to see the full traceback, use -vvv
2018-11-29 13:55:29,882 p=28716 u=tidb | the full traceback was:

Traceback (most recent call last):
File "/usr/bin/ansible-playbook", line 118, in
exit_code = cli.run()
File "/usr/lib/python2.7/site-packages/ansible/cli/playbook.py", line 122, in run
results = pbex.run()
File "/usr/lib/python2.7/site-packages/ansible/executor/playbook_executor.py", line 156, in run
result = self._tqm.run(play=play)
File "/usr/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py", line 292, in run
play_return = strategy.run(iterator, play_context)
File "/usr/lib/python2.7/site-packages/ansible/plugins/strategy/linear.py", line 315, in run
results += self._process_pending_results(iterator, max_passes=max(1, int(len(self._tqm._workers) * 0.1)))
File "/usr/lib/python2.7/site-packages/ansible/plugins/strategy/init.py", line 117, in inner
results = func(self, iterator, one_pass=one_pass, max_passes=max_passes)
File "/usr/lib/python2.7/site-packages/ansible/plugins/strategy/init.py", line 606, in _process_pending_results
self._variable_manager.set_host_facts(target_host, result_item['ansible_facts'].copy())
File "/usr/lib/python2.7/site-packages/ansible/vars/manager.py", line 597, in set_host_facts
self._fact_cache.update(host.name, facts)
TypeError: update expected at most 1 arguments, got 2`

It's seems that the sycner's source code was removed...

• MySQL master version (percona server) :

Connection id:		73439
Current database:	xxx
Current user:		[email protected]
SSL:			Not in use
Current pager:		stdout
Using outfile:		''
Using delimiter:	;
Server version:		5.7.19-17-log Source distribution
Protocol version:	10
Connection:		10.10.12.116 via TCP/IP
Server characterset:	utf8mb4
Db     characterset:	utf8mb4
Client characterset:	utf8mb4
Conn.  characterset:	utf8mb4
TCP port:		3306
Uptime:			50 days 15 hours 13 min 39 sec

• syncer ERROR LOG:

2018/10/26 13:49:00 db.go:130: [warning] [exec][sql]UPDATE `easi_pass`.`ep_edu_course_marketing_statistics` SET `id` = ?, `data_id` = ?, `data_type` = ?, `course_uv` = ?, `course_conversion_rate` = ?, `create_time` = ?, `update_time` = ? WHERE `id` = ? LIMIT 1;[args][1 a3274369923d47008a0ec821a88e6a15 1 10 0 2018-09-20 17:21:01 2018-10-26 13:48:46 1][error]driver: bad connection
2018/10/26 13:49:00 db.go:133: [error] [exec][sql]UPDATE xxxx 1 10 0 2018-09-20 17:21:01 2018-10-26 13:48:46 1][error]invalid connection
2018/10/26 13:49:00 db.go:102: [error] [exec][sql][UPDATE xxxx 1 10 0 2018-09-20 17:21:01 2018-10-26 13:48:46 1]][error]driver: bad connection
2018/10/26 13:49:00 syncer.go:490: [fatal] driver: bad connection
/home/jenkins/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/db.go:136:
/home/jenkins/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/db.go:103:

• tidb_version:

Release Version: v2.0.7
Git Commit Hash: 29ec059
Git Branch: release-2.0
UTC Build Time: 2018-09-07 12:36:02
GoVersion: go version go1.11 linux/amd64
TiKV Min Version: 2.0.0-rc.4.1

syncer  --V
Release Version: v1.0.0-39-gd68cb83
Git Commit Hash: d68cb83af5628aca4a98a84b4ef83ccd65e84d2b
Git Branch: master
UTC Build Time: 2018-10-17 09:03:40
Go Version: go version go1.11 linux/amd64

[root@echo-deployment-pvc-cccdf664b-8fszp bin]# ./checker -host 2002:ac1f:91c6::1 -port 10528 -user root -password test test
2018/10/22 09:19:36 main.go:74: [fatal] create database connection failed: dial tcp: address 2002:ac1f:91c6::1:10528: too many colons in address

It would be good to support IPv6 as it is being used widely today, can you consider this quick support in recent plan?

2018/10/11 09:41:06 diff.go:172: [error] check chunk data equal failed, error Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'group, groupjoin, groupthread, grouppost, post, wall, poke, click, sendpm, frien' at line 1
/home/jenkins/workspace/build_tidb_tools_master/go/src/github.com/pingcap/tidb-tools/pkg/dbutil/common.go:286: 
/home/jenkins/workspace/build_tidb_tools_master/go/src/github.com/pingcap/tidb-tools/pkg/diff/diff.go:205: 
/home/jenkins/workspace/build_tidb_tools_master/go/src/github.com/pingcap/tidb-tools/pkg/diff/diff.go:224:

so, add ` to surrounding objects please.

Feature Request

**由于历史原因，有2种不完全的分库分表实施方案：
1.分库名无后缀，表名有后缀，例如2个分库都为store，分库中的表名分别 为log_1，log_2
2.分库名有后续，表名无后缀，例如2个分库为store_1，store_2，分库中的表名都为log

**data migration工具强制要求分库分表有数字后缀，所以不支持以上2种情况的合并操作

**建议2种方案修改：

1. 在prefix==name的情况下，将schema id和table id默认为0，具体修改点在pkg/column-mapping/column.go文件，函数为computeID
2. 添加新的expression :-D

望采纳，谢谢！

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?

run sync_diff in sharding mode, remove-columns option changes the columns in table info slice, while index column keeps the table offset, which leads to index out of range error

[[check-tables]]
schema = "db_target"
tables = ["t_target"]

[[table-config]]
schema = "db_target"
table = "t_target"
remove-columns = ["id"]
is-sharding = true
index-fields = "uid"

[[table-config.source-tables]]
instance-id = "source-1"
schema = "sharding1"
table  = "~t.*"

[[table-config.source-tables]]
instance-id = "source-2"
schema = "sharding1"
table  = "~t.*"

mysql> show create table t1\G
*************************** 1. row ***************************
       Table: t1
Create Table: CREATE TABLE `t1` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `uid` int(11) DEFAULT NULL,
  `name` varchar(80) DEFAULT NULL,
  `info` varchar(100) DEFAULT NULL,
  `age` int(11) DEFAULT NULL,
  `info_json` json GENERATED ALWAYS AS (`info`) VIRTUAL,
  `id_gen` int(11) GENERATED ALWAYS AS ((`uid` + 1)) VIRTUAL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `uid` (`uid`),
  UNIQUE KEY `id_gen` (`id_gen`),
  KEY `multi_col_idx` (`uid`,`id_gen`)
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8mb4
1 row in set (0.00 sec)

2. What did you expect to see?

diff runs successfully

3. What did you see instead?

panic: runtime error: index out of range

goroutine 42 [running]:
github.com/pingcap/tidb-tools/pkg/dbutil.SelectUniqueOrderKey(0xc0003ba500, 0x0, 0x20, 0x203000, 0x203000, 0x0, 0x0)
        /home/apple/work/code/pingcap/gopath/src/github.com/pingcap/tidb-tools/pkg/dbutil/index.go:183 +0xb3b
github.com/pingcap/tidb-tools/pkg/diff.getChunkRows(0x1240ac0, 0xc0000b8010, 0xc00023c3c0, 0xc0002d1b50, 0x9, 0xc00031d19a, 0x2, 0xc0003ba500, 0xc00031dda0, 0xf, ...)
        /home/apple/work/code/pingcap/gopath/src/github.com/pingcap/tidb-tools/pkg/diff/diff.go:766 +0x76
github.com/pingcap/tidb-tools/pkg/diff.(*TableDiff).compareRows(0xc000238620, 0x1240ac0, 0xc0000b8010, 0xc0003b8af0, 0x3, 0x16, 0x0)
        /home/apple/work/code/pingcap/gopath/src/github.com/pingcap/tidb-tools/pkg/diff/diff.go:456 +0x224
github.com/pingcap/tidb-tools/pkg/diff.(*TableDiff).checkChunkDataEqual(0xc000238620, 0x1240ac0, 0xc0000b8010, 0x0, 0xc0003b8af0, 0x0, 0x0, 0x0)
        /home/apple/work/code/pingcap/gopath/src/github.com/pingcap/tidb-tools/pkg/diff/diff.go:421 +0x51a
github.com/pingcap/tidb-tools/pkg/diff.(*TableDiff).checkChunksDataEqual(0xc000238620, 0x1240ac0, 0xc0000b8010, 0x0, 0xc000390f60, 0xc000090700)
        /home/apple/work/code/pingcap/gopath/src/github.com/pingcap/tidb-tools/pkg/diff/diff.go:351 +0x17b
created by github.com/pingcap/tidb-tools/pkg/diff.(*TableDiff).CheckTableData
        /home/apple/work/code/pingcap/gopath/src/github.com/pingcap/tidb-tools/pkg/diff/diff.go:239 +0x21b

6. what versionof tool are you using (pump -V or tidb-lightning -V or syncer -V)?

➜  ./bin/sync_diff_inspector -V
version:
sync_diff_inspector: v1.0.0~rc2+git.661b5b4
Git Commit Hash: 661b5b4ff0d07f0cbf126a9d59a82e610ee84d12
UTC Build Time: 2019-04-23 08:03:21
Go Version: go1.11.2