支持 SQL Server <-> MySQL about yugong HOT 33 OPEN

你有分支么？我可以提前看看，YAML配置的变化能具体一些么？

from yugong.

@agapple 正在申请代码开源中。YAML 配置范例如下：

translators:
  record:
    '|BEFORE|':
      - class: com.taobao.yugong.translator.NameStyleDataTranslator
        properties:
          column_case_format_from: UPPER_CAMEL
          column_case_format_to: LOWER_UNDERSCORE
          table_case_format_from: UPPER_CAMEL
          table_case_format_to: LOWER_UNDERSCORE
          schema_to: hj_product
      - class: com.taobao.yugong.translator.ColumnFixDataTranslator
        properties:
          column_replace:
            i_d: id
    ATable:
      - class: com.taobao.yugong.translator.ColumnFixDataTranslator
        properties:
          column_alias:
            bussiness_id:
              - business_id
            is_high_light:
              - is_highlight
      - class: com.taobao.yugong.translator.ColumnFixDataTranslator
        properties:
          json_compress:
            ext_field:
              - mark
              - tags
              - ad_id

支持全局更换表名、列名风格；列名字符串替换；多个字段压缩到 JSON 字段内。

from yugong.

这个语法还是不错，清晰直接

from yugong.

请问这个PR什么进展。有计划合并进来么？

from yugong.

@stardustliu 苦逼被赶到其他项目奋斗了，如果大家有需求，我把 PR 弄出来。

from yugong.

@alswl 有需求。也是赶项目…………

from yugong.

@stardustliu @agapple https://blog.alswl.com/2018/03/sql-server-migration-1/
我稍微写了一点介绍。

from yugong.

@stardustliu 申请完成，已经提交了源码，在和 @agapple 沟通，想尝鲜可以直接使用我的分支 https://github.com/alswl/yugong/tree/feature/sql-server-to-mysql-overview
（这个分支其实也挺稳定，在线模式线上运行了 3 个月，离线模式也一直在使用）。

from yugong.

@alswl 仔细看了下代码，私聊我一下你的联系方式，weixin沟通一下合并的方案

from yugong.

@agapple 邮件已经发给了你的邮箱 jianghang115(a)gmail.com 。

from yugong.

请问下目前这个PR的进展如何了？现在本公司有sql server --> mysql 的需求

from yugong.

@hdpter 目前还没有合并到上游，如果急用可以使用我那个分支。

from yugong.

@alswl 这个PR现在进展如何？你提供的那个分支 jar 包 gitlab的连接无效了

from yugong.

from yugong.

@alswl 问下这个地方是放什么配置的，还需要我自己安装配置canal吗？
必须使用软连接？还是可以把配置放在canal_conf下
canal_conf -> ../yugong-conf/pass-state-2-pre/canal_conf

from yugong.

@eason0420 yugong 和 canal 没有依赖关系。我那个 mssql 分支对配置目录没有要求，路径在启动参数做配置，如 java -jar yugong-shaded.jar -c sync-mssql-mysql.properties -y mssql-mysql.yaml 。

from yugong.

@alswl 我下载你那分支的源码后有这个文件夹，还是个软连接，那就是可以把它删了吧？对编译没影响吧

lrwxrwxrwx 1 root root 42 Nov 27 09:58 canal_conf -> ../yugong-conf/pass-state-2-pre/canal_conf
-rw-r--r-- 1 root root 38584 Nov 27 09:58 codeformat.xml
-rw-r--r-- 1 root root 5692 Nov 27 09:58 codetemplates.xml
drwxr-xr-x 2 root root 4096 Nov 27 09:58 docs
-rw-r--r-- 1 root root 122 Nov 27 09:58 HJREADME.MD
-rw-r--r-- 1 root root 18045 Nov 27 09:58 LICENSE
-rw-r--r-- 1 root root 21976 Nov 27 09:58 pom.xml
-rw-r--r-- 1 root root 5586 Nov 27 09:58 README.md
-rw-r--r-- 1 root root 2629 Nov 27 09:58 README_OFFCIAL.md
-rwxr-xr-x 1 root root 691 Nov 27 09:58 release-mvn-to-git-release-binary-branch.sh
-rwxr-xr-x 1 root root 1382 Nov 27 09:58 saveVersion.sh
drwxr-xr-x 5 root root 4096 Nov 27 09:58 src
-rw-r--r-- 1 root root 2139 Nov 27 09:58 yugong.properties.sample
-rw-r--r-- 1 root root 3615 Nov 27 09:58 yugong.yaml.sample

from yugong.

@alswl 请看下，是不是有些依赖包有变化了，编译报错了
我下载这个源码 https://github.com/alswl/yugong.git

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 19.072 s
[INFO] Finished at: 2018-11-27T16:53:59+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project yugong: Could not resolve dependencies for project com.taobao.yugong:yugong:jar:1.2.0: The following artifacts could not be resolved: com.oracle:ojdbc14:jar:10.2.0.4.0, com.beust:jcommander:jar:1.71, com.alibaba.otter:canal.client:jar:1.0.25-SNAPSHOT: Could not find artifact com.oracle:ojdbc14:jar:10.2.0.4.0 in central (http://repo1.maven.org/maven2) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

from yugong.

@eason0420 Oracle 的 jar 包不在 Maven Central，你需要自己去 Oracle 官网下载驱动包。
com.oracle:ojdbc14:jar:10.2.0.4.0 in central - Google Search

from yugong.

@alswl jcommander:1.71这个jar也没有，我在pom中改成1.72版本是可以编译的，但过程还是有错，是不是与这个jar包版本有关，能先提供个编译好的jar包吗？

[ERROR] src/main/java/com/taobao/yugong/applier/FullRecordApplier.java:[478] (indentation) Indentation: 'if' child have incorrect indentation level 20, expected level should be 10.
[ERROR] src/main/java/com/taobao/yugong/applier/FullRecordApplier.java:[480] (indentation) Indentation: 'if' child have incorrect indentation level 20, expected level should be 10.
[ERROR] src/main/java/com/taobao/yugong/applier/FullRecordApplier.java:[481] (indentation) Indentation: 'if' child have incorrect indentation level 20, expected level should be 10.
[ERROR] src/main/java/com/taobao/yugong/applier/FullRecordApplier.java:[482] (indentation) Indentation: 'if' child have incorrect indentation level 20, expected level should be 10.
[ERROR] src/main/java/com/taobao/yugong/applier/FullRecordApplier.java:[483] (indentation) Indentation: 'if' child have incorrect indentation level 20, expected level should be 10.
[ERROR] src/main/java/com/taobao/yugong/applier/FullRecordApplier.java:[484] (indentation) Indentation: 'if rcurly' have incorrect indentation level 16, expected level should be 8.
[ERROR] src/main/java/com/taobao/yugong/applier/FullRecordApplier.java:[485] (indentation) Indentation: 'synchronized rcurly' have incorrect indentation level 12, expected level should be 6.
[ERROR] src/main/java/com/taobao/yugong/applier/FullRecordApplier.java:[486] (indentation) Indentation: 'if rcurly' have incorrect indentation level 8, expected level should be 4.
[ERROR] src/main/java/com/taobao/yugong/applier/FullRecordApplier.java:[488] (indentation) Indentation: 'method def' child have incorrect indentation level 8, expected level should be 4.
[ERROR] src/main/java/com/taobao/yugong/applier/FullRecordApplier.java:[489] (indentation) Indentation: 'method def rcurly' have incorrect indentation level 4, expected level should be 2.
[ERROR] src/main/java/com/taobao/yugong/applier/AbstractRecordApplier.java:[118,9] (naming) LocalVariableName: Local variable name 'i' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9]$'.
[ERROR] src/main/java/com/taobao/yugong/applier/AbstractRecordApplier.java:[130,9] (naming) LocalVariableName: Local variable name 'i' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9]$'.
[ERROR] src/main/java/com/taobao/yugong/applier/AbstractRecordApplier.java:[142,9] (naming) LocalVariableName: Local variable name 'i' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9]$'.
[ERROR] src/main/java/com/taobao/yugong/conf/Appiler.java:[7,18] (naming) MemberName: Member name 'a' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9]$'.
[ERROR] src/main/java/com/taobao/yugong/conf/Appiler.java:[8,18] (naming) MemberName: Member name 'b' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9]$'.
[ERROR] src/main/java/com/taobao/yugong/conf/Extractor.java:[7,18] (naming) MemberName: Member name 'a' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9]$'.
[ERROR] src/main/java/com/taobao/yugong/conf/Extractor.java:[8,18] (naming) MemberName: Member name 'b' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9]$'.
[ERROR] src/main/java/com/taobao/yugong/conf/Table.java:[7,18] (naming) MemberName: Member name 'a' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9]$'.
[ERROR] src/main/java/com/taobao/yugong/conf/Table.java:[8,18] (naming) MemberName: Member name 'b' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9]*$'.
[WARNING] checkstyle:check violations detected but failOnViolation set to false

from yugong.

@alswl 我用git clone下载的源码编译成功不了，但用下载的yugong-1.0.3版本的源码可以编译成功，但是没有生成yugong-shaded.jar而是生成了yugong-1.0.3.jar，这2个有什么区别？1.0.3支持mssql-->mysql吗

from yugong.

我修复了一个 maven 依赖问题，可以正常编译了。

from yugong.

@eason0420 打了一个包 https://github.com/alswl/yugong/releases/tag/541e5f8

from yugong.

@alswl 感谢，我这边编译还是有些错，与JDK版本有关系吗？我的是java version "1.8.0_191"

我先用你提供的编译好的jar包测试下

[ERROR] src/main/java/com/taobao/yugong/applier/FullRecordApplier.java:[489] (indentation) Indentation: 'method def rcurly' have incorrect indentation level 4, expected level should be 2.
[ERROR] src/main/java/com/taobao/yugong/applier/AbstractRecordApplier.java:[118,9] (naming) LocalVariableName: Local variable name 'i' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9]$'.
[ERROR] src/main/java/com/taobao/yugong/applier/AbstractRecordApplier.java:[130,9] (naming) LocalVariableName: Local variable name 'i' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9]$'.
[ERROR] src/main/java/com/taobao/yugong/applier/AbstractRecordApplier.java:[142,9] (naming) LocalVariableName: Local variable name 'i' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9]$'.
[ERROR] src/main/java/com/taobao/yugong/conf/Appiler.java:[7,18] (naming) MemberName: Member name 'a' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9]$'.
[ERROR] src/main/java/com/taobao/yugong/conf/Appiler.java:[8,18] (naming) MemberName: Member name 'b' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9]$'.
[ERROR] src/main/java/com/taobao/yugong/conf/Extractor.java:[7,18] (naming) MemberName: Member name 'a' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9]$'.
[ERROR] src/main/java/com/taobao/yugong/conf/Extractor.java:[8,18] (naming) MemberName: Member name 'b' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9]$'.
[ERROR] src/main/java/com/taobao/yugong/conf/Table.java:[7,18] (naming) MemberName: Member name 'a' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9]$'.
[ERROR] src/main/java/com/taobao/yugong/conf/Table.java:[8,18] (naming) MemberName: Member name 'b' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9]*$'.

from yugong.

我本地版本也是 1.8.0_192，我看你这个错误都是格式检查报错。你确认一下使用的 git 版本是 541e5f8 。

from yugong.

@alswl 我是用git clone https://github.com/alswl/yugong.git 以及直接从releases中下载yugong-541e5f8.tar 这个，2种方式编译都报这个错

from yugong.

@alswl 能提供一个yaml文件的配置案列吗？整库多个表，字段名不同的映射，以及目标和源库表中字段不一样如何设置等 ，还有就是表名以及列名的大小写不一样的话怎么处理 自己琢磨总是不成功。

from yugong.

@eason0420 字段映射：https://github.com/alswl/yugong/blob/feature/sql-server-to-mysql-overview/yugong.yaml.sample#L50

大小写设置：https://github.com/alswl/yugong/blob/feature/sql-server-to-mysql-overview/yugong.yaml.sample#L55 这是统一处理，处理之后还可以用 column_alias 做个性化处理。

from yugong.

@alswl 您好，全量同步的时候是否需要锁库？如果不需要加锁，如何保证数据库的一致性问题？mysql可以通过快照，读取快照，binlog解析保证，mssql该当如何？

from yugong.

@LiPL2017 看 https://blog.alswl.com/2018/03/sql-server-migration-1/

from yugong.

@alswl ,我的target表比souce多一个id字段，这个yugong能处理吗？应该怎么配置？

from yugong.

@wavelet123 如果是我那个版本参考 https://github.com/alswl/yugong/blob/feature/sql-server-to-mysql-overview/src/it/resources/sample_config/2_mysql-mssql-check.yaml#L61

如果是官方版本自己定制一个 Translator
https://github.com/alibaba/yugong/blob/master/src/main/java/com/taobao/yugong/translator/DataTranslator.java

from yugong.

@alswl 能提供一个yaml文件的配置案列吗？整库多个表，字段名不同的映射，以及目标和源库表中字段不一样如何设置等 ，还有就是表名以及列名的大小写不一样的话怎么处理 自己琢磨总是不成功。

参考配置文件目录：https://github.com/alswl/yugong/tree/feature/sql-server-to-mysql-overview/src/it/resources/sample_config

from yugong.