Giter Club home page Giter Club logo

open-replicator's Introduction

open-replicator

Open Replicator is a high performance MySQL binlog parser written in Java. It unfolds the possibilities that you can parse, filter and broadcast the binlog events in a real time manner.

releases

1.6.0

release: 2016-09-27
better microsecond timestamp precision, thanks Michel Pigassou
(all datetime2 classes now expose getTimestampValue(), which preserves all the time precision)

1.5.1

release: 2016-09-21
support NULL passwords

1.5.0

release: 2016-09-07
support a "stop-on-EOF" mode that borrows what mysqldump does, exiting when the server has no more
binlog events.

1.4.4

release: 2016-09-01
fixes incorrect nanosecond calcuation when using TIMESTAMP(X), DATETIME(X) and TIME(X) columns in 5.6

1.4.3

release: 2016-06-05
setup @slave_uid session variable for 5.6-compatible slave-uniqueness checks

1.4.2

release: 2016-03-01
Implment server-sent heartbeats and an API to check up on them.

1.4.1

release: 2016-01-29
pickup a couple of upstream fixes -- Gtid event parsing and something around closing the stream

1.4.0

release: 2016-01-06
support GIS extensions via vividsolution's 'jts' library

1.3.6

release: 2015-12-14
bug-fix: Fix issue with binlog_row_image = MINIMAL parsing overruns

1.3.5

release: 2015-11-04
bug-fix: don't emit format description events unless asked for in file replication
bug-fix: support checksums even if the consumer didn't ask for format description events

1.3.4

release: 2015-11-03
bug-fix: support mysql 5.6 file-based replication when the offset is greater than 4

1.3.3

release: 2015-10-30
support 5.6 checksums in file-based replication

1.3.2

release: 2015-10-07
bugfix for a big-endian issue that was corrupting BIT() columns longer than 2 bytes

1.3.1

release: 2015-09-28
provide a helpful error message when an old authentication scheme is detected

1.3.0

released: 2015-09-09
support mysql 5.6 and binlog_checksums

1.2.1

released: 2015-06-16
notify io reader thread on exception from buffered stream reader, so it doesn't wait infinitely

1.2.0

released: 2015-03-08
preserve invalid datetime values (0000-00-00, anyone?) as longs inside the datetime

1.1.2

released: 2015-02-20
expose binlog filename in event stream
rescue EOFException around doParse(); it's normal when the stream is closed

1.0.7

release date: 2014-05-12
support signed tinyint, smallint, mediumint, int, bigint

1.0.6

release date: 2014-05-08
remove dependency commons-lang, log4j
support MYSQL_TYPE_TIMESTAMP2, MYSQL_TYPE_DATETIME2, MYSQL_TYPE_TIME2

1.0.0

release date: 2011-12-29

maven

<dependency>
        <groupId>com.zendesk</groupId>
        <artifactId>open-replicator</artifactId>
        <version>1.3.0</version>
</dependency>

parsers

BinlogEventParser is plugable. All available implementations are registered by default, but you can register only the parsers you are interested in. Alt text

usage

final OpenReplicator or = new OpenReplicator();
or.setUser("root");
or.setPassword("123456");
or.setHost("localhost");
or.setPort(3306);
or.setServerId(6789);
or.setBinlogPosition(4);
or.setBinlogFileName("mysql_bin.000001");
or.setBinlogEventListener(new BinlogEventListener() {
    public void onEvents(BinlogEventV4 event) {
        // your code goes here
    }
});
or.start();

System.out.println("press 'q' to stop");
final BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
for(String line = br.readLine(); line != null; line = br.readLine()) {
    if(line.equals("q")) {
        or.stop();
        break;
    }
}

open-replicator's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

open-replicator's Issues

Failed to parse binlog

Getting following error :

[binlog-parser-1] ERROR com.google.code.or.binlog.impl.AbstractBinlogParser - failed to parse binlog
java.lang.RuntimeException: assertion failed! We left -45 unconsumed bytes in the buffer for event: BinlogEventV4HeaderImpl[timestamp=1477554196000,eventType=24,serverId=1,eventLength=100,nextPosition=41389314,flags=0,timestampOfReceipt=1477562393502]
at com.google.code.or.net.impl.EventInputStream.finishEvent(EventInputStream.java:73)
at com.google.code.or.binlog.impl.ReplicationBasedBinlogParser.doParse(ReplicationBasedBinlogParser.java:166)
at com.google.code.or.binlog.impl.AbstractBinlogParser$Task.run(AbstractBinlogParser.java:263)
at java.lang.Thread.run(Unknown Source)

merge this branch to The mainstream(whitesock/open-replicator )?

@osheroff
Are you going to merge this branch to The mainstream(whitesock/open-replicator)?

I want to add a feature to Maxwell which note down the processed GTID. That needs to process the “GTID_LOG_EVENT”.

Someone has already added the GtidEvent class to “whitesock/open-replicator”, but it is not in this branch.

Does not support null password

In AuthenticationImpl, it throws exception while password is null, but sometimes we may need to use a replicator without a password (from maxwell).

Getting ArrayIndexOutOfBoundsException

Trying to use open replicator to parse MySql bin logs.

java.lang.ArrayIndexOutOfBoundsException: 1
at com.google.code.or.common.glossary.column.BitColumn.get(BitColumn.java:84)
at com.google.code.or.binlog.impl.parser.AbstractRowEventParser.parseRow(AbstractRowEventParser.java:106)
at com.google.code.or.binlog.impl.parser.UpdateRowsEventParser.parseRows(UpdateRowsEventParser.java:76)
at com.google.code.or.binlog.impl.parser.UpdateRowsEventParser.parse(UpdateRowsEventParser.java:65)
at com.google.code.or.binlog.impl.ReplicationBasedBinlogParser.doParse(ReplicationBasedBinlogParser.java:159)
at com.google.code.or.binlog.impl.AbstractBinlogParser$Task.run(AbstractBinlogParser.java:263)
at java.lang.Thread.run(Unknown Source)

Getting com.google.code.or.io.ExceedLimitException when a image inserted into LONGBLOB column

Getting com.google.code.or.io.ExceedLimitException when a image inserted into LONGBLOB column.

  • I inserted a image file of size 48MB.
  • Record inserted successfully.
  • Got the following exceptions while parsing the bin log file using Open-Replicator:
    2017-10-08 14:25:25,272 +41119 ERROR {impl.AbstractBinlogParser} failed to parse binlog
    com.google.code.or.io.ExceedLimitException
    at com.google.code.or.io.impl.XInputStreamImpl.read(XInputStreamImpl.java:224)
    at com.google.code.or.io.impl.XInputStreamImpl.readBytes(XInputStreamImpl.java:70)
    at com.google.code.or.binlog.impl.parser.AbstractRowEventParser.parseRow(AbstractRowEventParser.java:149)
    at com.google.code.or.binlog.impl.parser.WriteRowsEventV2Parser.parseRows(WriteRowsEventV2Parser.java:75)
    at com.google.code.or.binlog.impl.parser.WriteRowsEventV2Parser.parse(WriteRowsEventV2Parser.java:64)
    at com.google.code.or.binlog.impl.ReplicationBasedBinlogParser.doParse(ReplicationBasedBinlogParser.java:128)
    at com.google.code.or.binlog.impl.AbstractBinlogParser$Task.run(AbstractBinlogParser.java:244)
    at java.lang.Thread.run(Thread.java:748)

Following is the Table Definition:
CREATE TABLE MT_FILETEST (
PIC_NAME VARCHAR(40),
PIC_FILE LONGBLOB
)

Here is the insert statement I used to load the file:
insert into MT_FILETEST values('48MB.jpg', LOAD_FILE('/var/lib/mysql-files/48MB.jpg'));

no support unsigned option

we found this plugin is not support unsigned cols, for example: one col is tiny unsigned ,the value may be negative. we hope someone can help us~

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.