leoplatform / connectors Goto Github PK

Description:
While working on a dev database with no changes happening in the db, the listener checkpoint gets stuck on a binlog file that doesn't exist anymore. This causes a rogue that makes it hard to develop because you have to reset the checkpoint every hour or so to restart the listener.

How to reproduce:

1. If I do show master logs; I get this:
   Log_name File_size
   mysql-bin-changelog.238847 578
   mysql-bin-changelog.238848 578
   mysql-bin-changelog.238849 578
   mysql-bin-changelog.238850 521

2. I set the checkpoint to start at the earliest log position that exists. mysql-bin-changelog.238847::0. Then restart the bot.

3. If I make a change to a table that is being listened to, the listener works as expected and catches up to the last log location. The checkpoint points to a location in the last file mysql-bin-changelog.238850

4. After an amount of time, the listener goes into rogue status with the error.
   Error: ER_MASTER_FATAL_ERROR_READING_BINLOG: Could not find first log file name in binary log index file at Binlog.Sequence._packetToError (/var/task/index.js:84094:14) at Binlog.Sequence.ErrorPacket (/var/task/index.js:84143:17) at Protocol._parsePacket (/var/task/index.js:79237:23) at Parser.write (/var/task/index.js:78556:12) at Protocol.write (/var/task/index.js:78997:16) at Socket. (/var/task/index.js:76656:28) at emitOne (events.js:116:13) at Socket.emit (events.js:211:7) at addChunk (_stream_readable.js:263:12) at readableAddChunk (_stream_readable.js:250:11) -------------------- at Protocol._enqueue (/var/task/index.js:79103:48) at Immediate._start [as _onImmediate] (/var/task/index.js:20906:29) at runCallback (timers.js:794:20) at tryOnImmediate (timers.js:752:5) at processImmediate [as _immediateCallback] (timers.js:729:5) code: 'ER_MASTER_FATAL_ERROR_READING_BINLOG', errno: 1236, sqlMessage: 'Could not find first log file name in binary log index file', sqlState: 'HY000'

5. Doing show master logs; indicates that the checkpoint location points to a file that has already been rotated out and deleted.

Expected behavior:
Our install is mysql setup on RDS with a standard configuration. It rotates files pretty quickly, even though there is no activity. I realize that I can change mysql parameters to not rotate and delete as quickly, and that this wouldn't be an issue on production. But I would expect the listener not to get stuck regardless of inactivity... Among other issues, it wastes a lot of time during development to have to reset the checkpoint.