Giter Club home page Giter Club logo

zettadb / kunlun-storage-081809 Goto Github PK

View Code? Open in Web Editor NEW
31.0 6.0 6.0 253.11 MB

Kunlun-storage is the storage component for Kunlun distributed DBMS. It's developed based on percona-mysql-8.0.x and contains exclusive features used by Kunlun distributed DBMS, performance enhancements and XA transaction crash safety enhancements without which MySQL would not be able to execute XA transactions reliably.

License: Other

CMake 0.79% Shell 0.31% Makefile 0.73% C 15.96% C++ 68.57% Batchfile 0.01% Objective-C 0.52% Perl 0.63% CSS 1.39% CoffeeScript 0.01% JavaScript 9.73% HTML 0.19% Python 0.48% sed 0.01% Roff 0.02% Starlark 0.02% M4 0.01% NASL 0.62% SAS 0.01% DIGITAL Command Language 0.01%
storage-engine mysql percona xa transaction sql storage

kunlun-storage-081809's Introduction

Kunlun-storage originated from percona-mysql-8.0.18-9, and it contains fixes to all known XA bugs in mysql-8.0.18-9. Without such fixes, Kunlun DDC will not be crash safe and may lose committed transactions or be harmed by other serious data consistency errors, in the event of various hardware/software/network failures.

Kunlun-storage also contains features required by the computing node program of Kunlun distributed DBMS, and thus Kunlun Distributed Database Cluster(DDC) requires the use of Kunlun-storage as meta data cluster and storage shards. Finally, we enhanced performance of XA transaction processing, and part of such enhancements are also in this open source edition.

To achieve all above, we modified percona-mysql extensively --- including innodb, binlog recovery, binlog format, etc. Consequently, kunlun-storage's innodb data file format and some binlog events format are different from community MySQL-8.0.x or percona-server-8.0.x, the data directory of kunlun-storage can not be used by community MySQL-8.0.x or percona-server-8.0.x, and vice versa. But percona xtrabackup can correctly backup a kunlun-storage data directory and restore it.

We also maintain an enterprise edition of Kunlun distributed DBMS, which contains exclusively all performance enhancements in kunlun-storage and kunlun computing node software. Kunlun enterprise edition has identical functionality as this open source version. And they share the same data file format, WAL(redo) log file format, binlog events format, general log format, slow query log format, mysqld log format, and config file content. They also share the same metadata table format in both kunlun-storage and kunlun computing node. Consequently, the data directory of kunlun open source edition and kunlun enterprise edition can be used interchangably.

To install Kunlun-storage database instances, see INSTALL.kunlun.md for instructions.


Percona Server 8.0
------------------

Percona Server is a branch of MySQL 8.0 bringing higher performance,
reliability and more features.

http://www.percona.com/software/percona-server/

Documentation: http://www.percona.com/doc/percona-server/8.0

Bug Tracker: https://jira.percona.com/projects/PS

Sources: https://github.com/percona/percona-server

kunlun-storage-081809's People

Contributors

chaojie1979 avatar david-zhao avatar jd-zhang avatar snowao avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

kunlun-storage-081809's Issues

Can not purge storage binary log remotely

Issue migrated from trac ticket # 145

component: kunlun-storage | priority: major | resolution: fixed

2021-08-25 10:29:07: @jd-zhang created the issue


Currently, we use pgx as user name for storage nodes to accept connections from computing nodes remotely, and also use this account to accept other admin connections.

Sometimes we need to admin the binary logs, but this account does not have the priority to do this.

We need to add BINLOG_ADMIN priority to this user.

Table distribution imbalance

Issue migrated from trac ticket # 11

component: kunlun-storage | priority: major

2021-03-15 11:35:42: @jd-zhang created the issue


The cluster:
One meta shard and two data shards.

  • Create 15 tables
  • first shard has 9 tables
  • second shard has 6 tables
  • create 255 tables, and all these tables fall into the same data shard.

The command to create 255 table is:
i=0
while test $i -lt 255; do psql -h localhost -p 5401 -Ukunlun postgres -c "create table batch$i(id integer primary key, f1 int)"; let i++; done

MTR-FIX [main.log_tables]

Issue migrated from trac ticket # 350

component: kunlun-storage | priority: major

2021-12-22 16:25:12: @snowao created the issue


Additional fields added like extra_info in mysql.general_log table which is attributed to the test case failed.

The type 'timetz' does not work correctly

Issue migrated from trac ticket # 104

component: kunlun-storage | priority: major

2021-06-16 10:54:35: @jd-zhang created the issue


sql code:

DROP TABLE if exists TIMETZ_TBL;
CREATE TABLE TIMETZ_TBL (f1 time(2) with time zone);
INSERT INTO TIMETZ_TBL VALUES ('00:01 PDT');
INSERT INTO TIMETZ_TBL VALUES ('01:00 PDT');
INSERT INTO TIMETZ_TBL VALUES ('02:03 PDT');
INSERT INTO TIMETZ_TBL VALUES ('07:07 PST');
INSERT INTO TIMETZ_TBL VALUES ('08:08 EDT');
INSERT INTO TIMETZ_TBL VALUES ('11:59 PDT');
INSERT INTO TIMETZ_TBL VALUES ('12:00 PDT');
INSERT INTO TIMETZ_TBL VALUES ('12:01 PDT');
INSERT INTO TIMETZ_TBL VALUES ('23:59 PDT');
INSERT INTO TIMETZ_TBL VALUES ('11:59:59.99 PM PDT');
INSERT INTO TIMETZ_TBL VALUES ('2003-03-07 15:36:39 America/New_York');
INSERT INTO TIMETZ_TBL VALUES ('2003-07-07 15:36:39 America/New_York');

select * from TIMETZ_TBL;
tdb1=# select * from TIMETZ_TBL;
f1
00:01:00+00
01:00:00+00
02:03:00+00
07:07:00+00
08:08:00+00
11:59:00+00
12:00:00+00
12:01:00+00
23:59:00+00
23:59:59.99+00
15:36:39+00
15:36:39+00

All the timezone info is removed when putting into the table.

mysqld crash when shutdown using debug version

Issue migrated from trac ticket # 241

component: kunlun-storage | priority: minor | resolution: cant-reproduce

2021-11-23 10:00:36: @jd-zhang created the issue


When using mysqldmin to shutdown the instance, the process crashes then quits, instead of normal quits.

stack:

(gdb) bt
#0 0x00007fc0b80f88d5 in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000044c76b2 in my_write_core (sig=6) at /home/kunlun/debugbuild/percona-server-8.0.18-9/mysys/stacktrace.cc:305
#2 0x000000000343136d in handle_fatal_signal (sig=6) at /home/kunlun/debugbuild/percona-server-8.0.18-9/sql/signal_handler.cc:172
#3
#4 0x00007fc0b61bf37f in raise () from /lib64/libc.so.6
#5 0x00007fc0b61a9db5 in abort () from /lib64/libc.so.6
#6 0x00000000048fd16e in ut_dbg_assertion_failed (
expr=0x646ecc8 "buf_flush_active_lru_managers() ## srv_buf_pool_instances || buf_flush_active_lru_managers() 0",
file=0x646dee8 "/home/kunlun/debugbuild/percona-server-8.0.18-9/storage/innobase/srv/srv0start.cc", line=3531)
at /home/kunlun/debugbuild/percona-server-8.0.18-9/storage/innobase/ut/ut0dbg.cc:98
#7 0x0000000004874ce5 in srv_shutdown_page_cleaners ()
at /home/kunlun/debugbuild/percona-server-8.0.18-9/storage/innobase/srv/srv0start.cc:3530
#8 0x00000000048756c9 in srv_shutdown () at /home/kunlun/debugbuild/percona-server-8.0.18-9/storage/innobase/srv/srv0start.cc:3759
#9 0x00000000045b2981 in innodb_shutdown ()
at /home/kunlun/debugbuild/percona-server-8.0.18-9/storage/innobase/handler/ha_innodb.cc:1621
#10 0x0000000002e20e6b in ha_finalize_handlerton (plugin=0x7fc0a3feabb8)
at /home/kunlun/debugbuild/percona-server-8.0.18-9/sql/handler.cc:746
#11 0x0000000003267dd7 in plugin_deinitialize (plugin=0x7fc0a3feabb8, ref_check=true)
at /home/kunlun/debugbuild/percona-server-8.0.18-9/sql/sql_plugin.cc:1101
#12 0x00000000032684a1 in reap_plugins () at /home/kunlun/debugbuild/percona-server-8.0.18-9/sql/sql_plugin.cc:1176
#13 0x000000000326b1ed in plugin_shutdown () at /home/kunlun/debugbuild/percona-server-8.0.18-9/sql/sql_plugin.cc:2017
#14 0x00000000030148c4 in clean_up (print_message=true) at /home/kunlun/debugbuild/percona-server-8.0.18-9/sql/mysqld.cc:2292
#15 0x0000000003025b18 in mysqld_main (argc=185, argv=0x7fc0b58c31b8)
at /home/kunlun/debugbuild/percona-server-8.0.18-9/sql/mysqld.cc:7514
#16 0x0000000002dffa17 in main (argc=10, argv=0x7fff9dfcd668) at /home/kunlun/debugbuild/percona-server-8.0.18-9/sql/main.cc:25

store procedure not support for and while

Issue migrated from trac ticket # 154

component: kunlun-storage | priority: major | resolution: fixed

2021-09-06 19:19:45: @charleszettadb created the issue


CREATE OR REPLACE FUNCTION LOOP1()
RETURNS void
AS $$
DECLARE
i integer=1;
begin
for i in 1..6 LOOP
RAISE NOTICE '%',i;
END LOOP;
END;
$$ LANGUAGE PLPGSQL;
ERROR: syntax error at or near "1"
LINE 9: for i in 1..6 LOOP

  1.                 ^
    

CREATE OR REPLACE FUNCTION LOOP1()
RETURNS void
AS $$
DECLARE
a integer=1;
begin
while a <= 6 loop
a = a+1
select n;
END LOOP;
END;
$$ LANGUAGE PLPGSQL;
ERROR: syntax error at or near "select"
LINE 9: select n;
^

MTR-FIX [main.mysql_client_test]

Issue migrated from trac ticket # 339

component: kunlun-storage | priority: major

2021-12-21 12:52:49: @snowao created the issue


File_query_log::write_general() will not append '\n' to the tail of the output string when no extrainfo to be print.\
That is the reason why the test case failed.

create sequences table in kunlun_sysdb

Issue migrated from trac ticket # 237

component: kunlun-storage | priority: major

2021-11-22 10:15:04: @david-zhao created the issue


In kunlun-storage-8.0.26, create the kunlun_sysdb database when installing a kunlun-storage instance, and create the heartbeat table and the below sequences table in kunlun_sysdb.

CREATE TABLE IF NOT EXISTS sequences( db varchar(512) not null, name varchar(512) not null, curval bigint not null, start bigint not null, step int not null, max_value bigint not null, min_value bigint not null, do_cycle bool not null, n_cache int unsigned not null, primary key(db,name) ) engine=innodb CHARACTER SET utf8 COLLATE utf8_general_ci STATS_PERSISTENT=0 comment='Kunlun sequence metadata used by computing nodes.' ROW_FORMAT=DYNAMIC

XA COMMIT ONE PHASE self-blocks at rotation

Issue migrated from trac ticket # 140

component: kunlun-storage | priority: major | keywords: binlog;XA COMMIT ONE PHASE;rotation

2021-08-19 09:51:12: @david-zhao created the issue


This issue only exists in open source version of kunlun-storage.

If binlog rotation is initiated by a XA COMMIT ONE PHASE transaction commit, the worker thread blocks itself.

This is caused by two fixes which make sure XA COMMIT ONE PHASE txns are crash safe. Such fixes are correct but not sufficient --- we have to avoid doing rotation at XA COMMIT ONE PHASE commits.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.