Giter Club home page Giter Club logo

cectc / dbpack Goto Github PK

View Code? Open in Web Editor NEW
349.0 8.0 45.0 3.07 MB

A db proxy for distributed transaction, read write splitting and sharding! Support any language! It can be deployed as a sidecar in a pod.

Home Page: https://cectc.github.io/dbpack-doc/#/en-us/

License: Apache License 2.0

Go 99.56% Makefile 0.32% Dockerfile 0.02% Shell 0.11%
distributed-transaction read-write-splitting sharding at database-mesh tcc database event-driven micro-services microservice

dbpack's Introduction

DBPack

LICENSE GitHub Workflow Status Go Report Card codecov GitHub Release Docker Pulls

dbpack.svg

DBPack is a database proxy pack aims to provide a distributed transaction solution of high performance and has no invasion into business logic. It can be deployed as a sidecar in a pod, which can shield complex basic logic, so that business development does not need to rely on a specific SDK, simplifying the development process and improving development efficiency. It supports any programming language and is easy to use.

Feature

  • support MYSQL protocol.
  • event driven distributed transaction solution inspired by kubernetes.
  • support read write splitting and custom SQL routing via Hint.
  • support audit log and SQL tracing.
  • sharding: support query on sharded table, support order by and limit.
  • sharding: support insert, delete, update on sharded tables.
  • can be deployed as a sidecar,support any programming language.
  • more features on the road.

Metrics

using prometheus metrics, default metric server runs at :18888

Getting Started

Requirements

  • Go >= 1.17
  • MYSQL >= 5.7+

Contribution Guide

Architecture

architecture

  • Listener: responsible for parsing the SQL protocol.
  • Executor: responsible for forwarding SQL requests to the backend physical DB.
  • Filter: responsible for metrics statistics, SQL interception, encryption and decryption of sensitive information, etc.
  • ConnectionFilter: process the intercepted SQL on the connection.

Stargazers

Stargazers repo roster for cectc/dbpack

License

DBPack is licensed under the Apache License, Version 2.0.

dbpack's People

Contributors

bohehe avatar dependabot[bot] avatar dk-lockdown avatar fatelei avatar github-actions[bot] avatar gorexlv avatar hehe-rakuten avatar hzliangbin avatar juwanxu avatar rocymp avatar tanryberdi avatar wybrobin avatar yx9o avatar zackzhangkai avatar zhu733756 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dbpack's Issues

.net samples

Is your feature request related to a problem? Please describe.
.net samples for distributed transaction.

unit test for `pkg/misc/time.go`

Is your feature request related to a problem? Please describe.

test func func FormatTimeMillis(tsMillis uint64) string and func func FormatDate(tsMillis uint64) string

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

unit test for `pkg/misc/other.go`

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

unit test for `pkg/misc/escape.go`

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

support avg(field) on sharded table

Is your feature request related to a problem? Please describe.
if topology is like that:

db: drug_0, table: drug_resource_0, drug_resource_1, drug_resource_2, drug_resource_3, drug_resource_4
db: drug_1, table: drug_resource_5, drug_resource_6, drug_resource_7, drug_resource_8, drug_resource_9

drug_resource table contains columns: id, name, code, purchase_price, etc.

when query select avg(purchase_price) from drug_resource where id > 15 and id <18, should rewrite the sql, such as select purchase_price from drug_resource_5 where id > 15 and id < 18, query on every shard, then calculate each result.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

php samples

Is your feature request related to a problem? Please describe.
php samples for distributed transaction.

feature: metrics for distributed transaction

global_transaction_count{appid="", transactionname=""} 0
global_transaction_commit_count{appid="", transactionname=""} 0
global_transaction_rollback_count{appid="", transactionname=""} 0
branch_transaction_count{appid="", resourceid=""} 0
branch_transaction_commit_count{appid="", resourceid=""} 0
branch_transaction_rollback_count{appid="", resourceid=""} 0

support min(field)、max(field) on sharded table

Is your feature request related to a problem? Please describe.
if topology is like that:

db: drug_0, table: drug_resource_0, drug_resource_1, drug_resource_2, drug_resource_3, drug_resource_4
db: drug_1, table: drug_resource_5, drug_resource_6, drug_resource_7, drug_resource_8, drug_resource_9

drug_resource table contains columns: id, name, code, purchase_price, etc.

when query select min(purchase_price) from drug_resource where id > 15 and id <18, should rewrite the sql, such as select min(purchase_price) from drug_resource_5 where id > 15 and id < 18, query on every shard, then calculate each result.

max same as min

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

feature: support `show tables`

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

add terminationDrainDuration to avoid unpaid requests to fail

  • I have searched the issues of this repository and believe that this is not a duplicate.

Describe the bug
In sidecar model, Kubernetes will send SIGTERM signals to all containers at the same time before destroying the pod, so dbpack container will stop at the same time as business containers, this may cause unpaid requests to fail.

I think DBPack should waiting for a length of time (terminationDrainDuration) before exiting can solve this problem. Just like Istio-Proxy does.

related code(cmd/cmd.go#L177)
`
dbpack.Start()

		ctx, cancel := context.WithCancel(context.Background())
		c := make(chan os.Signal, 2)
		signal.Notify(c, os.Interrupt, syscall.SIGTERM)
		go func() {
			<-c
			cancel()
			<-c
			os.Exit(1) // second signal. Exit directly.
		}()

		<-ctx.Done()

`

feature: support tcc mode distributed transaction

Describe the solution you'd like
There is a DistributedTransactionManager

func (manager *DistributedTransactionManager) branchCommit(bs *api.BranchSession) (api.BranchSession_BranchStatus, error) {
	db := resource.GetDBManager().GetDB(bs.ResourceID)
	if db == nil {
		return 0, fmt.Errorf("DB resource is not exist, db name: %s", bs.ResourceID)
	}

	if err := GetUndoLogManager().DeleteUndoLogByXID(db, bs.XID); err != nil {
		return api.PhaseTwoCommitting, err
	}
	if err := manager.storageDriver.DeleteBranchSession(context.Background(), bs.BranchID); err != nil {
		log.Error(err)
	}
	log.Debugf("branch session committed, branch id: %s, lock key: %s", bs.BranchID, bs.LockKey)
	return api.Complete, nil
}

When calling branchCommit method, should judge the type of BranchSession, if the branch type is api.TCC, should send http request to CommitRequestPath, the CommitRequestPath is serialized into ApplicationData of BranchSession.

When calling branchRollback should to the same thing as above.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

feature: support query from global table

Is your feature request related to a problem? Please describe.
In sharding mode, global tables are the same on every db

Describe the solution you'd like
Support query from global table

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

unit test for `pkg/misc/sql.go`

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

feature: support `show databases`

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Python samples

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

/assign @zhu733756

feature: metrics for sql query\execute

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

test: unit test for `pkg/filter/dt/executor.go`

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

feature: support multiple columns `order by` when query sharded table

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Currently only one column order by is supported

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

unexpected make build result

As described in the [compile section](https://cectc.github.io/dbpack-doc/#/getting_started?id=%e7%bc%96%e8%af%91) of getting started doc, a binary file is expected to be built in the dist directory when make build executed. When i run make build on my MacBook, a binary file is built in the project root directory and the file can not be executed because of an exec format error. I check the makefile and find the build doesn't work as it's expected to.

build:  ## build dbpack cli, and put in dist dir
	GOOS="linux"  GOARCH="amd64" CGO_ENABLED=0 go build -o dbpack ./cmd

feature: distributed transaction support `ComQuery` request

Is your feature request related to a problem? Please describe.
refer to

// todo process distributed transaction of comQuery request

in func PreHandle()

// todo process distributed transaction of comQuery request

in func PostHandle()

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

unit test for `pkg/misc/xid.go`

Is your feature request related to a problem? Please describe.
test func GenerateXID() and func GetTransactionID()

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

test: unit test for `pkg/function/function.go`

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

feature: `YearMod ` shard algorithm

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
implement ShardingAlgorithm interface.

type ShardingAlgorithm interface {
	HasShardingKey(key string) bool
	Shard(condition *KeyCondition) (Condition, error)
	ShardRange(cond1, cond2 *KeyCondition) (Condition, error)
	AllShards() Condition
	AllowFullScan() bool
}

type YearMod struct {

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

support count(*) on sharded table

Is your feature request related to a problem? Please describe.
if topology is like that:

db: drug_0, table: drug_resource_0, drug_resource_1, drug_resource_2, drug_resource_3, drug_resource_4
db: drug_1, table: drug_resource_5, drug_resource_6, drug_resource_7, drug_resource_8, drug_resource_9

drug_resource table contains columns: id, name, code, purchase_price, etc.

when query select count(*) from drug_resource where id > 15 and id <18, should rewrite the sql, query on every shard, then sum each result.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

feature: support http2 proxy for distributed transaction

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

feature: delete on sharded table

Is your feature request related to a problem? Please describe.
Depending on the where condition, the delete request may need to be executed on a different DB.
You can refer to the sharding derivation method of the query request.

feature: support sql audit log

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
there is a sql audit log specific, https://support.huaweicloud.com/usermanual-rds/rds_download_sql_auditing_log.html

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

SET @@session.autocommit = OFF lead server panic for interface type assertion.

  • I have searched the issues of this repository and believe that this is not a duplicate.

Describe the bug
SET @@session.autocommit = OFF lead server panic for interface type assertion.

To Reproduce
Steps to reproduce the behavior:

  1. run dbpack from command line
  2. run python-samples order demo

Expected behavior
SET @@session.autocommit = OFF execute well, not panic the server.

Screenshots

image

Additional context
Add any other context about the problem here.

HttpDistributedTransaction filter support prefix matching

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

  • match TransactionInfo RequestPath
  • match TCCResource PrepareRequestPath

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

feature: support PostgreSql communication protocol

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

step1: support pgsql client connect to dbpack

step2: Forward pgsql request to PostgreSql database

unit test for `pkg/cond/condition.go`

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

help us improve unit test coverage,please refer to https://app.codecov.io/gh/CECTC/dbpack/blob/dev/pkg/cond/condition.go

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

feature: `MonthMod` shard algorithm

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

implement ShardingAlgorithm interface.

type ShardingAlgorithm interface {
	HasShardingKey(key string) bool
	Shard(condition *KeyCondition) (Condition, error)
	ShardRange(cond1, cond2 *KeyCondition) (Condition, error)
	AllShards() Condition
	AllowFullScan() bool
}

type MonthMod struct {

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

etcd client init problem

  • I have searched the issues of this repository and believe that this is not a duplicate.

Describe the bug
if we not start the etcd server, the code below

client, err := clientv3.New(config)

will connect the etcd background, so the dbpack server will hang.

Expected behavior
if the etcd server not start, dbpack should panic soon

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

feature: insert on sharded table

Is your feature request related to a problem? Please describe.

If the primary key is not specified when inserting, the distributed primary key is used to insert.

test: unit test for `pkg/misc/strings.go`

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

HttpDistributedTransaction filter support regular matching

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

  • match TransactionInfo RequestPath
  • match TCCResource PrepareRequestPath

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

feature: encryption and decryption of sensitive information

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

feature: merge result support limit without orderby

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

log.Panic("unsupported limit without order by")

such as : select id, name, age from employees where id > 1000 limit 100,10

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

feature: update on sharded table

Is your feature request related to a problem? Please describe.

Depending on the where condition, the update request may need to be executed on a different DB.
You can refer to the sharding derivation method of the query request.

lint action run failed

  • [√] I have searched the issues of this repository and believe that this is not a duplicate.

Describe the bug
same as title

Expected behavior
run success!

unit test for `pkg/misc/misc.go`

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Are you using DBPack?

Are you using DBPack?

If you are using DBPack, first we would like to Thank You. Our goal is to grow the community, improve DBPack and help each other.

The purpose of this issue

We are always interested in finding out who is using DBPack, what attracted you to using it, how we can listen to your needs and if you are interested, help promote your organization.

  • We have people reaching out to us asking, who uses DBPack in production?
  • We’d like to listen to what you would like to see in DBPack and your scenarios?
  • We'd like to help promote your organization and work with you

What we would like from you

Submit a comment in this issue to include the following information:

  • Organization: CECTC (Required)
  • Location: wuhan, hubei, China(Required)
  • Usage:used to solving distributed transaction problem、read write splitting、sharding (Required)
  • TransactionMode: AT\TCC
  • Version: 0.1.0(Optional)
  • Icon (Preferred)

Thanks again for your participation!

欢迎使用 DBPack,首先感谢您的使用,其次您可以参考下面的样例来提供您的信息以收集使用场景帮助提升改进:

* 组织:**电子云(Required)
* 地点:**武汉(Required)
* 用途:解决分布式事务问题、读写分离、分库分表(Required)
* 事务模式:AT\TCC
* 版本: 0.1.0(Optional)
* 公司图标:(Preferred)

feature: support read config from etcd

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

watch config key in etcd for dynamic update.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

distributed primary key for insert on sharded table.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.