A full ecosystem that is built around kafka powered by golang.
Hope it can help you.
+-----------+
| VirtualIP |
+-----------+
|
+--------------+ Alert SOS Dashboard
| | | | | gk
+-------------------------------------------------------------------------------------------+
| | | | | | |
| +----------+ +----------+ | | | |
| | ehaproxy | | ehaproxy | | V | |
| +----------+ +----------+ | | | |
| | | | discovery | | | |
| +----------------+ | +-------------+ |
| | LB | | |
| | | +--------------------+ +--------+ |
| | +---| | election | | watch |
| |keepalive | zookeeper ensemble |-----------| kguard |-------------+ |
| | +---| | | | aggragator | |
| | | +--------------------+ +--------+ | |
| | +-------+ | | |
| | | registry | orchestration | | +- Pub
| +---------------+ |-----------+ +---------+ | | REST |
| | | | | | kateway |--|----|------|
| +---------+ +---------+ +--------+ +--------+ +---------+ | | |
| | kateway | | kateway | | actord | | actord | | | +- Sub
| +---------+ +---------+ +--------+ +--------+ | |
| | hh | | hh | | executor | |
| +----+ +----+ +--------------+ | |
| | | | | |
| | +---------+ +---------+ push | |
| +--------+ | JobTube | | Webhook |------------>-----------------|----|---Endpoints
| | | +---------+ +---------+ | |
| | | | scheduler | sub | |
| auth | |job WAL | dispatch | | |
| +------+ +---------------------------+------------------+ | |
| | | tenant shard | pubsub | flush | |
| +----------+ +---------+ +-------+ +------+ | |
| | auth DB | | DB Farm | | kafka | | TSDB | | |
| +----------+ +---------+ +-------+ +------+ | |
| | | | | | |
| | +----------------------------------------------+ | |
| | | | |
| | +-------------------------------------------+ |
| | |
| | zone |
+-------------------------------------------------------------------------------------------+
|
WebConsole
-
Elastic haproxy that sits in front of kateway.
-
A fully-managed real-time secure and reliable RESTful Cloud Pub/Sub streaming message/job service.
-
kateway job scheduler and webhook dispatcher.
-
Unified multi-datacenter multi-cluster kafka swiss-knife management console.
-
A handy zookeeper CLI that supports recursive operation without any dependency.
-
Kafka clusters body guard that emits health info to InfluxDB and exports key warnings to zabbix for alarting.
export PATH=$PATH:$GOPATH/bin
#========================================
# install go-bindata and go annotations
#========================================
go install github.com/jteeuwen/go-bindata/go-bindata
go install github.com/funkygao/goannotation
#========================================
# install gafka
#========================================
go get github.com/funkygao/gafka
cd $GOPATH/src/github.com/funkygao/gafka
./build.sh -a # build all components
#========================================
# try the gafka command 'gk'
#========================================
gk -h
Currently gafka manages:
- 4 data centers
- 50+ kafka clusters
- 100+ kafka brokers
- 500+ kafka topics
- 2000+ kafka partitions
- 10Billion messages per day
- peak load
- 1Million message per second
- 8TB transfered per hour
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent