Giter Club home page Giter Club logo

pan's Introduction

image-20200803155136931


Background


Pan is a high performance and stable production side agent of messager-oriented middleware written in pure Go language. It supports mainstream message queues in the market, such as Kafka, RabbitMQ, RocketMQ, NSQ, etc. Moreover, it is easy to be extended and can meet different business requirements in the production environment.

Document


Document

中文文档

Framework


The framework of Pan is shown as below.

image-20200803155136931

Quickstart

Produce messages to kafka by Pan.


1. Start zookeeper

./bin/zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties

2. Start kafka

./bin/kafka-server-start /usr/local/etc/kafka/server.properties

3. Create topic

./bin/kafka-topics  --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test

4. Modify config for kafka in Pan

[KafkaProxy]
enable=true
KafkaWaitAll=true
KafkaCompression=true
KafkaPartitioner=round
KafkaProducerTimeout=10
brokers=localhost:9092
sasl=false
user=
password=
valid= //topic whitelist,if empty, all topic can be sended
failMode=retry/save/discard

5. Run

tar -zxvf pan.tar.gz
cd pan/
make
./bin/pan -c ../conf/conf.ini

6. Send Message

package main
 
import (
    "fmt"
    "time"
 
    "github.com/tal-tech/xtools/kafkautil"

    "github.com/spf13/cast"
)
 
func main() {
    t := time.Tick(5 * time.Second)
    count := 0
    for {
        select {
        case <-t:
            count++
            err := kafkautil.Send2Proxy("test", []byte("kafka "+cast.ToString(count)))
            if err != nil {
                fmt.Println(err)
            }
            continue
        }
    }
}

modify conf

[KafkaProxy]
unix=/home/www/pan/pan.sock   //sock in pan
host=localhost:9999  //ip and post pan listen

warn

replace in go.mod

replace github.com/henrylee2cn/teleport v5.0.0+incompatible => github.com/hhtlxhhxy/github.com_henrylee2cn_teleport v1.0.0

或

replace github.com/henrylee2cn/teleport v0.0.0 => github.com/hhtlxhhxy/github.com_henrylee2cn_teleport v1.0.0

Contact us

Contact Us

(微信扫一扫,申请加入开发讨论微信群)

pan's People

Contributors

hhtlxhhxy avatar wgffgw avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pan's Issues

关于Pan研发的意义

Pan研发的背景是什么,它是怎么保证消息不丢失和消息重复的,希望作者给予解答

使用 kafkautil 发送消息失败

环境

  1. mac os
  2. master 分支代码

复现

  1. 使用 kafkautil 发送消息
func SendKafkaMsg() error {
	kconfMap :=  make(map[string][]string)
	kconfMap["host"] = []string{"0.0.0.0:9999"}
	kafkautil.InitKafkautil(kconfMap)
	err := kafkautil.Send2Proxy("test", []byte("kafka"))
	if err != nil {
		fmt.Println(err)
	}
	return err
}

错误

  1. client
2020/12/16 23:12:15 CONF INIT,path:../conf/conf.ini
[23:12:15 CST 2020/12/16] [EROR] (kafkautil.go:165:32030000000066       SocketError      Connect err:EOF msg:[107 97 102 107 97]
[23:12:15 CST 2020/12/16] [EROR] (kafkautil.go:168:32030000000066       SocketError      InvalidSend back: msg:[107 97 102 107 97]
[23:12:15 CST 2020/12/16] [EROR] (kafkautil.go:173:32030000000066       SocketErrorRetry SendMsg err:EOF msg:[107 97 102 107 97] back 
EOF
[23:12:15 CST 2020/12/16] [EROR] (kafkautil.go:173:32030000000066       SocketErrorRetry SendMsg err:EOF msg:[107 97 102 107 97] back 
[23:12:15 CST 2020/12/16] [EROR] (kafkautil.go:173:32030000000066        SocketErrorRetry SendMsg err:EOF msg:[107 97 102 107 97] back 
[23:12:15 CST 2020/12/16] [EROR] (kafkautil.go:179:32030000000066    SocketErrorRetry SendMsg err:EOF msg:[107 97 102 107 97] back:
[23:12:15 CST 2020/12/16] [EROR] (kafkautil.go:179:32030000000066    SocketErrorRetry SendMsg err:EOF msg:[107 97 102 107 97] back:
[23:12:15 CST 2020/12/16] [EROR] (kafkautil.go:179:32030000000066    SocketErrorRetry SendMsg err:EOF msg:[107 97 102 107 97] back:
  1. pan
2020/12/16 14:56:51 CONF INIT,path:/app/conf/conf.ini
LoadConfiguration: Warning: Unknown property "daily" for file filter in conf/log.xml
---------store type is redis--------
-------mqtype is kafka ---------fail mode is retry--------

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.