philchia / agollo Goto Github PK

[agollo] 2020/07/14 19:56:50 [INFO] sync namespace application with remote config server
[agollo] 2020/07/14 19:56:52 [INFO] handle namespace application update
[agollo] 2020/07/14 19:56:52 [INFO] sync namespace application with remote config server

这三句日记一直无线打印。。。。
weChat pywvsai

如图：

目前lock 范围较大，openfile后写文件是有可能存在阻塞，lock 就一直无法得到释放，这样导致其他go程读写cache。
把lock 范围缩小到红色框会更好

此问题由apollo服务端没有正确理解namespace=application/application.properties造成客户端的循环调用问题。

client := agollo.NewClient(&agollo.Conf{
AppID: "agollotest",
Cluster: "default",
NameSpaceNames: []string{"application"},
//NameSpaceNames: []string{"application.properties"},
CacheDir: "c:\tmp",
IP: "10.255.242.227:8080",
})
如果采用没有注释掉的配置，则
notifications/v2?appId=agollotest&cluster=default&notifications=接口
会正常等待 60 second，则agollo客户端是正常的

当采用注释掉的配置时“NameSpaceNames: []string{"application.properties"},”
notifications/v2?appId=agollotest&cluster=default&notifications=接口
会立刻返回，导致agollo客户端循环调用此接口，造成一定的性能损耗

在一些应用场景下，并不知道具体的key值是多少，需要获取到全量的配置来进行遍历。
现在客户端的方法都是在已知key的场景下调用的。
java 的Config提供了public abstract Set getPropertyNames()方法获取所有的key，是否具有参考价值。

While I use GetNameSpaceContent get .properties namespace content by agollo, I would got empty string.

I have been filed an issue to ctripcorp apollo repo, and confirmed that it was support by apollo itself.

Thanks.

see #70
request for comment

我基于agollo弄了一个项目apollo-app，它依赖的agollo的tag明显打了v2.2.0，但是go mod tidy -v的时候只能拉到v2.1.0的tag。不符合期望。

$ go mod init github.com/xujintao/apollo-app
go: creating new go.mod: module github.com/xujintao/apollo-app

$ go mod tidy -v
go: finding github.com/xujintao/agollo v2.1.0+incompatible
go: downloading github.com/xujintao/agollo v2.1.0+incompatible
go: finding github.com/philchia/agollo/internal/mockserver latest
go: finding github.com/philchia/agollo/internal latest
go: finding github.com/philchia/agollo v2.1.0+incompatible
go: downloading github.com/philchia/agollo v2.1.0+incompatible

我现在只能在go mod tidy -v完了以后，继续执行go get github.com/xujintao/[email protected]，才能勉强拉取到v2.2.0

$ go get github.com/xujintao/[email protected]
go: finding github.com/xujintao/agollo v2.2.0
go: finding github.com/mitchellh/mapstructure v1.1.2
go: finding gopkg.in/yaml.v2 v2.2.2
go: finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: downloading github.com/xujintao/agollo v0.0.0-20181228033236-47a518f5a0d1

而这个明显不够友好。

我查阅了相关资料，比如Semantic Import Versioning以及releasing-modules-v2-or-higher
了解到当前agollo已经是>=v2.0.0也就是落到了"v2+"规范。
然而规范我看了半天，没看懂，求指导。
@philchia
@wweir

xxdeMacBook-Pro:ph-agollo-master xx$ go run agollo_test.go go run: cannot run *_test.go files (agollo_test.go)

我通过git clone 下载下来工程，通过上述命令跑了下测试，跑不起来。
希望能够提供一个样例，这样方便使用。

go 的client 通过GetStringValue() 这个函数获取某个key对应的配置，但获取的配置是没有类型的，都是作为string处理，这点是否和其他语言的客户端一致？如果一致，为何不区分类型？

哎。违反我们公司安全红线，配置文件不准超过700

我地址信息和权限都没问题，老报这个错误

Pre-knowledge
Golang adds modules support in version 1.11. It has a semi version calculator in the golang compiler (go binary).
You can find the calculator implementation code via these links:

• https://github.com/golang/go/blob/release-branch.go1.11/src/cmd/go/internal/semver/semver.go#L115
• https://github.com/golang/go/blob/release-branch.go1.11/src/cmd/go/internal/semver/semver.go#L150

The trouble maker is:

Real Problem
While use agollo in a project with go modules, go mod tidy will use master as the default version but not latest stable version 2.1.0.

Suggestion
Release a new version with a tag starting with "v", eg: v2.1.0.

For Now
We can manually lock agollo by running go get github.com/philchia/agollo 2.1.0 in project(go.mod) dirctory.

你好， 运行实例代码报错 the system cannot find the path specified

ENV:

CLIENT SYSTEM OS: WINDOWS SERVER 2016 DATACENTER
CLIENT GO VERSION: 1.14.4

CODE:

func main() {
        apollo := agollo.NewClient(&agollo.Conf{
                AppID: "8ea0afb1e63",
                Cluster: "DEV",
                NameSpaceNames: []string{"myapp"},
                MetaAddr: "http://10.2.1.11:8080",
        })
        if err := apollo.Start(); err != nil {
                fmt.Println(err)
                panic(err)
        }

        ns := agollo.WithNamespace("myapp")
        namespaceContent := agollo.GetContent(ns)
        fmt.Println(namespaceContent)
}

OUTPUT:

panic: mkdir: The system cannot find the path specified.

最近在写配置文件的时候，误以为是一种风格的字符串，造成了长时间的跟踪调试，建议作者采用统一的tag命名方式。 😄

// Conf ...
type Conf struct {
	AppID              string   `json:"appId,omitempty"`
	Cluster            string   `json:"cluster,omitempty"`
	NameSpaceNames     []string `json:"namespaceNames,omitempty"`
	CacheDir           string   `json:"cacheDir,omitempty"`
	MetaAddr           string   `json:"meta_addr"`
	AccesskeySecret    string   `json:"accesskey_secret"`
	InsecureSkipVerify bool     `json:"insecure_skip_verify"`
}

当有多个 apollo server，发布新配置时，客户端可能获取不到最新的配置，并且之后也不会重试。

假设有以下场景：

有 A、B 两个 apollo server，新发布一个配置后，A 率先同步完成，B 正在同步中。此时，如果客户端连接的是 B，便会收到 GET /notifications/v2 的通知，进而执行 GET /configs 请求获取最新配置。但是 B 还没有同步完成，于是返回了一个 304 Not Modified 给客户端。那么客户端便会错过这次更新发布，更糟糕的是，即便在以后的 Long Polling 中，客户端也无法获得最新的配置。

本质上是因为客户端在收到 GET /notifications/v2 的通知后（拿到了 NotificationID），没有正确处理 GET /configs 获取的状态，便更新了客户端的 NotificationID。导致后续的 GET /notifications/v2 请求不会再下发通知，无法触发 GET /configs 请求。

相关代码：

L126 的 handler 此时返回的是 nil，其真正调用的是：

L165 此时返回的是两个 nil，最简单的修复就是让 c.requester.request() 在不是 200 时，返回一个 error: #49

是有什么缺点？

能不能提供一个获取.properties文件全量的key，value的json数据的方法

app.properties:

{
    "appId": "test9910",
    "cluster": "default",
    "namespaceNames": ["application"],
    "ip": "xxx:8080"
}

agollo.Start()
err:

open .test9910_default: The system cannot find the file specified.

But http api is OK!

Why?

监听自定义的集群名称下的命名空间，当default集群下同样名称的命名空间产生变化，会响应200，返回default集群修改的配置命名空间。 代码会调用/config请求去获取自定义集群下面的命名空间的配置变化，因为没有变化返回304。

代码会无线循环此步骤。 /notification请求立即返回200，紧接着立即发起/config请求返回304。

意见：
1、 当notification请求响应200， 立即更改内存里的notificationId值，然后在调用cofing请求
2、同时起个定时任务，定时去拉取最新配置。以防更改通知值后，去掉用config请求失败，导致永远不会与配置中心的值一致

go version 1.13.1

go: finding github.com/philchia/agollo v2.3.1
go: finding github.com/philchia/agollo v2.3.1
go get github.com/philchia/[email protected]: github.com/philchia/[email protected]: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2

see : https://github.com/golang/go/wiki/Modules#semantic-import-versioning
should change the package name when major version greater than 1

func (c *Conf) normalize() {
if !strIn(c.NameSpaceNames, defaultNamespace) {
c.NameSpaceNames = append(c.NameSpaceNames, defaultNamespace)
}
}

在docker中使用时，每次都是新的环境，不需要本地缓存

(mypy3) ➜ automaticzabbix go run autozabbix.go

[agollo] 2021/03/05 10:08:23 [INFO] start agollo client...
[agollo] 2021/03/05 10:08:23 [ERROR] fail to create cache dir: mkdir : no such file or directory
2021/03/05 10:08:23 apollo初始化失败： mkdir : no such file or directory
2021/03/05 10:08:23

func (this *Obnfo) NewApollo() {
	conf := apollo.Conf{
		AppID:              this.Para.LocalInfo[0].Game.Code + "_config",
		Cluster:            "center",
		NameSpaceNames:     []string{"application"},
		AccesskeySecret:    "xxxxx",
		InsecureSkipVerify: false,
		MetaAddr:           "xxxx",
	}
	if err := apollo.Start(&conf); err != nil {
		log.Println("apollo初始化失败：", err)
	}
	value := apollo.GetString("root_pub")
	log.Println(value)
}

go版本：1.16

是不是可以丰富一下Apollo的这个文档啊，看着有点抽象，不知道杂用的

In namespace with properties format of Apollo Portal, I can set multiline content (like indented json) as a value.

Like this:

This is a valid value in properties format, Apollo will store this value to single line by escape newline or quotation.

But if I use client.GetContent() to get content of this namespace, every multiline value will be parsed to multiline, I cannot use any multiline value by parse the result of it.

Mac 上使用这个库，总是报错 open .6e77bd897fe903ad_default: no such file or directory 目测是创建这个文件出错，看代码是根据APPID和namespace 拼接的，实际应用中，可能程序运行的目录没有创建权限，能够公开方法指定保存目录呢？

还有一个就是，我们的Apollo 配置的是 https 访问，我看代码里面是写死的HTTP。

您好，我在文档中有一个 agollo.Unmarshal 的方法，但是在使用时当前的 release 版本好像还没法使用。近期有发版本的打算吗？

// testother 是一个私有的json内容	
agollo.Start(&agollo.Conf{
		AppID:          "147258369",
		Cluster:        "xi-nan",
		NameSpaceNames: []string{"application", "markets", "testother"},
		CacheDir:       ".",
		MetaAddr:       "http://106.54.227.205:8080",
	})
	fmt.Println(agollo.GetString("gggg", agollo.WithNamespace("application")))
	fmt.Println(agollo.GetString("147", agollo.WithNamespace("markets")))
	fmt.Println(agollo.GetContent(agollo.WithNamespace("testother")))

谢了

// NewClient create client from conf
func NewClient(conf *Conf) *Client {
client := &Client{
conf: conf,
caches: newNamespaceCahce(),
releaseKeyRepo: newCache(),

	requester: newHTTPRequester(&http.Client{Timeout: queryTimeout}),
}

client.longPoller = newLongPoller(conf, longPollInterval, client.handleNamespaceUpdate)
client.ctx, client.cancel = context.WithCancel(context.Background())
return client

}

此段代码应该开启 是否略过证书的校验

这是我的代码，用的演示环境的地址：
演示环境（Demo）: 106.54.227.205 账号/密码:apollo/admin
` // Some code here...

agollo.Start(&agollo.Conf{
	AppID:              "my-test",
	Cluster:            "default",
	NameSpaceNames:     []string{"asfdasdfafadsf.properties"},
	CacheDir:           "/tmp",
	AccesskeySecret:    "589e4825d8c2431588d30488d7cbcee1",
	InsecureSkipVerify: true,
	MetaAddr:           "http://106.54.227.205" ,
})

//allKyes := agollo.GetAllKeys(agollo.WithNamespace("application"))
val := agollo.GetString("redis")

fmt.Println("allKyes:", val)`

报错是：
fail to preload open /tmp/.my-test_default: no such file or directory
调试的时候发现，请求的是singin页面去了，secret没有生效？

其实ip应该是config server ，单纯叫ip，好像与配置所代表的业务没关联。

总觉得怪怪得。

See this line , is ip deprecated, and we suggest use MetaAddr?

Thanks.

有实现下面这个接口吗
URL: {config_server_url}/configfiles/{appId}/{clusterName}/{namespaceName}?ip={clientIp}

Hi,

It seems that there is no public field to told us that which config version with the namespace for this time, and apollo actually has the releaseKey as the release version.

The release key can help us easier to compare the version between local config file cache with the remote apollo version, and ONLY update if it's necessary.

Thanks.

Hi,

It seems that we don't handled the following case well:

# app.properties
{
    "appId":"eng-demo",
    "cluster":"default",
    "namespaceNames":["application"],
    "ip":"www.google.com"
}

agollo will hang with start, and no errors output.
Such case could be reproduced while our apollo meta service broken, and the tcp port is still connectable, but no valid results return.

Thanks.

I have been executed go test -v *.go and got errors as follow:

=== RUN   TestAgolloStart
err: open app.properties: no such file or directory
err: open fake.properties: no such file or directory
2019/09/17 17:57:03 [agollo] err preload: http resp code not ok
--- FAIL: TestAgolloStart (0.00s)
    agollo_test.go:46: Start with app.properties should return nil, got :open /tmp/agollo/.SampleApp_default: no such file or directory
=== RUN   TestCache
--- PASS: TestCache (0.00s)
=== RUN   TestCacheDump
--- PASS: TestCacheDump (0.00s)
=== RUN   TestChangeType
--- PASS: TestChangeType (0.00s)
=== RUN   TestMakeDeleteChange
--- PASS: TestMakeDeleteChange (0.00s)
=== RUN   TestMakeModifyChange
--- PASS: TestMakeModifyChange (0.00s)
=== RUN   TestMakeAddChange
--- PASS: TestMakeAddChange (0.00s)
=== RUN   TestLocalIp
--- PASS: TestLocalIp (0.00s)
=== RUN   TestNotificationURL
--- PASS: TestNotificationURL (0.00s)
=== RUN   TestConfigURL
--- PASS: TestConfigURL (0.00s)
=== RUN   TestNewConf
err: open fakename: no such file or directory
--- PASS: TestNewConf (0.00s)
=== RUN   TestNotification
--- PASS: TestNotification (0.00s)
=== RUN   TestRequest
--- PASS: TestRequest (0.00s)
FAIL
2019/09/17 17:57:03 http: Server closed
FAIL	command-line-arguments	0.029s

Thanks.

比如应用连接了两个application，先初始化了APP1，后初始化APP2，.agollo文件里面的内容就只有APP2的配置，APP1的配置被覆盖了，如果APP1的配置有变化，.agollo的内容就又变成了APP1的配置了。
Java版本的client本地缓存文件命名是appid+cluster+namespace.properties，请确认是否可以参考

client.Start() will take some minutes to start when the namespace not exists on Apollo.

The reason is very simple, Start() method will invoke the client.preload() to long polling configuration data, the polling will hang for 1 minute by Apollo server if the namespace is missing.

Is there any way to speed up the client.Start() logic (except add the missing namespace on Apollo)?

My options:

1. For optional namespace, it's no need to pre-load when start;
2. For mandatory namespace, it must be fond from Apollo server or local cache.
   So users could tag the namespace mandatory or optional when invoking client.Start(), client.preload() could invoke Apollo's configs API instead of notifications/v2, the configs API will return failure quickly when access missing optional namespaces, it will speed up the start process.

HTTP 1.1 protocol is used to communicate with configuration center

client.go中的GetPropertiesContent，GetString等方法for循环处理namespace最终只会处理数组的最后一个。
var op = defaultOperation()
for _, opt := range opts {
opt(op)
}

代码中新增了关闭配置，但是debug下 在发送请求还是拿着http包下的client对象发送。

启动客户端之后，调用client的stop方法panic，示例代码：

func newClient(server, cluster, appid string, namespaces ...string) *agollo.Client {
	return agollo.NewClient(&agollo.Conf{
		AppID:          appid,
		Cluster:        cluster,
		NameSpaceNames: namespaces,
		IP:             server,
	})
}

// DecodeFunc 配置解码函数原型
type DecodeFunc func([]byte, interface{}) error

// Open 打开某个配置，conf传入指针类型
func Open(conf interface{}, decode DecodeFunc, server, cluster, appid string, namespaces ...string) error {
	cli := newClient(server, cluster, appid, namespaces...)
	err := cli.Start()
	if err != nil {
		return err
	}
	defer cli.Stop()

	for _, namespace := range namespaces {
		_ = cli.GetNameSpaceContent(namespace, "")
	}
	return nil
}

经过排查，原因是：
poller.go文件的newLongPoller函数在初始化长连接时未初始化cancel方法，而是在长连接start的时候在watchUpdates方法中进行初始化，而start方法又使用了携程，有可能会在客户端调用stop方法的时候cancel还未初始化，所以panic，报错信息如下：

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x123f334]

goroutine 1 [running]:
github.com/philchia/agollo.(*longPoller).stop(0xc0000b42d0)
/Users/yunzhanghu081/go/pkg/mod/github.com/philchia/[email protected]+incompatible/poller.go:85 +0x24
github.com/philchia/agollo.(*Client).Stop(0xc0000a41e0, 0x7, 0x12d2b61)
/Users/yunzhanghu081/go/pkg/mod/github.com/philchia/[email protected]+incompatible/client.go:77 +0x34
main.Open(0x12666a0, 0x14e54c0, 0x0, 0x12d6115, 0x13, 0x12d3492, 0x7, 0x12d2b61, 0x3, 0x0, ...)
/Users/yunzhanghu081/go/src/test/test.go:43 +0x16f
main.main()
/Users/yunzhanghu081/go/src/test/test.go:11 +0xe4

客户端建立长连接随机到其中一个服务端实例上，服务端会每隔1s去轮询是否有新配置，如果有就返回200。客户端收到通知后发起另外一个连接去获取配置，连接随机到其中一个服务端实例，如果这时该实例因为缓存恰好未同步到最新配置，就会返回老配置。

1. feature

Since apollo v1.6.0, the setting of access keys has been supported.

2. how is it now

If you can't get the configuration and access without setting the access key, agollo-client will also encounter the same problem.

3. expect

agollo-client supports access key config.