Config memagement with Nacos about clickhouse_sinker HOT 5 CLOSED

Maybe consul ?

from clickhouse_sinker.

@ArtemChekunov Unfortunately consul is unavailable in our product.

from clickhouse_sinker.

We can abstract the interface of configs, like IConfig, HttpConfig, LocalConfig, NacosConfig

from clickhouse_sinker.

This issue can be combined with #84 .

I suggest following Config structure which all clickhouse_sinker instances share(fetched from Nacos, Consul, ZK etc.):

type Config struct {
	Kafka      map[string]*KafkaConfig
	Clickhouse map[string]*ClickHouseConfig
	Tasks []*TaskConfig
	Common struct {
		FlushInterval     int
		BufferSize        int
		MinBufferSize     int
		MsgSizeHint       int
		ConcurrentParsers int
		LayoutDate        string
		LayoutDateTime    string
		LayoutDateTime64  string
		LogLevel          string
	}
    Assigns map[string][]string  //map instance_name to a list of task_name
}

Each instance can run multiple tasks.
Each task can be assigned to multiple instances. Each task declares how many instances it needs.

The coordinator

• The coordinator (ckman, another project) provides API and/or webui to add/delete/modify tasks.
• The coordinator watches (do service discovery) instance startup/disappear events, and assign tasks to instances (do publish config).

The schedule platform

The schedule platform(or maybe manually, outside my concern) start some clickhouse_sinker instances and start another one if a instance fail.

clickhouse_sinker

• Every clickhouse_sinker instance register itself to a service manager (Nacos, Consul, ZK etc.).
• clickhouse_sinker watches (do get config) cofig, compare its assignment with the current one, apply changes.
• clickhouse_sinker shall try to recover a task in endless loop if the task fail. This ensures the tasks run on an instance match the coordinator assigned.

from clickhouse_sinker.

Implemented with #86.

from clickhouse_sinker.