This repository contains the organization profile.
guancecloud / datakit Goto Github PK
View Code? Open in Web Editor NEWDataKit is collection agent for Guance Cloud
License: MIT License
DataKit is collection agent for Guance Cloud
License: MIT License
Hello,按照这个文档想接入更多的prometheus数据 https://docs.guance.com/integrations/prom/#prometheus-exporter
配置了两个外部地址
monitor上也看到了响应的input
但是观测云指标集里一直看不到数据上来
用了 datakit debug --prom-conf prom.conf
能看到有input,datakit容器没有任何的错误日志,datakit_filter_point_dropped_total
似乎默认也没有被采集无法查询到
为了防止1000个指标过多,relabel到17个之后还是不行
请问还有别的排障手段吗,datakit跟gateway的网络没问题,其他指标一直都正常写入
kafkamq 提供一种插件机制:将数据([]byte)通过 HTTP 发送到外部handle,经过处理后再通过response返回行协议的json格式数据。实现定制化数据。
增加如下配置:(以最终配置为准)
http url
stringmessage_points
int 一次发送的消息点数debug
bool 值, 当开启debug功能, message_points
则无效,如果开启debug模式,则将原始byte数据发送,不再进行消息合并。threads
int 多线程工作is_response_point
是否将行协议数据发送回来pipeline
脚本header_check
特殊的头部检测(bfy定制化,并非通用)外部插件有一些约束:
Host Environment: MacBook 12.0.1 (21A559) with Apple M1
Dockerfile: Attached in the thread
installation logs: Attached below
How to reproduce:
root@dd2e01a6b885:/# ./install.sh
Detect installer installer-linux-arm64-1.1.9-rc7
* Downloading installer /tmp/dk-installer
##################################################################################################################################################################################################### 100.0%
* Installing DataKit...
tee: /var/log/datakit/install.log: No such file or directory
2021-12-19T04:26:11.306Z INFO installer installer/main.go:245 stoping datakit...
2021-12-19T04:26:11.310Z WARN installer installer/main.go:247 stop service: Failed to stop datakit: exit status 1, ignored
2021-12-19T04:26:11.310Z DEBUG installer installer/main.go:659 deprecated install path /usr/local/cloudcare/dataflux/datakit not exists, ignored
Downloading(datakit)... 17 MB/17 MB
Downloading( data)... 34 MB/34 MB
2021-12-19T04:26:53.322Z INFO installer installer/main.go:269 Installing version 1.1.9-rc7...
2021-12-19T04:26:53.322Z WARN installer installer/main.go:348 uninstall service: Failed to uninstall datakit: remove /etc/init.d/datakit: no such file or directory, ignored
2021-12-19T04:26:53.323Z DEBUG dataway dataway/dataway.go:208 httpCli: 0x4000244a00
2021-12-19T04:26:53.323Z INFO installer installer/main.go:498 set cloud provider to ok
2021-12-19T04:26:53.324Z DEBUG installer installer/main.go:501 main config:
default_enabled_inputs = ["cpu", "disk", "diskio", "mem", "swap", "system", "hostobject", "net", "host_processes", "container"]
install_date = 2021-12-19T04:26:53Z
upgrade_date = 0001-01-01T00:00:00Z
namespace = ""
install_version = "1.1.9-rc7"
protect_mode = true
enable_election = false
[[black_lists]]
hosts = []
inputs = []
[[white_lists]]
hosts = []
inputs = []
[dca]
enable = false
listen = "0.0.0.0:9531"
white_list = []
[http_api]
rum_origin_ip_header = "X-Forwarded-For"
listen = "localhost:9529"
disable_404page = false
rum_app_id_white_list = []
[io]
feed_chan_size = 1024
high_frequency_feed_chan_size = 2048
max_cache_count = 1024
cache_dump_threshold = 512
max_dynamic_cache_count = 1024
dynamic_cache_dump_threshold = 512
flush_interval = "10s"
output_file = ""
output_file_inputs = []
enable_cache = false
[dataway]
urls = ["https://aws-openway.guance.com?token=tkn_I_hope_there_is_no_revision_history"]
timeout = "5s"
http_proxy = ""
max_fail = 20
[logging]
log = "/var/log/datakit/log"
gin_log = "/var/log/datakit/gin.log"
level = "info"
disable_color = false
rotate = 32
[global_tags]
cluster = ""
project = ""
site = ""
[environments]
ENV_HOSTNAME = ""
[cgroup]
enable = true
cpu_max = 30.0
cpu_min = 5.0
[tracer]
trace_enabled = false
host = ""
port = ""
service = ""
version = ""
logs_startup = false
debug = false
env = ""
[git_repos]
pull_interval = "1m"
[[git_repos.repo]]
enable = false
url = ""
ssh_private_key_path = ""
ssh_private_key_password = ""
branch = "master"
2021-12-19T04:26:53.324Z INFO config config/cfg.go:520 here is hostname:
2021-12-19T04:26:53.324Z INFO config config/cfg.go:523 set hostname to dd2e01a6b885
2021-12-19T04:26:53.324Z INFO installer installer/main.go:449 installing service datakit...
2021-12-19T04:26:53.328Z INFO installer installer/main.go:274 starting service datakit...
2021-12-19T04:26:53.355Z DEBUG config config/cfg.go:839 remove link /usr/local/bin/datakit...
2021-12-19T04:26:53.355Z WARN config config/cfg.go:841 remove /usr/local/bin/datakit: no such file or directory, ignored
2021-12-19T04:26:53.355Z DEBUG config config/cfg.go:839 remove link /usr/local/sbin/datakit...
2021-12-19T04:26:53.355Z WARN config config/cfg.go:841 remove /usr/local/sbin/datakit: no such file or directory, ignored
2021-12-19T04:26:53.355Z DEBUG config config/cfg.go:839 remove link /sbin/datakit...
2021-12-19T04:26:53.355Z WARN config config/cfg.go:841 remove /sbin/datakit: no such file or directory, ignored
2021-12-19T04:26:53.356Z DEBUG config config/cfg.go:839 remove link /usr/sbin/datakit...
2021-12-19T04:26:53.356Z DEBUG config config/cfg.go:839 remove link /usr/bin/datakit...
2021-12-19T04:26:53.356Z WARN config config/cfg.go:841 remove /usr/bin/datakit: no such file or directory, ignored
2021-12-19T04:26:53.356Z INFO installer installer/main.go:289 :) Install Success!
Visit http://localhost:9529/man/changelog to see DataKit change logs.
Visit http://localhost:9529/monitor to see DataKit running status.
Visit http://localhost:9529/man to see DataKit manuals.
root@dd2e01a6b885:/# echo $?
1
root@dd2e01a6b885:/# ./install.sh
Detect installer installer-linux-arm64-1.1.9-rc7
* Downloading installer /tmp/dk-installer
##################################################################################################################################################################################################### 100.0%
* Installing DataKit...
2021-12-19T04:27:35.719Z INFO installer installer/main.go:245 stoping datakit...
2021-12-19T04:27:35.728Z DEBUG installer installer/main.go:659 deprecated install path /usr/local/cloudcare/dataflux/datakit not exists, ignored
Downloading(datakit)... 17 MB/17 MB
Downloading( data)... 34 MB/34 MB
2021-12-19T04:28:45.151Z INFO installer installer/main.go:269 Installing version 1.1.9-rc7...
2021-12-19T04:28:45.157Z DEBUG dataway dataway/dataway.go:208 httpCli: 0x4000132140
2021-12-19T04:28:45.158Z INFO installer installer/main.go:498 set cloud provider to ok
2021-12-19T04:28:45.160Z DEBUG installer installer/main.go:501 main config:
default_enabled_inputs = ["cpu", "disk", "diskio", "mem", "swap", "system", "hostobject", "net", "host_processes", "container"]
install_date = 2021-12-19T04:28:45Z
upgrade_date = 0001-01-01T00:00:00Z
namespace = ""
install_version = "1.1.9-rc7"
protect_mode = true
enable_election = false
[[black_lists]]
hosts = []
inputs = []
[[white_lists]]
hosts = []
inputs = []
[dca]
enable = false
listen = "0.0.0.0:9531"
white_list = []
[http_api]
rum_origin_ip_header = "X-Forwarded-For"
listen = "localhost:9529"
disable_404page = false
rum_app_id_white_list = []
[io]
feed_chan_size = 1024
high_frequency_feed_chan_size = 2048
max_cache_count = 1024
cache_dump_threshold = 512
max_dynamic_cache_count = 1024
dynamic_cache_dump_threshold = 512
flush_interval = "10s"
output_file = ""
output_file_inputs = []
enable_cache = false
[dataway]
urls = ["https://aws-openway.guance.com?token=tkn_I_hope_there_is_no_revision_history"]
timeout = "5s"
http_proxy = ""
max_fail = 20
[logging]
log = "/var/log/datakit/log"
gin_log = "/var/log/datakit/gin.log"
level = "info"
disable_color = false
rotate = 32
[global_tags]
cluster = ""
project = ""
site = ""
[environments]
ENV_HOSTNAME = ""
[cgroup]
enable = true
cpu_max = 30.0
cpu_min = 5.0
[tracer]
trace_enabled = false
host = ""
port = ""
service = ""
version = ""
logs_startup = false
debug = false
env = ""
[git_repos]
pull_interval = "1m"
[[git_repos.repo]]
enable = false
url = ""
ssh_private_key_path = ""
ssh_private_key_password = ""
branch = "master"
2021-12-19T04:28:45.161Z INFO config config/cfg.go:520 here is hostname:
2021-12-19T04:28:45.161Z INFO config config/cfg.go:523 set hostname to dd2e01a6b885
2021-12-19T04:28:45.163Z INFO installer installer/main.go:449 installing service datakit...
2021-12-19T04:28:45.167Z INFO installer installer/main.go:274 starting service datakit...
2021-12-19T04:28:45.198Z DEBUG config config/cfg.go:839 remove link /usr/local/bin/datakit...
2021-12-19T04:28:45.199Z DEBUG config config/cfg.go:839 remove link /usr/local/sbin/datakit...
2021-12-19T04:28:45.199Z DEBUG config config/cfg.go:839 remove link /sbin/datakit...
2021-12-19T04:28:45.199Z DEBUG config config/cfg.go:839 remove link /usr/sbin/datakit...
2021-12-19T04:28:45.199Z DEBUG config config/cfg.go:839 remove link /usr/bin/datakit...
2021-12-19T04:28:45.199Z INFO installer installer/main.go:289 :) Install Success!
Visit http://localhost:9529/man/changelog to see DataKit change logs.
Visit http://localhost:9529/monitor to see DataKit running status.
Visit http://localhost:9529/man to see DataKit manuals.
root@dd2e01a6b885:/# echo $?
0
root@dd2e01a6b885:/#
Impact version range:1.30.0 - 1.34.0
The last normal version: 1.29.1
Env:
remote_write:
- url: "http://127.0.0.1:9529/prom_remote_write"
log:2024-08-05T15:19:42.300+0800 INFO config config/load_inputconf.go:162 find 13 confs: []string{"/usr/local/datakit/conf.d/container/container.conf", "/usr/local/datakit/conf.d/datakit.conf", "/usr/local/datakit/conf.d/host/cpu.conf", "/usr/local/datakit/conf.d/host/disk.conf", "/usr/local/datakit/conf.d/host/diskio.conf", "/usr/local/datakit/conf.d/host/dk.conf", "/usr/local/datakit/conf.d/host/host_processes.conf", "/usr/local/datakit/conf.d/host/hostobject.conf", "/usr/local/datakit/conf.d/host/mem.conf", "/usr/local/datakit/conf.d/host/net.conf", "/usr/local/datakit/conf.d/host/swap.conf", "/usr/local/datakit/conf.d/host/system.conf", "/usr/local/datakit/conf.d/prom/prom_remote_write.conf"}
log:2024-08-05T15:19:42.303+0800 INFO config config/inputcfg.go:143 load input "prom_remote_write" from conf file
log:2024-08-05T15:19:42.306+0800 INFO http httpapi/reg.go:64 register PUT@/prom_remote_write to HTTP server
log:2024-08-05T15:19:42.306+0800 INFO http httpapi/reg.go:64 register POST@/prom_remote_write to HTTP server
log:2024-08-05T15:19:42.307+0800 INFO usage_trace usagetrace/trace.go:139 add reserved inputs []string{"rum", "kafkamq", "prom_remote_write", "beats_output"}
log:2024-08-05T15:19:42.307+0800 INFO usage_trace usagetrace/trace.go:152 add reserved input "prom_remote_write"
log:2024-08-05T15:19:51.242+0800 INFO inputs inputs/inputs.go:361 starting input prom_remote_write ...
log:2024-08-05T15:19:51.242+0800 INFO prom_remote_write promremote/input.go:94 prom_remote_write input started...
log:2024-08-05T15:19:51.242+0800 INFO inputs inputs/inputs.go:365 input prom_remote_write exited, this maybe a input that only register a HTTP handle
log:2024-08-05T15:44:47.615+0800 INFO config config/load_inputconf.go:162 find 13 confs: []string{"/usr/local/datakit/conf.d/container/container.conf", "/usr/local/datakit/conf.d/datakit.conf", "/usr/local/datakit/conf.d/host/cpu.conf", "/usr/local/datakit/conf.d/host/disk.conf", "/usr/local/datakit/conf.d/host/diskio.conf", "/usr/local/datakit/conf.d/host/dk.conf", "/usr/local/datakit/conf.d/host/host_processes.conf", "/usr/local/datakit/conf.d/host/hostobject.conf", "/usr/local/datakit/conf.d/host/mem.conf", "/usr/local/datakit/conf.d/host/net.conf", "/usr/local/datakit/conf.d/host/swap.conf", "/usr/local/datakit/conf.d/host/system.conf", "/usr/local/datakit/conf.d/prom/prom_remote_write.conf"}
log:2024-08-05T15:44:47.617+0800 INFO config config/inputcfg.go:143 load input "prom_remote_write" from conf file
log:2024-08-05T15:44:47.620+0800 INFO usage_trace usagetrace/trace.go:139 add reserved inputs []string{"rum", "kafkamq", "prom_remote_write", "beats_output"}
log:2024-08-05T15:44:47.620+0800 INFO http httpapi/reg.go:64 register POST@/prom_remote_write to HTTP server
log:2024-08-05T15:44:47.620+0800 INFO http httpapi/reg.go:64 register PUT@/prom_remote_write to HTTP server
log:2024-08-05T15:44:47.620+0800 INFO usage_trace usagetrace/trace.go:152 add reserved input "prom_remote_write"
log:2024-08-05T15:44:49.616+0800 INFO inputs inputs/inputs.go:361 starting input prom_remote_write ...
log:2024-08-05T15:44:49.616+0800 INFO prom_remote_write promremote/input.go:94 prom_remote_write input started...
log:2024-08-05T15:44:49.616+0800 INFO inputs inputs/inputs.go:365 input prom_remote_write exited, this maybe a input that only register a HTTP handle
log:2024-08-05T15:49:50.212+0800 INFO config config/load_inputconf.go:162 find 13 confs: []string{"/usr/local/datakit/conf.d/container/container.conf", "/usr/local/datakit/conf.d/datakit.conf", "/usr/local/datakit/conf.d/host/cpu.conf", "/usr/local/datakit/conf.d/host/disk.conf", "/usr/local/datakit/conf.d/host/diskio.conf", "/usr/local/datakit/conf.d/host/dk.conf", "/usr/local/datakit/conf.d/host/host_processes.conf", "/usr/local/datakit/conf.d/host/hostobject.conf", "/usr/local/datakit/conf.d/host/mem.conf", "/usr/local/datakit/conf.d/host/net.conf", "/usr/local/datakit/conf.d/host/swap.conf", "/usr/local/datakit/conf.d/host/system.conf", "/usr/local/datakit/conf.d/prom/prom_remote_write.conf"}
log:2024-08-05T15:49:50.215+0800 INFO config config/inputcfg.go:143 load input "prom_remote_write" from conf file
log:2024-08-05T15:49:50.217+0800 INFO usage_trace usagetrace/trace.go:139 add reserved inputs []string{"rum", "kafkamq", "prom_remote_write", "beats_output"}
log:2024-08-05T15:49:50.218+0800 INFO http httpapi/reg.go:64 register PUT@/prom_remote_write to HTTP server
log:2024-08-05T15:49:50.218+0800 INFO http httpapi/reg.go:64 register POST@/prom_remote_write to HTTP server
log:2024-08-05T15:49:50.218+0800 INFO usage_trace usagetrace/trace.go:152 add reserved input "prom_remote_write"
log:2024-08-05T15:49:51.977+0800 INFO inputs inputs/inputs.go:361 starting input prom_remote_write ...
log:2024-08-05T15:49:51.978+0800 INFO prom_remote_write promremote/input.go:94 prom_remote_write input started...
log:2024-08-05T15:49:51.978+0800 INFO inputs inputs/inputs.go:365 input prom_remote_write exited, this maybe a input that only register a HTTP handle
log:2024-08-05T15:50:23.835+0800 INFO config config/load_inputconf.go:162 find 13 confs: []string{"/usr/local/datakit/conf.d/container/container.conf", "/usr/local/datakit/conf.d/datakit.conf", "/usr/local/datakit/conf.d/host/cpu.conf", "/usr/local/datakit/conf.d/host/disk.conf", "/usr/local/datakit/conf.d/host/diskio.conf", "/usr/local/datakit/conf.d/host/dk.conf", "/usr/local/datakit/conf.d/host/host_processes.conf", "/usr/local/datakit/conf.d/host/hostobject.conf", "/usr/local/datakit/conf.d/host/mem.conf", "/usr/local/datakit/conf.d/host/net.conf", "/usr/local/datakit/conf.d/host/swap.conf", "/usr/local/datakit/conf.d/host/system.conf", "/usr/local/datakit/conf.d/prom/prom_remote_write.conf"}
log:2024-08-05T15:50:23.837+0800 INFO config config/inputcfg.go:143 load input "prom_remote_write" from conf file
log:2024-08-05T15:50:23.839+0800 INFO http httpapi/reg.go:64 register PUT@/prom_remote_write to HTTP server
log:2024-08-05T15:50:23.839+0800 INFO http httpapi/reg.go:64 register POST@/prom_remote_write to HTTP server
log:2024-08-05T15:50:23.840+0800 INFO usage_trace usagetrace/trace.go:139 add reserved inputs []string{"rum", "kafkamq", "prom_remote_write", "beats_output"}
log:2024-08-05T15:50:23.840+0800 INFO usage_trace usagetrace/trace.go:152 add reserved input "prom_remote_write"
log:2024-08-05T15:50:26.299+0800 INFO inputs inputs/inputs.go:361 starting input prom_remote_write ...
log:2024-08-05T15:50:26.299+0800 INFO prom_remote_write promremote/input.go:94 prom_remote_write input started...
log:2024-08-05T15:50:26.299+0800 INFO inputs inputs/inputs.go:365 input prom_remote_write exited, this maybe a input that only register a HTTP handle
log:2024-08-05T16:05:32.793+0800 INFO config config/load_inputconf.go:162 find 13 confs: []string{"/usr/local/datakit/conf.d/container/container.conf", "/usr/local/datakit/conf.d/datakit.conf", "/usr/local/datakit/conf.d/host/cpu.conf", "/usr/local/datakit/conf.d/host/disk.conf", "/usr/local/datakit/conf.d/host/diskio.conf", "/usr/local/datakit/conf.d/host/dk.conf", "/usr/local/datakit/conf.d/host/host_processes.conf", "/usr/local/datakit/conf.d/host/hostobject.conf", "/usr/local/datakit/conf.d/host/mem.conf", "/usr/local/datakit/conf.d/host/net.conf", "/usr/local/datakit/conf.d/host/swap.conf", "/usr/local/datakit/conf.d/host/system.conf", "/usr/local/datakit/conf.d/prom/prom_remote_write.conf"}
log:2024-08-05T16:05:32.795+0800 INFO config config/inputcfg.go:143 load input "prom_remote_write" from conf file
log:2024-08-05T16:05:32.798+0800 INFO usage_trace usagetrace/trace.go:139 add reserved inputs []string{"rum", "kafkamq", "prom_remote_write", "beats_output"}
log:2024-08-05T16:05:32.798+0800 INFO http httpapi/reg.go:64 register PUT@/prom_remote_write to HTTP server
log:2024-08-05T16:05:32.798+0800 INFO http httpapi/reg.go:64 register POST@/prom_remote_write to HTTP server
log:2024-08-05T16:05:32.798+0800 INFO usage_trace usagetrace/trace.go:152 add reserved input "prom_remote_write"
log:2024-08-05T16:05:34.446+0800 INFO inputs inputs/inputs.go:361 starting input prom_remote_write ...
log:2024-08-05T16:05:34.447+0800 INFO prom_remote_write promremote/input.go:94 prom_remote_write input started...
log:2024-08-05T16:05:34.447+0800 INFO inputs inputs/inputs.go:365 input prom_remote_write exited, this maybe a input that only register a HTTP handle
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.