Comments (18)
大概原因是,在重载的时候,先已经stopping关闭了通道,但是指标正好采集回来,正往里面写数据,导致的
from categraf.
复现可以把quitChan分支时间调长
for {
select {
case <-r.quitChan:
time.Sleep(interval * 10)
close(r.quitChan)
return
default:
time.Sleep(interval)
r.gatherOnce()
}
}
修复方案,close(r.queue) 从Stop()中移到close(r.quitChan)之前
from categraf.
panic: close of closed channel
goroutine 1 [running]:
flashcat.cloud/categraf/agent.(*InputReader).Stop(...)
/root/categraf/agent/metrics_reader.go:46
flashcat.cloud/categraf/agent.(*Agent).stopMetricsAgent(...)
/root/categraf/agent/metrics_agent.go:90
flashcat.cloud/categraf/agent.(*Agent).Stop(0xc0003eae38)
/root/categraf/agent/agent.go:67 +0xf3
flashcat.cloud/categraf/agent.(*Agent).Reload(0xc000383d58?)
/root/categraf/agent/agent.go:74 +0x57
main.handleSignal(0xc0008f1e60?)
/root/categraf/main.go:94 +0x206
main.runAgent(0x0?)
/root/categraf/main_posix.go:12 +0x28
main.main()
/root/categraf/main.go:69 +0x205
from categraf.
连续两次快速的reload可能引发 panic: close of closed channel
/root/categraf/agent/metrics_reader.go:46
close(r.queue)
from categraf.
@UlricQin 如你所说,之前我被两次测试误解了,之前的判断是错误的。 我之前关闭的issue,在0.1.6测试的,也是quitchan写入时panic
from categraf.
from categraf.
panic是写入r.queue,0.1.13已经修复了。 可以再测试下https://github.com/flashcatcloud/categraf/blob/main/agent/metrics_reader.go#L63
from categraf.
如果没有其他问题,这个issue先close了
from categraf.
并未解决
2022/07/14 22:57:00 main.go:90: I! received signal: hangup
2022/07/14 22:57:00 agent.go:80: I! agent reloading
2022/07/14 22:57:00 agent.go:71: I! agent stopping
panic: send on closed channel
goroutine 1 [running]:
flashcat.cloud/categraf/agent.(*InputReader).Stop(...)
/home/runner/work/categraf/categraf/agent/metrics_reader.go:44
flashcat.cloud/categraf/agent.(*Agent).stopMetricsAgent(...)
/home/runner/work/categraf/categraf/agent/metrics_agent.go:94
flashcat.cloud/categraf/agent.(*Agent).Stop(0xc000bdce38)
/home/runner/work/categraf/categraf/agent/agent.go:73 +0xe5
flashcat.cloud/categraf/agent.(*Agent).Reload(0xc000bdcd58?)
/home/runner/work/categraf/categraf/agent/agent.go:81 +0x57
main.handleSignal(0xc000c2d1c0?)
/home/runner/work/categraf/categraf/main.go:95 +0x206
main.runAgent(0x0?)
/home/runner/work/categraf/categraf/main_posix.go:12 +0x28
main.main()
/home/runner/work/categraf/categraf/main.go:70 +0x209
from categraf.
如之前所说,panic是
/home/runner/work/categraf/categraf/agent/metrics_reader.go:44
写入 quitChan ,所以诡异
from categraf.
./categraf -version
v0.2.2-a7b408226694a3a3bad08f1cfe06fc96cf92cd18
from categraf.
复现方法还是两次reload? 加载了哪些插件?
from categraf.
没有两次reload,就是一次普通的reload,ping net_response http_response
from categraf.
一次reload 就只写一次quitchan,两次的话,倒是有可能
from categraf.
这个问题,还有出现吗? @tanxiao1990
from categraf.
老版本一直都有,新版本没更新,如果新版本没有对这段逻辑做处理的话,肯定是存在的。
from categraf.
新版本没有对这块再增加处理逻辑。 我这边复现不出来,好像也没有其他用户再反馈类似的问题。看否有其他线索。
from categraf.
感谢 @tanxiao1990 复现和PR #477
#478
from categraf.
Related Issues (20)
- filters不起作用,多个instances共用1个instance table HOT 3
- snmp 获取到的内容mapping ,怎么设置默认值,比如我把所有查询的数据全部mapping为1 HOT 1
- 支持resin指标采集吗? HOT 6
- oracle.go:184: E! failed to query: ORA-01000: HOT 8
- 作为系统服务报错 HOT 5
- snmp_ 默认指标前缀是否可以关闭? HOT 5
- 用systemctl管理启动不起来 HOT 1
- 0.3.57版本新增的input.supervisor导致categraf启动异常。 HOT 2
- . HOT 2
- 新增nfs-server监控 HOT 1
- 新增Raid和磁盘监控 HOT 1
- oracle插件的address填写错误时,直接panic导致categraf异常退出 HOT 4
- categraf有提供采集的指标汇聚上报的功能吗? HOT 1
- categraf v0.3.58配置ES监控配置,启动时异常 HOT 4
- Categraf 0.3.58 使用 gosmi + path 模式初始化字段出错 netsnmp 正常 HOT 11
- categraf v0.3.58 conf/logs.toml采集文件的能力,支持限流吗? HOT 2
- categraf 使用prometheus采集ceph报错 HOT 3
- categraf 0.3.59想确认下远程配置拉取,是否必须version和key发生变化才能进行更新? HOT 4
- categraf 0.3.59版本监控ES,无磁盘水位线指标 HOT 5
- 采集日志推送kafka报错 HOT 14
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from categraf.