Comments (11)
Thank you for reporting!
I've fixed performance issue of file access:
1215256
However, we might encounter performance issue.
When I ran (new) hsbeat with profiler, it spent CPU time in libbeat publisher and JSON processing.
It might be improved with index structure in ES.
I will continue to work for it in performance-improvement branch .
from hsbeat.
I improved performance of HSBeat than before:
4031b8b
I've attached CPU profile callgraph of it.
- Machine: Fedora23 x86_64 Hyper-V virtual machine (2 vcpu / 2GB mem)
- Host CPU: Intel Core i3 2367M 1.4GHz
- HSBeat collection interval: 10 sec
This profile shows that top overhead is GC routine.
To eliminate them, I implemented entry cache of hsperfdata entries.
It is too difficult to improve more performance.
But I will try it :-)
from hsbeat.
Thanks
I give it a try and found io wait is low ,but user and system cpu is very high and use so much more memory finally cause the jvm killed by oom!
from hsbeat.
I guess that high CPU usage is caused by Go runtime.
diff --git a/hsbeat/hsbeat.go b/hsbeat/hsbeat.go
index 4dede96..c2a5632 100644
--- a/hsbeat/hsbeat.go
+++ b/hsbeat/hsbeat.go
@@ -135,18 +135,22 @@ func (this *HSBeat) publishCached(b *beat.Beat) error {
}
func (this *HSBeat) Run(b *beat.Beat) error {
+/*
err := this.publishAll(b)
if err != nil {
return err
}
+*/
for !this.ShouldTerminate {
time.Sleep(this.Interval)
+/*
err := this.publishCached(b)
if err != nil {
return err
}
+*/
}
diff --git a/main.go b/main.go
index f2d0c83..3912898 100644
--- a/main.go
+++ b/main.go
@@ -28,7 +28,7 @@ import (
hsbeat "github.com/YaSuenag/hsbeat/hsbeat"
- //"runtime/pprof"
+ "runtime/pprof"
)
@@ -38,14 +38,18 @@ func main() {
log.Fatal(err)
}
-/*
- prof, err := os.Create("hsbeat.prof")
+ prof, err := os.Create("hsbeat.pprof")
if err != nil {
log.Fatal(err)
}
pprof.StartCPUProfile(prof)
defer pprof.StopCPUProfile()
-*/
+
+ mprof, err := os.Create("hsbeat.mprof")
+ if err != nil {
+ log.Fatal(err)
+ }
+ defer pprof.WriteHeapProfile(mprof)
hb :=&hsbeat.HSBeat{os.Args[1], time.Duration(interval), "",
false, nil, nil, nil}
I applied above patch (this patch makes empty loop in Beat), but I saw high CPU usage.
CPU profiling shows futex call is the most expensive in profiling session.
profile.zip
Conclusion:
We have to work for more performance improvement. However, we might not be able to avoid overhead in Go runtime.
from hsbeat.
@YaSuenag Which libbeat version are you currently using for hsbeat?
from hsbeat.
@ruflin I use v1.0.0 (be66518)
from hsbeat.
@YaSuenag I didn't check the issue in detail but in general it could help to upgrade the most recent version of libbeat.
from hsbeat.
I fixed this issue in e02c891 !
I close this issue but kibana,json in current HEAD (above commit) does not support this changes.
So I will fix it soon in another commit.
from hsbeat.
CPU usage in e02c891 is < 1% in my machine. Try it!
from hsbeat.
Hi
Thanks for your fix.
I give it a try ,every things is fine.
But there no any data in elasticsearch except the common header!!
Is that correct ?? You have comment too many code ?
from hsbeat.
It is correct.
I've disabled _source in dynamic template to reduce cost of Elasticsearch:
c0a9dff
You can see data on Kibana dashboard if you import etc/kibana.json .
from hsbeat.
Related Issues (6)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hsbeat.