Comments (11)
It clearly looks like a nasty memory leak.
Just to try to pinpoint what's going on. You're using RRD provider right, and you just let the service run with a refresh interval (if yes what was the value)?
from facette.
Hi Vincent,
yes I am using the RRD provider. The refresh interval for the UI is set to 10s, this is default I guess.
As far as I know there is no other refresh interval setting, or did I miss something?
The memory leak occurs purely on startup of the server, even if it is running completely without any client interaction.
I assume it initially scans the rrd sources and the memory leak occurs in that initialization phase.
Can I provide you with more information to help pinning down the cause for the memory leak?
Best,
Andreas
from facette.
Sadly, there is no pprof endpoint on Facette to help gathering information on this leak.
I'll try to reproduce and build a custom version with such endpoint.
Regarding the refresh interval, I was talking about the one from the provider definition (default is 0
, i.e. no refresh):
from facette.
hey @akohlbecker,
Quick additional questions:
- Do you have any symbolic links in your RRD folders, and is there any possible symlink loop (this case is indeed not handled)?
- Did you try running Facette in debug mode (see https://github.com/facette/facette/blob/master/docs/examples/facette.yaml#L6) to see if there is something weird in the logs?
Regards,
Vincent
from facette.
The refresh interval for the rrd provider was set to 10
. After setting this to 500
facette seems to behave normally.
These are milliseconds, not seconds as I was assuming originally, right?
I will report long term results from this settings change tomorrow.
from facette.
These are milliseconds, not seconds as I was assuming originally, right?
This setting unit is seconds: https://docs.facette.io/latest/api/providers/#create-a-provider
It would be surprising that raising this interval fixes the issue. It might take longer to trigger though.
from facette.
Hi Vincent,
your expectation was correct, the memory consumption has increased over night and is now at about 10 GB.
I checked the rrd folders for symlinks and found none.
The debug log contains many entries like these (ellypsed here):
2021/09/28 09:59:51.058919 DEBUG: poller[collectd]: inserted record {Origin: "collectd", Source: .... in "collectd" catalog
2021/09/28 09:59:51.058926 DEBUG: poller[collectd]: does not match "/average$" sieve pattern, discarding: ....
apart from these 588313 lines after running facette for 10 minutes with an rrd provider refresh interval of 500, the log only has these entries:
2021/09/28 09:59:49.834419 INFO: http: started
2021/09/28 09:59:49.834420 INFO: poller: started
2021/09/28 09:59:49.834630 INFO: http: listening on "127.0.0.1:12003"
2021/09/28 09:59:49.835399 DEBUG: poller[collectd]: started
2021/09/28 09:59:50.867371 DEBUG: poller[collectd]: restored previous catalog state in 1.031827415s
2021/09/28 09:59:50.867445 DEBUG: poller[collectd]: refreshing "collectd" provider
2021/09/28 10:08:10.867708 DEBUG: poller[collectd]: refreshing "collectd" provider
from facette.
Which version are you running, the latest release or a build from master
?
What's your platform/architecture, linux/amd64?
I built a custom version yesterday having a pprof HTTP endpoint that might would allow us to visualize heap usage while running the service. I'll try to push it in a dedicated branch tonight but I can build the binary for you to test if you want.
from facette.
I am running version 0.5.1 on linux/amd64 (4.9.0-15-amd64 #1 SMP Debian 4.9.258-1 (2021-03-08) x86_64 GNU/Linux)
It would be great if you could build the binary for me.
TNX
Andreas
from facette.
Hi @akohlbecker,
Sorry for the delay here.
I just pushed changes to a dedicated branch that registers debugging pprof endpoints to the web server, see 593ce3f.
Here comes a .deb file embedding those changes (note: had to gzip it to make GitHub accept it 🤷 ):
facette_0.6.0-0~git20211005.593ce3f7_amd64.deb.gz
Once installed and the issue triggered, you should be able to visualize heap information from the running service using:
go tool pprof -http=:8080 http://your-facette-instance:12003/debug/pprof/heap
If you could extract it for me, it would be great too:
curl -s http://your-facette-instance:12003/debug/pprof/heap >facette-heap.out
from facette.
Hi Vincent,
thank you for the binary.
BTW: Since I've set the refresh interval for the rrd provider to 500 I no longer have problems.
In installed the debug build anyway and here is the pprof output:
facette-heap.out.gz
Cheers
Andreas
from facette.
Related Issues (20)
- left menu & mobile version HOT 2
- Data-browse feature HOT 2
- Fix template detection in administration panes
- create new graph/collection -> create new graph/collection (template) HOT 2
- Graph's `Minimum value` auto is zero
- graph's vertical space
- When Maximum/Minimum value is specified the red line is shown
- Build failds: cannot use *"github.com/...".Graph literal as type *"github.com/..." .Graph in field value HOT 2
- base_path/api/v1 doesn't work HOT 1
- Timeout on /api/v1/series/points HOT 2
- Natural sorting of sources and metrics HOT 2
- Allow minutes and hours for durations
- Rename Series Names HOT 2
- Duration reports NaN with base set to "s"
- Test & verify connection with provider HOT 3
- Suggestion: Ctrl + Click to drag graphs left and right HOT 2
- Add ability to build graphs from VictoriaMetrics data HOT 1
- Looking for an example for InfluxDB 1.8 connection HOT 2
- My property with move on
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from facette.