Comments (12)
I know I already closed this issue. Just incase anyone finds themselves here.
My suspicion based on some of the comments above. Is that users where creating Meters regularly and not managing their lifecycles properly.
When a Meter is created you must call Meter.end() when you are done with it. The end method clears the interval which would allow the garbage collector to fully destroy the object.
When you use the Collection or Self Reporting Metrics Registry, shutdown methods to properly end their and the registered metrics lifecycle are exposed. SelfReportingMetricsRegistry#shutdown, Collection#end
from node-measured.
Update: the thing is, that if you use meters (which use a timer to periodically aggregate the data), and then you swap them periodically to avoid a meter going out of range - you get dangling timers.
In addition if you use many meters, let's say 100 in an app, then you have 100 timers running - does that affect performance?
from node-measured.
@maxcountryman I never observed this when I still used the library. The code was all written to avoid unbound memory growth.
from node-measured.
Perhaps this has something to do with the event emitters attached to these objects?
I implemented my own structures and observed no memory issues whatsoever.
from node-measured.
I believe the source of these leaks is setInterval
and perhaps the usage of unref
. It's a fairly catastrophic leak in environments where a lot of data is being thrown at meters (it's taken down my ec2 instances several times now). I'm not really sure what the best "fix" for this is.
from node-measured.
This is important enough to put a notice in the README, if it is verified to be true. As an example -we've taken this as a dependency to avoid writing our own data structures and then as a result to be having to write code for handling unbounded memory edge cases, but looks like it also has these. There might be other people with the same motives and it can save them time to know these trade offs.
from node-measured.
I use this library in production code for several years with node 0.12, 4 and 6. We constantly monitor our process memory and I can NOT confirm that there is a leak. To anyone reading this thread, please trace the source of your leaks using a profiler and point us in the right direction. Should there be a memory leak, I'd instantly merge a PR with a fix.
from node-measured.
IIRC the problem turned out to be the binary heap that underpins much of this library. This structure eventually causes an OOM error, especially when fed a lot of data quickly.
Coda Hale's metrics library is a much better example of how to do similar things without binary heaps.
I personally stopped using this library as soon as I discovered the problem and so I don't have a vested interest in fixing it. So I'll leave this note here and let the authors decide how they'd like to proceed.
from node-measured.
Any update on this issue, does anyone run this library with Node 8 or 10?
from node-measured.
@eugef i'm in the process of writing some Gatling performance tests to validate or invalidate this.
from node-measured.
Can you help me understand this issue better so I can do some TDD / reproduce the issue and verify that the issue has been resolved.
I believe the source of these leaks is setInterval and perhaps the usage of unref.
Do you have an example of what you where doing, when you saw the leak.
Update: the thing is, that if you use meters (which use a timer to periodically aggregate the data), and then you swap them periodically to avoid a meter going out of range - you get dangling timers.
What do you mean by the above?
I just want to either write a test that fails so I can do TDD, or if that isn't possible just knowing how to reproduce would be nice so I can manually verify a fix.
When I wrote a naive performance test, I saw the memory do a sawtooth pattern indicating that garbage collection was happening and memory was not linearly growing.
from node-measured.
Closing this issue as I can't reproduce it, feel free to re-open with steps to reproduce.
from node-measured.
Related Issues (20)
- Is it possible to enable and disable the metrics through configurations? HOT 1
- toJSON() is named incorrectly - doesn't return JSON HOT 1
- Histograms - possible to show windowed averages HOT 5
- Cannot use EWMA with Histogram HOT 2
- Histogram - quantiles weighting HOT 1
- How to use this with Graphite? HOT 2
- Create Type File for Typescript compatibility HOT 5
- Decide whether or not this should be hard forked or add new maintainers. HOT 1
- Support for Value and Rate in Histogram HOT 5
- The return value of funtion:SettableGauge.getType is difference from the requested Type in funtion:Collection.settableGauge. HOT 1
- Object.values is not a function on Nodejs 6.9.0 HOT 1
- "_unknown" uri passed for POSTs to signalfx HOT 7
- bunyan dependency on dtrace-provider is problematic HOT 3
- docs: document release process HOT 5
- Why isn't DimensionAwareMetricsRegistry a subclass of Collection? HOT 5
- Re-hydration from previous state HOT 2
- Get max value of a counter - Feature request (or potentially stupid user) HOT 2
- [RFC] 3.0? HOT 12
- "measure-node-metrics" package is missing Koa middleware support
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from node-measured.