blueconic / node-oom-heapdump Goto Github PK

Do we have any option which will enable to mask the sensitive information from the heapdumps ?

I'm running my node server in a Docker container. I was already using --inspect flag so I could bind VSCode debugging system to my server inside the container so I tried using the port option to match the custom port number I was using. However I'm getting this error in the console:
Debugger is not listening on port 9229, 'node-oom-heapdump' cannot function correctly. Is the Node.js process started with the --inspect=9229 flag? { Error: connect ECONNREFUSED 127.0.0.1:9229

Here is the command I run to execute my server:
npx nodemon -e html,js,ts --watch src --inspect=0.0.0.0:5858 --nolazy dist/index.js

Hence in the main file of my server, I did:

import path from 'path';
import oom from 'node-oom-heapdump';

oom({ path: path.resolve(__dirname, 'my_heapdump'), port: 5858 });

If I change my command to:
npx nodemon -e html,js,ts --watch src --inspect=0.0.0.0:9229 --nolazy dist/index.js

it now works.

Hello,

I am running node v12.x, which includes improved async stack traces.
I'm trying to debug a potential memory leak in one of my applications, where taking snapshots is consistently crashing the process, and found this library which looks really promising.

Unfortunately the build is failing on node 12.7.0; I get a little further after naively updating all dependencies to their latest versions:

npm install

> [email protected] install ******/node-oom-heapdump/node_modules/gc-stats
> node-pre-gyp install --fallback-to-build

node-pre-gyp WARN Using needle for node-pre-gyp https download
node-pre-gyp WARN Tried to download(403): https://node-binaries.s3.amazonaws.com/gcstats/v1.4.0/Release/node-v72-darwin-x64.tar.gz
node-pre-gyp WARN Pre-built binaries not found for [email protected] and [email protected] (node-v72 ABI, unknown) (falling back to source compile with node-gyp)
  CXX(target) Release/obj.target/gcstats/src/gcstats.o
  SOLINK_MODULE(target) Release/gcstats.node
  COPY ******/node-oom-heapdump/node_modules/gc-stats/build/gcstats/v1.4.0/Release/node-v72-darwin-x64/gcstats.node
  TOUCH Release/obj.target/action_after_build.stamp

> [email protected] install ******/node-oom-heapdump
> node-gyp rebuild

  CXX(target) Release/obj.target/node_oom_heapdump_native/lib/node_oom_heapdump_native.o
../lib/node_oom_heapdump_native.cc:77:44: error: no matching member function for call to
      'ToString'
  String::Utf8Value fArg(isolate, args[0]->ToString());
                                  ~~~~~~~~~^~~~~~~~
/Users/david/.node-gyp/12.7.0/include/node/v8.h:2535:44: note: candidate function not
      viable: requires single argument 'context', but no arguments were provided
  V8_WARN_UNUSED_RESULT MaybeLocal<String> ToString(
                                           ^
/Users/david/.node-gyp/12.7.0/include/node/v8.h:2551:31: note: candidate function not
      viable: requires single argument 'isolate', but no arguments were provided
                Local<String> ToString(Isolate* isolate) const);
                              ^
../lib/node_oom_heapdump_native.cc:83:27: error: no matching member function for call to
      'BooleanValue'
  addTimestamp = args[1]->BooleanValue();
                 ~~~~~~~~~^~~~~~~~~~~~
/Users/david/.node-gyp/12.7.0/include/node/v8.h:2566:8: note: candidate function not
      viable: requires single argument 'isolate', but no arguments were provided
  bool BooleanValue(Isolate* isolate) const;
       ^
/Users/david/.node-gyp/12.7.0/include/node/v8.h:2569:51: note: candidate function not
      viable: requires single argument 'context', but no arguments were provided
                V8_WARN_UNUSED_RESULT Maybe<bool> BooleanValue(
                                                  ^
2 errors generated.
make: *** [Release/obj.target/node_oom_heapdump_native/lib/node_oom_heapdump_native.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:266:23)
gyp ERR! stack     at ChildProcess.emit (events.js:203:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12)
gyp ERR! System Darwin 18.7.0
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd ******/node-oom-heapdump
gyp ERR! node -v v12.7.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

Do you expect to add support for node 12 anytime soon?
Thanks!

Hello,

Thanks for your efforts on this library! We find it fills an important part of our post-mortem toolkit, and we're particularly grateful for the work @paulrutter has done in working with the Node and V8 communities to get this implemented.

We have the pleasure of deploying on both Windows and Linux, and as this library needs a native component, we currently need to run an npm install on both OSes to get a suitable distribution.

Would you be interested in releasing pre-built binaries via node-pre-gyp? I would be happy to have a go at contributing a patch to support this via Travis / Appveyor.

See https://github.com/blueconic/node-oom-heapdump/actions/runs/5643075202

@spmiller any idea whats causing this to fail? Might be that the github action scripts need adjusting, since we didn’t release for a while.

Let me know

The native module doesn't build on Node20, due to the method signature being changed.
See https://v8.github.io/api/head/namespacev8.html.

(const char* location, const OOMDetails)

I'm trying to dump heap or cpu profile and get:

/workspace/cf-abacus/node_modules/node-oom-heapdump/lib/index.js:66
      cmd: path.dirname(require.main.filename),
                                     ^

TypeError: Cannot read property 'filename' of undefined
    at NodeOomHeapDumpImpl._callWorker (/workspace/cf-abacus/node_modules/node-oom-heapdump/lib/index.js:66:38)
    at NodeOomHeapDumpImpl.createHeapSnapshot (/workspace/cf-abacus/node_modules/node-oom-heapdump/lib/index.js:113:17)
    at NodeOomHeapdumpAPI.createHeapSnapshot (/workspace/cf-abacus/node_modules/node-oom-heapdump/index.js:31:23)
    at process.on (/workspace/cf-abacus/node_modules/abacus-webapp/src/index.js:134:21)
    at emitNone (events.js:111:20)
    at process.emit (events.js:208:7)
    at Signal.wrap.onsignal (internal/process.js:208:44)

Running on:
macOS 10.13.2
node 8.9.4
yarn 1.3.2

Recently we upgraded to 1.3.0 from 1.2.1 and we now see this error when our application starts up:

        ^Error: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by node-oom-heapdump/build/Release/node_oom_heapdump_native.node)
    at Object.Module._extensions..node (internal/modules/cjs/loader.js:807:18)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)

Downgraded to 1.2.1 resolves the issue for us -- is this a breaking change, or is this something we need to change about our environment?

I updated some dependencies and added a few Node versions (and removed some older ones).
@spmiller can you check why this action doesn't succeed anymore?

https://github.com/blueconic/node-oom-heapdump/actions/runs/1822665059

Thanks in advance.

Please upgrade to a newer tar version which is not vulnerable to CVE-2021-32803

I am curious if there is a way to determine how far a heap dump creation (which is triggered by NATIVE_HOOK integration) has progressed?

We have an OOM issue in our application and installed this library. The on-demand heap dumps work flawlessly (note: little memory is used for them). However, when an OOM error is triggered, the console log "Generating Heapdump to '/var/log/nodejs/heapdumps/xxxxxxxxxx.heapsnapshot' now..." is printed and we never get the dump being written to disk. The file is created, but its size remains 0.

I think this is because 1 min after the OOM, the container is killed by health checks and container orchestration tooling. This 1 min seems not to be enough time to create and write the file to disk. That is why I would like to check if the application is actively working on creating the heap dump. If that is the case, I would look for a solution to keep the container alive after health checks failed until the heap dump creation finishes.

I looked at the chrome-remote-interface and noticed that there are several callbacks used by heapdumpWorker.js. Does anybody know if it is possible to obtain and log more information during heap dump creation?

We are seeing segmentation fault errors with nodejs v14.18.3. Its intermittent

Test : npm run test from the module
OS : Linux 5.14.0-1054-oem #61-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux

NOTE : No issues seen with nodejs v16.x.x

handrasekharposhamolla@KI0535:~/workspace/research/node_modules/node-oom-heapdump$ npm run test

[email protected] test workspace/research/node_modules/node-oom-heapdump
node --max_old_space_size=80 --inspect ./tests/oom_app.js

Debugger listening on ws://127.0.0.1:9229/5dc48fd2-806d-4383-9f71-73bdaa030127
For help, see: https://nodejs.org/en/docs/inspector
Debugger is listening on port 9229, 'node-oom-heapdump' can function correctly.
Leaks: 100000
Leaks: 200000
Leaks: 300000
Leaks: 400000
Leaks: 500000
Leaks: 600000
Leaks: 700000
Leaks: 800000
Leaks: 900000
Leaks: 1000000
Leaks: 1100000
Leaks: 1200000
Leaks: 1300000
Leaks: 1400000
Leaks: 1500000
Leaks: 1600000
Leaks: 1700000
Leaks: 1800000
Leaks: 1900000
Leaks: 2000000
Leaks: 2100000
Leaks: 2200000
Leaks: 2300000
Leaks: 2400000

<--- Last few GCs --->

[531231:0x54f6830] 2892 ms: Mark-sweep (reduce) 80.1 (83.2) -> 80.0 (85.0) MB, 121.2 / 0.0 ms (+ 0.0 ms in 13 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 122 ms) (average mu = 0.250, current mu = 0.032) allocati[531231:0x54f6830] 3052 ms: Mark-sweep (reduce) 81.0 (85.0) -> 81.0 (86.0) MB, 158.5 / 0.0 ms (+ 0.0 ms in 10 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 159 ms) (average mu = 0.127, current mu = 0.008) allocati

<--- JS stacktrace --->

Generating Heapdump to 'workspace/research/node_modules/node-oom-heapdump/tests/my_heapdump.heapsnapshot' now...
Segmentation fault

::: Segmentation trace :::

.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal4Heap25GarbageCollectionPrologueEv+0x184)[0xd7ab34]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal4Heap14CollectGarbageENS0_15AllocationSpaceENS0_23GarbageCollectionReasonENS_15GCCallbackFlagsE+0x17e)[0xd828ee]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal4Heap24PreciseCollectAllGarbageEiNS0_23GarbageCollectionReasonENS_15GCCallbackFlagsE+0x40)[0xd84460]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal21HeapSnapshotGenerator16GenerateSnapshotEv+0x35)[0x1021e15]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal12HeapProfiler12TakeSnapshotEPNS_15ActivityControlEPNS_12HeapProfiler18ObjectNameResolverEb+0x74)[0x1012174]
workspace/research/node_modules/node-oom-heapdump/build/Release/node_oom_heapdump_native.node(+0x27e1)[0x7f19acf1a7e1]
workspace/research/node_modules/node-oom-heapdump/build/Release/node_oom_heapdump_native.node(_Z10OnOOMErrorPKcb+0x19)[0x7f19acf1a8e9]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v85Utils16ReportOOMFailureEPNS_8internal7IsolateEPKcb+0x20)[0xbb7e90]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal2V823FatalProcessOutOfMemoryEPNS0_7IsolateEPKcb+0x347)[0xbb8237]
.nvm/versions/node/v14.18.3/bin/node[0xd74445]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal23EvacuateNewSpaceVisitor5VisitENS0_10HeapObjectEi+0x24e)[0xda4dde]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal13FullEvacuator15RawEvacuatePageEPNS0_11MemoryChunkEPl+0xe56)[0xdb0e16]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal9Evacuator12EvacuatePageEPNS0_11MemoryChunkE+0x7f)[0xd9cfaf]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal18PageEvacuationTask13RunInParallelENS0_15ItemParallelJob4Task6RunnerE+0xa8)[0xd9d228]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal15ItemParallelJob3RunEv+0x1f9)[0xd8fb09]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal24MarkCompactCollectorBase31CreateAndExecuteEvacuationTasksINS0_13FullEvacuatorENS0_20MarkCompactCollectorEEEvPT0_PNS0_15ItemParallelJobEPNS0_17MigrationObserverEl+0x590)[0xdb2d70]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal20MarkCompactCollector23EvacuatePagesInParallelEv+0x2ac)[0xdb360c]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal20MarkCompactCollector8EvacuateEv+0x125)[0xdb37d5]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal20MarkCompactCollector14CollectGarbageEv+0xf1)[0xdc57d1]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal4Heap11MarkCompactEv+0xd8)[0xd81a98]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal4Heap14CollectGarbageENS0_15AllocationSpaceENS0_23GarbageCollectionReasonENS_15GCCallbackFlagsE+0xe18)[0xd83588]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal4Heap34AllocateRawWithRetryOrFailSlowPathEiNS0_14AllocationTypeENS0_16AllocationOriginENS0_19AllocationAlignmentE+0x5c)[0xd869cc]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal7Factory11AllocateRawEiNS0_14AllocationTypeENS0_19AllocationAlignmentE+0x9d)[0xd4c19d]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal11FactoryBaseINS0_7FactoryEE26AllocateRawWithImmortalMapEiNS0_14AllocationTypeENS0_3MapENS0_19AllocationAlignmentE+0x14)[0xd46024]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal11FactoryBaseINS0_7FactoryEE19NewRawOneByteStringEiNS0_14AllocationTypeE+0x50)[0xd48030]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal7Factory11SmiToStringENS0_3SmiENS0_15NumberCacheModeE+0x88)[0xd5c108]
.nvm/versions/node/v14.18.3/bin/node(_ZN2v88internal26Runtime_NumberToStringSlowEiPmPNS0_7IsolateE+0x45)[0x10a8f95]
.nvm/versions/node/v14.18.3/bin/node[0x14467f9]
Segmentation fault

Hi, thanks for the package, it's been very useful to me!

I've read the readme and I know that the increased memory usage is expected during on request heapdumps, but is it expected that the memory usage never goes back to it's initial value after the dump is done, even after manually calling garbage collection ?

I have a web server running on Node v14.18.2 and pm2 v5.1.2 . Right after starting the application the memory usage is around 300mb, and after creating an on demand heapdump it's about 600mb. I tried waiting, and after 12 hours it's still at 600mb. Manually calling garbage collection doesn't do anything. I tried creating another heapdump, and this time the memory didn't double, but still increased to around 900mb. Is this expected ? This becomes an issue when creating multiple dumps in order to compare them

Hi

We are using node v6.11 at our production. We need to take heapdump when node out of memory happened. What are the changes needed if we want to use this module in our production env ?

looks to be an issue w/ line 24 in lib/index.js

let gcStats = new require('gc-stats')();

I think let gcStats = new (require('gc-stats'))(); is a possible fix.
webpack/webpack#4940

I'd like to use this module while debugging a memory leak on a non-public, but deployed environment.

Following the node.js spec, you can specify an IP, but this module doesn't provide configurability for that layer.

Is it possible to make this configurable?

Using version 1.0.11 on node 6.12.0; node-oom-heapdump started at server boot with only path option.
Got confirmation that it started:
Debugger is listening on port 9229, 'node-oom-heapdump' can function correctly.

But when a OOM occurs, node-oom-heapdump isn't triggered and no dump file is written.

Console output from OOM attached.

<--- Last few GCs --->

326386181 ms: Mark-sweep 1322.2 (1436.6) -> 1322.2 (1436.6) MB, 1091.8 / 0.5 ms [allocation failure] [GC in old space requested].
326387262 ms: Mark-sweep 1322.2 (1436.6) -> 1322.2 (1436.6) MB, 1080.8 / 0.5 ms [allocation failure] [GC in old space requested].
326388400 ms: Mark-sweep 1322.2 (1436.6) -> 1331.4 (1420.6) MB, 1138.3 / 1.0 ms [last resort gc].
326389575 ms: Mark-sweep 1331.4 (1420.6) -> 1340.6 (1420.6) MB, 1174.8 / 1.0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x363367acf781 <JS Object>
    1: set [/srv/apinode/node_modules/bookshelf/lib/base/model.js:~189] [pc=0x394ff81b7535] (this=0x26f3563a4069 <JS Object>,key=0x372aaa99a6d1 <an Object with map 0x26b5a7d84f79>,val=0x19e2c84a8819 <an Object with map 0x26de185605b9>,op
tions=0x363367a04381 <undefined>)
    2: arguments adaptor frame: 2->3
    3: constructor(aka ModelBase) [/srv/apinode/node_modules/bookshelf/lib/base/model.js:...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/usr/local/bin/node]
 2: 0x10a04bc [/usr/local/bin/node]
 3: v8::Utils::ReportApiFailure(char const*, char const*) [/usr/local/bin/node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
 5: v8::internal::Factory::NewTransitionArray(int) [/usr/local/bin/node]
 6: v8::internal::TransitionArray::Insert(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Map>, v8::internal::SimpleTransitionFlag) [/usr/local/bin/node]
 7: v8::internal::Map::CopyReplaceDescriptors(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::DescriptorArray>, v8::internal::Handle<v8::internal::LayoutDescriptor>, v8::internal::TransitionFlag, v8::internal:
:MaybeHandle<v8::internal::Name>, char const*, v8::internal::SimpleTransitionFlag) [/usr/local/bin/node]
 8: v8::internal::Map::CopyAddDescriptor(v8::internal::Handle<v8::internal::Map>, v8::internal::Descriptor*, v8::internal::TransitionFlag) [/usr/local/bin/node]
 9: v8::internal::Map::CopyWithField(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::FieldType>, v8::internal::PropertyAttributes, v8::internal::Representation, v8::in
ternal::TransitionFlag) [/usr/local/bin/node]
10: v8::internal::Map::TransitionToDataProperty(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::StoreF
romKeyed) [/usr/local/bin/node]
11: v8::internal::LookupIterator::PrepareTransitionToDataProperty(v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::StoreFromKeyed) [/usr/loc
al/bin/node]
12: v8::internal::StoreIC::LookupForWrite(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::Object::StoreFromKeyed) [/usr/local/bin/node]
13: v8::internal::StoreIC::UpdateCaches(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::Object::StoreFromKeyed) [/usr/local/bin/node]
14: v8::internal::StoreIC::Store(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::Object::StoreFromKeyed) [/usr/local/bin/node]
15: v8::internal::KeyedStoreIC::Store(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
16: v8::internal::Runtime_KeyedStoreIC_Miss(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
17: 0x394febd092a7