Reporting on memory usage about kit HOT 4 CLOSED

@josephjclark , during the fair pricing conversations today we noted that we'd want a memory limit for a run of a certain type (small, medium, large) but that we'd probably only want to stop a run when it exceeded the highest limit... i think this type of thing might be useful: https://stackoverflow.com/questions/44829364/monitor-maximum-memory-consumption-in-node-js-process

in other words, even if the limit is 1GB, we'll want to know the max used during runtime so we can assess a run afterwards in terms of work done. we'll need to measure, at least runtime and the max memory usage at any point in the run.... but that's just to fairly distribute resources for customers... from the "what stats do i want on my runtime" perspective we may want lots more available.

from kit.

We want urgently to know how much memory each Attempt required to run. This should be logged now and later reported to lightning

from kit.

This issue talks about memory useage per operation but I really don't think that's a helpful level of reporting for now. That's one for a summer sunny day.

This issue is now about reporting the peak memory usage during a run/job, captured in the Engine or the Runtime itself.

Using the API above I think I can report this to logs pretty quickly and easily, and that's what I intend to do in this issue. Just get it out there because we urgently need numbers.

@taylordowns2000 if I can return the maximum memory used for a job, where should I send it? To Lightning in run:complete maybe? Or is this something I need to speak to Rory about?

from kit.

A design:

TL;DR

Report Job memory and System memory at the end of a job, in logs and events.

Or at the end of an operation, if a flag is passed.

Memory usage

Node will report a bunch of different types of memory used. From the docs:

* heapTotal and heapUsed refer to V8's memory usage.
* external refers to the memory usage of C++ objects bound to JavaScript objects managed by V8.
* rss, Resident Set Size, is the amount of space occupied in the main memory device (that is a subset of the total allocated memory) for the process, including all C++ and JavaScript objects and code.
* arrayBuffers refers to memory allocated for ArrayBuffers and SharedArrayBuffers, including all Node.js Buffers

When using Worker threads, rss will be a value that is valid for the entire process, while the other fields will only refer to the current thread.

I think we can reasonably explain this as JOB memory and SYSTEM memory.

Job memory is the internal memory used by your job code, by the stuff imported into your application code. This is heapUsed.

System memory is the total memory used by the job and the runtime itself, it's the memory that the thread is using on your system. This includes our runtime code, the compiler, CLI, worker code (well, the stuff in the worker thread anyway), etc. This is rss memory.

This is snapshot reporting: it tells you the usage at the time you call the function. If garbage collection runs the moment before memoryUsage is called, the numbers will all be lower.

There is a performance cost associated with counting the memory (likely trivial in most cases, but there we go)

Default Reporting

By default we will report the job and system memory at the end of each job.

Memory usage will be logged in mb:

Final memory usage: job 60mb / system: 180 mb

For Lightning, we will add a mem property, with { job: number, system: number } values, in bytes (let's leave rounding and presentation as a frontend problem).

This will not impact performance at all (or at least, very minimally). It is likely to be misleading.

Trace reporting

The runtime will include a traceMemory option, exposed by the CLI and worker.

When enabled, it will log the memory used after each top-level operation.

This is still a bit misleading because an fn() operation might have a 100 lines of code, and we only read memory at the end. But still, it gives us slightly more accurate reporting.

We will use this to report PEAK memory usage (rather than final) at the end of a job:

Peak memory usage: job 60mb / system: 180 mb

This will be returned to lightning as { job: number, system: number } , as before. So it won't be explicit in the lightning event.

Other strategies?

We can listen for garbage collection events in node, but only after they've run. Even so, this will give us a better peak memory usage (at what performance cost I am not sure).

I am not sure what other options we have I suggest this is a starting point and see where it gets us.

from kit.