Comments (27)
Before a more official solution is in place, this is what we ended up using:
# serverless.yml
custom:
serverless-offline:
allowCache: true # Important: This will prevent serverless-offline from eating all of your memory
# ...
esbuild:
plugins : ./esbuild/plugins.js
# ....
// esbuild/plugins.js
let refreshServer = {
name: "refresh-server",
setup(build) {
build.onEnd((result) => {
if (build.initialOptions.incremental) {
console.log(`refresh-server: Clearing Cache for ${build.initialOptions.entryPoints.join(", ")}...`);
// Remove all items from the cache (this will force node to reload all of the built artifacts)
Object.keys(require.cache).forEach(function (key) {
const resolvedPath = require.resolve(key);
if (resolvedPath.includes(build.initialOptions.outdir)) {
delete require.cache[key];
}
});
}
});
},
};
module.exports = [refreshServer];
By utilising the esbuild plugin system we just did the clearing there instead of in the serverless-esbuild
side. We now have a watch that can run for a long time, that gets updated whenever we change code and our memory usage remains relatively static. Would strongly recommend for anyone running into any of these memory issues or having to restart serverless whenever code changes (because of attempts to resolve the memory issues).
Hope that helps someone else...
from serverless-esbuild.
I rewrote it to use rebuild. Sadly this does not resolve the memory issue. #101 so I think it might actually be an esbuild issue.
But I am unable to verify it with:
async function example() {
let result = await require('esbuild').build({
entryPoints: ['src/index.ts'],
bundle: true,
outfile: 'out.js',
incremental: true,
});
for (let i = 0; i < 150; i++) {
console.log('a' + i);
await result.rebuild();
}
console.log('done');
}
example();
from serverless-esbuild.
In addition to ameliorating memory eating-issues, I was also having issues where hot reloading wasn't working for me, and the solution from @mattisdada also fixed that. Thank you!
from serverless-esbuild.
@mattisdada I'm trying to implement your solution on a TypeScript project with around 8 functions, but I'm getting a weird issue, after running serverless offline
, make one code change and save, the 'refreshServer' plugin runs in an infinite loop incrementing the ram usage until exhaustion.
// serverless.ts
custom: {
'serverless-offline': {
apiKey: API_KEY,
allowCache: true,
},
esbuild: {
bundle: true,
minify: false,
sourcemap: true,
exclude: ['aws-sdk', 'pg-native'],
target: 'node14',
define: { 'require.resolve': undefined },
platform: 'node',
concurrency: 10,
plugins: './esbuild/plugins.js',
},
},
Any idea why this could be happening?
from serverless-esbuild.
I can confrim this. I am running into FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
continuously - and my colleagues on the project too.
It is probably linked to the alchemy between chokidar
and esbuild
.
from serverless-esbuild.
From what I can see this does the leak:
serverless-esbuild/src/index.ts
Line 160 in ad3e93e
true
option here is using incremental bundling. Maybe a lead.from serverless-esbuild.
incremental build does keep things in memory to speed up subsequent bundling: https://esbuild.github.io/api/#incremental. Could it be a leak in esbuild itself? Should we call dispose
to free up resources from time to time ?
Or we could leave the watching to esbuild itself : https://esbuild.github.io/api/#watch, and see if memory is better handled but I guess this was sub-optimal @floydspace ?
from serverless-esbuild.
@olup I think it's just used incorrectly. We need to call rebuild()
instead of initiating a new esbuild instance with build()
. That way rebuilding would be even faster aswell.
from serverless-esbuild.
Yes true, otherwise the cache is a bit lost anyway. I'll flesh a pr tomorrow. Beyond this one subject, serverless offline, in watch mode, is terribly leaking so the rest of the process will end up eating the whole memory anyway. But we can improve what depends on us.
from serverless-esbuild.
I am reporting I've had similar issues, each build just linearly increases the amount of memory esbuild is using (about 2.5GB per file change, it pretty quickly gets up to 20GB), if I was to hazard a guess, I'm thinking it has something to do with node_modules
and this quote from the esbuild docs:
Files are stored in memory and are not re-read from the file system if the file metadata hasn't changed since the last build. This optimization only applies to file system paths. It does not apply to virtual modules created by plugins.
I have a feeling something is causing all the metadata change from ESBuilds perspective and it might not garbage collect old metadata (I imagine it doesn't watch for that). But I haven't looked at any of the code for serverless-esbuild
so I might be completely wrong...
incremental build does keep things in memory to speed up subsequent bundling: https://esbuild.github.io/api/#incremental. Could it be a leak in esbuild itself? Should we call
dispose
to free up resources from time to time ?Or we could leave the watching to esbuild itself : https://esbuild.github.io/api/#watch, and see if memory is better handled but I guess this was sub-optimal @floydspace ?
I'd be curious what this responds back with when logging is set to max
I might experiment a little tonight with this
from serverless-esbuild.
I've been experiencing the same problem of exponential growth in memory usage.
As far as I can tell, rebuild
seems to have better memory usage results than calling build
every time.
With the changes in my PR, memory usage seems to cap out around 1 GB (for my current codebase), whereas it used to increase exponentially until crashing.
Let me know if the PR #123 solves your problem @olup or @thomaschaaf.
from serverless-esbuild.
That PR partially fixed the issue for me (it still uses a lot of memory, but not as much), but serverless-offline
still consumes a lot of memory unless allowCache
is enabled and eventually exhausts itself (and dies) - this may be an issue with our particular code base though. Not sure if others are experiencing this particular pain point.
The problem of enabling allowCache
is that when ESBuild recompiles, the running code does not change to the newly compiled files, so to see changes it requires resetting the server or having a large memory leak.
I put in a very blunt workaround in the compiled serverless-esbuild
files after the build that does seem to work wonderfully which is just:
Object.keys(require.cache).forEach(function(key) { delete require.cache[key] })
This patched version + using serverless-offline
in allowCache mode, has allowed us to have best of both worlds. An "official" solution should only clear the cache of the compiled files, not everything like this...
Not sure if serverless-offline has an "official" way of bundlers to manage cache...
from serverless-esbuild.
Indeed, I put the link somewhere, but the memory leak in serverless offline is well known and affecting everyone, and the relation with the caching option is discussed around, but the accepted answer seems to be "that's how it is". So no official way to fix this.
A workaround in our codebase would be a very good idea, as the memory leak is a real pain, thanks a lot for sharing this ! Only we should probably put it behind some options and disabled by default, and document it.
from serverless-esbuild.
Brilliant ! And it's much better than having inside the lib, indeed.
But it is such a central pb with serverless offline we should definitely link this in the doc!
from serverless-esbuild.
Only small problem is the esbuild plugins don't get any serverless context, I'll continue playing around with this and see if I find any issues regarding that.
EDIT: Already ran into an issue with this if it's running during pack
it'll cause some issue with the archiver lib (should hopefully be resolved by checking if incremental is enabled)
I made a few changes to the plugin to be much more selective instead of clearing out everything, it should only clear out entries in the esbuild output directory.
from serverless-esbuild.
@mattisdada your solution seems to work for use with require
, is there a equivalent solution for when using ES Modules?
from serverless-esbuild.
@mattisdada I'm trying to implement your solution on a TypeScript project with around 8 functions, but I'm getting a weird issue, after running
serverless offline
, make one code change and save, the 'refreshServer' plugin runs in an infinite loop incrementing the ram usage until exhaustion.// serverless.ts custom: { 'serverless-offline': { apiKey: API_KEY, allowCache: true, }, esbuild: { bundle: true, minify: false, sourcemap: true, exclude: ['aws-sdk', 'pg-native'], target: 'node14', define: { 'require.resolve': undefined }, platform: 'node', concurrency: 10, plugins: './esbuild/plugins.js', }, },
Any idea why this could be happening?
Hmm, it's a pretty simple and dumb plugin, so I have to guess the events is being mangled somehow, none of the actions it's doing should be triggering a new esbuild build. Out of curiosity when it is removed (but nothing else has been changed) does the infinite loop still occur? I have to guess it's a file system related issue (esbuild is generating something that is then being picked up by esbuild, and repeats; or something similiar)
I'm also not sure why this was added, I can only expect it to break things 🤔 :
define: { 'require.resolve': undefined },
, it will def break that plugin but it might break everything. I'd actually try removing that first unless you have a particular reason for needing it
from serverless-esbuild.
@mattisdada I'm trying to implement your solution on a TypeScript project with around 8 functions, but I'm getting a weird issue, after running
serverless offline
, make one code change and save, the 'refreshServer' plugin runs in an infinite loop incrementing the ram usage until exhaustion.// serverless.ts custom: { 'serverless-offline': { apiKey: API_KEY, allowCache: true, }, esbuild: { bundle: true, minify: false, sourcemap: true, exclude: ['aws-sdk', 'pg-native'], target: 'node14', define: { 'require.resolve': undefined }, platform: 'node', concurrency: 10, plugins: './esbuild/plugins.js', }, },
Any idea why this could be happening?
Hmm, it's a pretty simple and dumb plugin, so I have to guess the events is being mangled somehow, none of the actions it's doing should be triggering a new esbuild build. Out of curiosity when it is removed (but nothing else has been changed) does the infinite loop still occur? I have to guess it's a file system related issue (esbuild is generating something that is then being picked up by esbuild, and repeats; or something similiar)
I'm also not sure why this was added, I can only expect it to break things 🤔 :
define: { 'require.resolve': undefined },
, it will def break that plugin but it might break everything
When the plugin is removed, and after a single save the memory usage keeps incrementing, I have stopped the process when it reaches to 14 GB or so.
I removed the line you mentioned define: { 'require.resolve': undefined },
, and ran serverless offline
again, made a dummy code change and saved, the ram usage keeps incrementing but somehow stabilizes between the 5-9 GB, in the meanwhile, the plugin is in the infinite loop.
from serverless-esbuild.
@mattisdada I'm trying to implement your solution on a TypeScript project with around 8 functions, but I'm getting a weird issue, after running
serverless offline
, make one code change and save, the 'refreshServer' plugin runs in an infinite loop incrementing the ram usage until exhaustion.// serverless.ts custom: { 'serverless-offline': { apiKey: API_KEY, allowCache: true, }, esbuild: { bundle: true, minify: false, sourcemap: true, exclude: ['aws-sdk', 'pg-native'], target: 'node14', define: { 'require.resolve': undefined }, platform: 'node', concurrency: 10, plugins: './esbuild/plugins.js', }, },
Any idea why this could be happening?
Hmm, it's a pretty simple and dumb plugin, so I have to guess the events is being mangled somehow, none of the actions it's doing should be triggering a new esbuild build. Out of curiosity when it is removed (but nothing else has been changed) does the infinite loop still occur? I have to guess it's a file system related issue (esbuild is generating something that is then being picked up by esbuild, and repeats; or something similiar)
I'm also not sure why this was added, I can only expect it to break things 🤔 :
define: { 'require.resolve': undefined },
, it will def break that plugin but it might break everythingWhen the plugin is removed, and after a single save the memory usage keeps incrementing, I have stopped the process when it reaches to 14 GB or so.
I removed the line you mentioned
define: { 'require.resolve': undefined },
, and ranserverless offline
again, made a dummy code change and saved, the ram usage keeps incrementing but somehow stabilizes between the 5-9 GB, in the meanwhile, the plugin is in the infinite loop.
I think you've got something else going wrong that's triggering the builds to re-trigger unrelated to the plugin:
Try just logging that esbuild build is being triggered:
// esbuild/plugins.js
let refreshServer = {
name: "refresh-server",
setup(build) {
build.onEnd((result) => {
if (build.initialOptions.incremental) {
console.log(`refresh-server: Clearing Cache for ${build.initialOptions.entryPoints.join(", ")}...`);
}
});
},
};
module.exports = [refreshServer];
Do you have anything that would be writing to your src directory? And what other plugins do you have installed?
from serverless-esbuild.
@mattisdada I'm trying to implement your solution on a TypeScript project with around 8 functions, but I'm getting a weird issue, after running
serverless offline
, make one code change and save, the 'refreshServer' plugin runs in an infinite loop incrementing the ram usage until exhaustion.// serverless.ts custom: { 'serverless-offline': { apiKey: API_KEY, allowCache: true, }, esbuild: { bundle: true, minify: false, sourcemap: true, exclude: ['aws-sdk', 'pg-native'], target: 'node14', define: { 'require.resolve': undefined }, platform: 'node', concurrency: 10, plugins: './esbuild/plugins.js', }, },
Any idea why this could be happening?
Hmm, it's a pretty simple and dumb plugin, so I have to guess the events is being mangled somehow, none of the actions it's doing should be triggering a new esbuild build. Out of curiosity when it is removed (but nothing else has been changed) does the infinite loop still occur? I have to guess it's a file system related issue (esbuild is generating something that is then being picked up by esbuild, and repeats; or something similiar)
I'm also not sure why this was added, I can only expect it to break things 🤔 :
define: { 'require.resolve': undefined },
, it will def break that plugin but it might break everythingWhen the plugin is removed, and after a single save the memory usage keeps incrementing, I have stopped the process when it reaches to 14 GB or so.
I removed the line you mentioneddefine: { 'require.resolve': undefined },
, and ranserverless offline
again, made a dummy code change and saved, the ram usage keeps incrementing but somehow stabilizes between the 5-9 GB, in the meanwhile, the plugin is in the infinite loop.I think you've got something else going wrong that's triggering the builds to re-trigger unrelated to the plugin:
Try just logging that esbuild build is being triggered:
// esbuild/plugins.js let refreshServer = { name: "refresh-server", setup(build) { build.onEnd((result) => { if (build.initialOptions.incremental) { console.log(`refresh-server: Clearing Cache for ${build.initialOptions.entryPoints.join(", ")}...`); } }); }, }; module.exports = [refreshServer];Do you have anything that would be writing to your src directory? And what other plugins do you have installed?
I don't think there is anything writing to the src dir. We have the following plugins: serverless-esbuild, serverless-offline, serverless-openapi-documentation-v2 and serverless-hooks.
This is a small sample of the logs (the plugin is just logging):
refresh-server: Clearing Cache for src/functions/internal/status/handler.ts...
refresh-server: Clearing Cache for src/functions/merchant/handler.ts...
refresh-server: Clearing Cache for src/functions/webhooks/handler.ts...
refresh-server: Clearing Cache for src/functions/hello/handler.ts...
refresh-server: Clearing Cache for src/functions/loanRequests/handler.ts...
refresh-server: Clearing Cache for src/functions/webhooks/handler.ts...
refresh-server: Clearing Cache for src/functions/webhooks/handler.ts...
refresh-server: Clearing Cache for src/functions/internal/status/handler.ts...
refresh-server: Clearing Cache for src/functions/hello/handler.ts...
refresh-server: Clearing Cache for src/functions/hello/handler.ts...
The ram usage never stops incrementing.
from serverless-esbuild.
The revised plugin only does logging, and does nothing else, your problem is outside the plugin. I recommend copying the project elsewhere and just keep removing things until it works as expected. (plugins, functions, config, etc)
I suspect serverless-hooks
might be the problem depending on what they are configured to do, but I cannot really say without looking at the complete project sorry. But you have something going on that is triggering esbuild to run continuously (almost certainly something writing to disk). I'd also make sure that the api doc generation is outside the directory that esbuild is looking at as well or ignore it for compilation.
Once you've discovered the source, modify the watch pattern
and ignore
, example:
https://github.com/floydspace/serverless-esbuild#serverless-offline
from serverless-esbuild.
The revised plugin only does logging, and does nothing else, your problem is outside the plugin. I recommend copying the project elsewhere and just keep removing things until it works as expected. (plugins, functions, config, etc)
I suspect
serverless-hooks
might be the problem depending on what they are configured to do, but I cannot really say without looking at the complete project sorry. But you have something going on that is triggering esbuild to run continuously (almost certainly something writing to disk). I'd also make sure that the api doc generation is outside the directory that esbuild is looking at as well or ignore it for compilation.Once you've discovered the source, modify the watch
pattern
andignore
, example: https://github.com/floydspace/serverless-esbuild#serverless-offline
Thank very much for the guidance, I'll look into it and get back with the fix.
from serverless-esbuild.
In my cause I had the same issue before I refactored packages import. When I have import like this:
import * as firebaseAdmin from 'firebase-admin' - memory increases exponentially on rebuild.
Once I rewrote import into following approach
import { initializeApp, getApp, getApps, cert } from 'firebase-admin/app'; - memory leaks fixed.
from serverless-esbuild.
The revised plugin only does logging, and does nothing else, your problem is outside the plugin. I recommend copying the project elsewhere and just keep removing things until it works as expected. (plugins, functions, config, etc)
I suspectserverless-hooks
might be the problem depending on what they are configured to do, but I cannot really say without looking at the complete project sorry. But you have something going on that is triggering esbuild to run continuously (almost certainly something writing to disk). I'd also make sure that the api doc generation is outside the directory that esbuild is looking at as well or ignore it for compilation.
Once you've discovered the source, modify the watchpattern
andignore
, example: https://github.com/floydspace/serverless-esbuild#serverless-offlineThank very much for the guidance, I'll look into it and get back with the fix.
@mrjavi91 Did you ever find a fix? I'm having the exact same issue
from serverless-esbuild.
@SamPF53 No, I haven't found a fix yet. I tried what @svoitovych0218 suggested, with no luck. Did you try that too? Please let me know if that works for you or not.
from serverless-esbuild.
@mrjavi91 - I just upgraded to the latest esbuild (0.15.5) and latest serverless-esbuild (1.32.8) and am not getting the endless loop anymore! Hopefully that resolves it for you. I was having an issue with hot reloading as well, but realized the handler was never reloading. Note for anyone having issues with hot reloading to set the --reloadHandler
flag. In package.json
:
"scripts": {
"local": "set NODE_OPTIONS=\"--enable-source-maps\"&& serverless offline --stage local --reloadHandler",
from serverless-esbuild.
Anyone still experiencing this with Node20 or Node21? I'm using this pattern which seems to cause the memory to grow until it runs the executor out of resources. Removing the plugins does prevent the memory issue but I bump into other problems with the build regarding node internals and external dependencies.
from serverless-esbuild.
Related Issues (20)
- Bun support HOT 1
- Unable to serverless package when no lambdas exist HOT 3
- serverless deploy fails without a reason when using esbuild-plugin-tsc HOT 2
- Rewrite Module Packaging Logic HOT 3
- I'm not sure if it supports layers
- pnpm: external modules bundled without their nested modules. HOT 2
- Indirect dependency marked as external missing in package HOT 1
- Slow performance when using external property
- Esbuild cannot rebuild error
- `patterns` not respected at `invoke local`
- sls package fails with individually: true for specific function Error: ENOENT: no such file or directory, copyfile
- update esbuild peerDependency HOT 1
- Support ESBuild 0.20.0 HOT 2
- watch function work with serverless-localstack.
- Possible memory-use regression in v1.48.1 HOT 1
- Stuck on Need a faster logging experience when using serverless-stack with serverless-esbuild HOT 1
- Support Lambda custom runtimes HOT 2
- Trouble importing to esbuild.config.js from an internal monorepo package HOT 1
- Hangs indefinitely on esbuild error HOT 1
- Overriding the keyword used to get the handler in the lambda function defintion HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from serverless-esbuild.