Possibility to reduce packages size by 4x about pkg HOT 26 CLOSED

Btw I wrote a node.js wrapper for UPX, https://github.com/roccomuso/upx

from pkg.

The idea behind pkg is to introduce minimal changes over original nodejs. Btw, if you run strip you can lose the ability to use native addons (.node files).

from pkg.

Some interesting projects we could get idea/ use:

• https://github.com/megastep/makeself
• http://www.peazip.org/self-extracting-archives.html

from pkg.

It seems that the nexe owner gets a clue on this: nexe/nexe#366 (comment)

He uses UPX over node.js itself (~/.nexe/linux-x64-8.9.4), and I can say it works.
However this do not work on the pkg equivalent: ~/.pkg-cache/v2.5/fetched-v8.9.0-linux-x64:

$ pkg -t node8-x64 app.js -o dist/app
> [email protected]
assert.js:42
  throw new errors.AssertionError({
  ^

AssertionError [ERR_ASSERTION]: Placeholder for not found
    at injectPlaceholder (~/project/node_modules/pkg/lib-es5/producer.js:217:38)
    at injectPlaceholders (~/project/node_modules/pkg/lib-es5/producer.js:236:3)
    at ~/project/node_modules/pkg/lib-es5/producer.js:158:9
    at FSReqWrap.oncomplete (fs.js:135:15)

Any idea?

from pkg.

Hey, any update on the matter? UPX + pkg doesn't seems to work so far and that would be really neat if it was.

from pkg.

FWIW:

Just tested here using gzexe (Linux) and works good. From 134Mb to 43MB, and i good only 1s performance penalty on load (did not test other runtime operations).

from pkg.

Referencing #121 for UPX issue (https://github.com/upx/upx).

from pkg.

@pi0 Do you see any potential impact your NodeApp's performance?

I'm considering this for my Ghost installation. Cheers!

from pkg.

@pi0 How did you do that compression? could you provide an example?

from pkg.

Hi,

as we're talking about bundle size, does pkg apply some kind of optimization / tree shaking / minification internally?

If I apply something like prepack + uglify before packaging, will my final packaged app be lighter?

from pkg.

running strip on the generated binary should also help on UNIX platforms

from pkg.

Hi. Sorry answering late :)

@pascalandy About performance impact: I think if we do extraction on startup, it may only affect startup time and not run-time itself.

@roccomuso My first test was just using tar! But it would be easy adding self-extract bits on binary's head.(exe inside exe)

@anmonteiro About js minification, pkg is absolutely not webpack! And i don't think this would help a lot, main size suffer is from Node.js binary itself. And we could just make a blind compression on the whole things. (Also @igorklopov is correct about .node files)

from pkg.

@anmonteiro strip doesn't really work. It reduces the binary size but the app doesn't run anymore. You get the interactive node console 😆

That is very weird. Why is this happening?

from pkg.

strip strips payload that contains your project packed. Without that payload the binary behaves like original node.js binary.

from pkg.

Oh hrm, sorry for the bad suggestion. I use strip with Nexe and it works, apparently it doesn't with pkg's approach. Sorry for the noise

from pkg.

What about the idea of using libsquash similar to https://github.com/pmq20/node-packer ?

upx may cause false positives on some antivirus detection

from pkg.

@Clement-TS I got the same error trying to compress the node binary before using pkg. Let's wait @igorklopov

from pkg.

Tried today with upx 3.95 on Linux and the problem still persists.
upx -oNodeJSProjectOutput --overlay=copy --strip-relocs=0 --compress-exports=0 --compress-icons=0 --compress-resources=0 --no-align NodeJSProject

from pkg.

My windows .exe build size reduced 7.xx MB from 21.xx MB with upx, thanks for mentioning it 🌟

from pkg.

EDIT: Here is an example of how I use upx to compress Nodejs in my Dockerfile for Ghost: https://github.com/firepress-org/ghostfire/blob/3.0.3-alpine/Dockerfile#L49

from pkg.

My windows .exe build size reduced 7.xx MB from 21.xx MB with upx, thanks for mentioning it 🌟

Can you say what parameters you used to pack Windows .exe? I tried (UPX 3.95 Win10x64)

upx -o"V:\packed.exe" --overlay=copy --strip-relocs=0 --compress-exports=0 --compress-icons=0 --compress-resources=0 --no-align "V:\app.exe"

but the executable is failing to start.
Thanks in advance!

from pkg.

My windows .exe build size reduced 7.xx MB from 21.xx MB with upx, thanks for mentioning it star2

Can you say what parameters you used to pack Windows .exe? I tried (UPX 3.95 Win10x64)

upx -o"V:\packed.exe" --overlay=copy --strip-relocs=0 --compress-exports=0 --compress-icons=0 --compress-resources=0 --no-align "V:\app.exe"

but the executable is failing to start.
Thanks in advance!

I didn't use any special flags. Just dragged my .exe and dropped over upx. Thats what I did

from pkg.

I didn't use any special flags. Just dragged my .exe and dropped over upx. Thats what I did

Thanks, but just Drag&Drop didn't work for me. I'll try packaging with Nexe later, I've read somewhere that they were natively optimised for upx (and support customising .exe file icon out of the box)

from pkg.

Any progress here? When I compress node.exe then it works, but whe I use pkg I always get Pkg: Error reading from file., even for simple console.log().

from pkg.

Regarding UPX:
https://sourceforge.net/p/upx/discussion/6806/thread/79e2a6b8/
That also explains why it works fine on windows, since PE executables and their overlays can be and are handled by the upx code.

As it is now, the readPrelude function patched into the nodejs binary will open /proc/self/exe and try reading the overlay from the offset replaced into the binary at packaging time. However since the binary found at /proc/self/exe is the compressed version, all those offsets are of course just wrong and even if they weren't they would still read compressed data.

To get pkg to work with upx, one approach could be to update the readPrelude function so that it uses relative offsets from the end of the file and extend the pkg process so that it first bakes in the commandline options, then packs that binary and finally appends the rest of the code.

That would still result in the js code itself being uncompressed, however, it would at least be able to shrink the nodejs binary by a fair amount.

Another approach could be to change the way the js payload is stored in the final binary so that it is accessible by the code in the readPrelude function directly instead of it calling fs.readSync on /proc/self/exe. That way, it might even be possible to keep the pkg build process as it is.
Since we do seem to be able to template the offsets into that function, why not template in the whole code?

Edit: because it will segfault if the length of that string changes. :)

from pkg.

I've just checked a built binary in a hex editor and discovered that

FILESIZE - PRELUDE_SIZE = PRELUDE_POSITION

as well as

FILESIZE - PRELUDE_SIZE - PAYLOAD_SIZE = PAYLOAD_POSITION

Is there any reason I'm missing for why these two positions are stored at all?

from pkg.