zips in s3 bucket are corrupted about aws-s3-zipper HOT 10 CLOSED

Thank you for the tip, will try to update archiver module.

Do you use your module wrapped into AWS Lambda function?

As side note I had to change this line

https://github.com/DanielHindi/aws-s3-zipper/blob/master/index.js#L187

to this

var tempFile = '/tmp/__' + Date.now() + '.zip';

to make it work in AWS Lambda, as tmp directory is the only writable place for lambdas

from aws-s3-zipper.

I believe 1.0.1 has the potential to fix your issue

from aws-s3-zipper.

So I use this module on a daily basis and zips up hundreds of megabytes of images yet in smaller zip increments.

I'm using npm/archiver to zip. It seems there are updated on [archiver] that may solve this. It needs to be tested.

That being said, you think 100-200 images at 2-5mb each would be a good test?

from aws-s3-zipper.

No i dont use it in Lambda... its triggered in an api.

if you make the tempFile configurable and send it in a pull request I'll accept it

from aws-s3-zipper.

@d3m3tr1s @DanielHindi

https://github.com/DanielHindi/aws-s3-zipper/blob/master/index.js#L265
https://github.com/DanielHindi/aws-s3-zipper/blob/master/index.js#L303

I think this line assume all the file will be zip in that timeframe (1s) which might be fine for small zip files, zip.on('finish') would be a better solution to avoid corruption.

ref:
https://archiverjs.com/docs/Archiver.html#finalize
https://github.com/archiverjs/node-archiver/blob/master/examples/pack-zip.js

from aws-s3-zipper.

Thank you for the tip, @n7best !

from aws-s3-zipper.

The call back should only happen when the file has been zipped and released the problem is the module keeps a lock on the file for a few moments after the callback. Thats why i put a breathe time before moving on

from aws-s3-zipper.

https://github.com/DanielHindi/aws-s3-zipper/blob/master/index.js#L144
finalize or append from archiver does not guarantee the file is zipped.

from aws-s3-zipper.

The end, close or finish events on the destination stream may fire right after calling this method so you should set listeners beforehand to properly detect stream completion.

Which means there must be a different event or callback that gives you the true release of the file

output.on('close', function() {
  console.log(archive.pointer() + ' total bytes');
  console.log('archiver has been finalized and the output file descriptor has closed.');
});

this might be it. need to test

from aws-s3-zipper.

I test all three, not sure why, only finish works for me. My files are pretty large.

from aws-s3-zipper.