archiverjs / node-zip-stream Goto Github PK

What about a good zip reading API ?

Any advices ?

default time is UTC ,I want to change time by local, I found set forceLocalTime true isn't change!After read the source, I found entry.setTime(data.date). In compress-commons ,setTime can recieve 2 params,like that ZipArchiveEntry.prototype.setTime = function(time, forceLocalTime). I hope you can fix it. Thanks!

Try the following implementation and notice how it causes a dead lock:

'use strict'

const { promisify } = require('node:util')
const ZipStream = require('zip-stream')

const sleep = promisify(setTimeout)

async function* generateItems() {
  // Fake ressource usage. This will cause the process to never exit if the finally block is never executed.
  const interval = setInterval(() => {}, 10)

  console.error('generate start')

  let i = 0
  try {
    while (i++ < 5) {
      await sleep(1000)
      console.error('generate item', i)
      yield { data: Buffer.alloc(100_000_990), name: 'hello.txt', date: new Date() }
    }
  } finally {
    // Clean ressources
    clearInterval(interval)
    console.error('generate done', i)
  }
}

async function populateZip(zip, asyncIterable) {
  try {
    for await (const item of asyncIterable) {
      await new Promise((resolve, reject) => {
        zip.entry(item.data, { name: item.name, date: item.date }, (err, res) => {
          if (err) reject(err)
          else resolve(res)
        })
      })
    }
    zip.finalize()
  } catch (err) {
    console.error('populateZip err', err)
    // Item generation failed or zip.entry() failed
    zip.destroy(err)
  }
}

async function main() {
  const zip = new ZipStream({ zlib: { level: 9 } })

  populateZip(zip, generateItems())

  setTimeout(() => {
    zip.destroy()
  }, 1000)

  zip.once('error', (err) => {
    console.error('zip err', err)
  })

  zip.on('data', (chunk) => {
    console.error('got zip chunk of size', chunk.length)
  })
}

void main()

This is due to the fact that the entry callback is never called when the stream is destroyed.

IMHO the zip.entry callback should always be called, in this case with either a "the stream was destroyed" error or a (fake) success. The second option has the advantage of being backwards compatible.

$ node -v
v16.19.1

@harnasz had mentioned on archiverjs/node-archiver#53 that following issue:

When using Zip type with the store option set to true with more than 8 files I get the the warning below. I'm using Node v0.10.24

(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
    at EventEmitter.addListener (events.js:160:15)
    at Readable.on (_stream_readable.js:689:33)
    at ChecksumStream.Readable.pipe (_stream_readable.js:491:8)
    at ArchiverZip._processFile (streamer/node_modules/archiver/lib/archiver/zip.js:159:15)
    at Archiver._processQueue (streamer/node_modules/archiver/lib/archiver/core.js:121:10)
    at Archiver.append (streamer/node_modules/archiver/lib/archiver/core.js:182:8)

The error is not produced when setting the store option to 'false'.

I'm running archiver 0.6.1 which includes zip-stream 0.2.2

When building a specific zip, the code runs into an uncaught exception.
I haven't yet been able to provoke this error so as to reproduce it - but am dropping it in here in case you have an idea what might have provoked it, or for helping to harden the code.

If I do get it consistenly reproducible I'll update the issue.

Uncaught exception: value is out of bounds TypeError: value is out of bounds
at TypeError ()
at checkInt (buffer.js:784:11)
at Buffer.writeUInt32LE (buffer.js:841:5)
at /some/path/archiver/node_modules/zip-stream/lib/headers.js:49:32
at Array.forEach (native)
at ZipHeaderFileDescriptor.ZipHeader.toBuffer (/some/path/archiver/node_modules/zip-stream/lib/headers.js:36:15)
at Object.exports.encode (/some/path/archiver/node_modules/zip-stream/lib/headers.js:221:24)
at onend (/some/path/archiver/node_modules/zip-stream/lib/zip-stream.js:127:24)
at DeflateRawChecksum. (/some/path/archiver/node_modules/zip-stream/lib/zip-stream.js:142:5)
at DeflateRawChecksum.g (events.js:175:14)
at DeflateRawChecksum.EventEmitter.emit (events.js:117:20)

The file headers for zero length files should have a compression method of 0 (zero) (same as directories). Otherwise various zip utilities choke/complain.

See section 4.3.8 in https://www.pkware.com/documents/casestudies/APPNOTE.TXT

I ran into this warning message:

[email protected]: module has been merged into crc32-stream

Is it related to your module? If it's the case can you fix this warning.

-- [email protected] -- [email protected]
-- [email protected] -- [email protected]

Hello,

I'm wondering if it is possible to add files into the zip part by part. By that, i mean something like this :

archive.entry('part1', { name: 'file.txt' }, function(err, entry) {
  if (err) throw err;
  archive.entry('part2', { name: 'file.txt' }, function(err, entry) {
    if (err) throw err;
    archive.finish();
  });
});

Which create only one file without overwriting, but appending instead, with 'part1part2' as content.

If not, will it ever be implemented in the future ? If you have any other alternative, i'm taking it.

Thanks by advance

I have a variable amount of data I am reading out of a database in chunks and need to stream to an entry in a zip file.

I see that archive.entry() takes a readable stream potentially as it's first argument however I don't have a stream.

I am really looking for a way that archive.entry returns a writable stream that I can write my data to incrementally and then close the entry when the stream is closed.

Am I missing a way to do this?

When I use it to generate a zip file: A.zip, then I uncompress it and compress a file B.zip using system zip tool. Why the size of B.zip is smaller than A.zip.

In addition to the error reported in #166, destroying a zip stream will not be handled properly internally:

Running this code (slightly different from the one in #166) will return the error bellow

'use strict'

const { promisify } = require('node:util')
const ZipStream = require('zip-stream')

const sleep = promisify(setTimeout)

async function* generateItems() {
  // Fake ressource usage. This will cause the process to never exit if the finally block is never executed.
  const interval = setInterval(() => {}, 10)

  console.error('generate start')

  let i = 0
  try {
    while (i++ < 5) {
      await sleep(1000)
      console.error('generate item', i)
      yield { data: Buffer.alloc(100_000_990), name: 'hello.txt', date: new Date() }
    }
  } finally {
    // Clean ressources
    clearInterval(interval)
    console.error('generate done', i)
  }
}

async function populateZip(zip, asyncIterable) {
  try {
    for await (const item of asyncIterable) {
      await new Promise((resolve, reject) => {
        zip.entry(item.data, { name: item.name, date: item.date }, (err, res) => {
          if (err) reject(err)
          else resolve(res)
        })
      })
    }
    zip.finalize()
  } catch (err) {
    console.error('populateZip err', err)
    // Item generation failed or zip.entry() failed
    zip.destroy(err)
  }
}

async function main() {
  const zip = new ZipStream({ zlib: { level: 9 } })

  populateZip(zip, generateItems())

  setTimeout(() => {
    zip.destroy() // (1)
  }, 2000)

  zip.on('error', (err) => {
    console.error('zip err', err)
  })

  zip.on('data', (chunk) => {
    console.error('got zip chunk of size', chunk.length)
  })
}

void main()

generate start
generate item 1
got zip chunk of size 4
got zip chunk of size 2
got zip chunk of size 2
got zip chunk of size 2
got zip chunk of size 4
got zip chunk of size 4
got zip chunk of size 4
got zip chunk of size 4
got zip chunk of size 2
got zip chunk of size 2
got zip chunk of size 9
got zip chunk of size 16384
got zip chunk of size 16384
got zip chunk of size 16384
got zip chunk of size 16384
got zip chunk of size 16384
got zip chunk of size 12570
got zip chunk of size 2715
got zip chunk of size 4
got zip chunk of size 4
got zip chunk of size 4
got zip chunk of size 4
generate item 2
zip err NodeError: Cannot call write after a stream was destroyed
    at doWrite (~/node_modules/readable-stream/lib/_stream_writable.js:409:38)
    at writeOrBuffer (~/node_modules/readable-stream/lib/_stream_writable.js:398:5)
    at ZipStream.Writable.write (~/node_modules/readable-stream/lib/_stream_writable.js:307:11)
    at ZipStream.ArchiveOutputStream.write (~/node_modules/compress-commons/lib/archivers/archive-output-stream.js:116:36)
    at ZipStream.ZipArchiveOutputStream._writeLocalFileHeader (~/node_modules/compress-commons/lib/archivers/zip/zip-archive-output-stream.js:390:8)
    at ZipStream.ZipArchiveOutputStream._appendBuffer (~/node_modules/compress-commons/lib/archivers/zip/zip-archive-output-stream.js:74:8)
    at ZipStream.ArchiveOutputStream.entry (~/node_modules/compress-commons/lib/archivers/archive-output-stream.js:86:10)
    at ZipStream.entry (~/node_modules/zip-stream/index.js:166:49)
    at ~/test.js:32:13
    at new Promise (<anonymous>)
zip err NodeError: Cannot call write after a stream was destroyed
    at doWrite (~/node_modules/readable-stream/lib/_stream_writable.js:409:38)
    at writeOrBuffer (~/node_modules/readable-stream/lib/_stream_writable.js:398:5)
    at ZipStream.Writable.write (~/node_modules/readable-stream/lib/_stream_writable.js:307:11)
    at ZipStream.ArchiveOutputStream.write (~/node_modules/compress-commons/lib/archivers/archive-output-stream.js:116:36)
    at ZipStream.ZipArchiveOutputStream._writeLocalFileHeader (~/node_modules/compress-commons/lib/archivers/zip/zip-archive-output-stream.js:393:8)
    at ZipStream.ZipArchiveOutputStream._appendBuffer (~/node_modules/compress-commons/lib/archivers/zip/zip-archive-output-stream.js:74:8)
    at ZipStream.ArchiveOutputStream.entry (~/node_modules/compress-commons/lib/archivers/archive-output-stream.js:86:10)
    at ZipStream.entry (~/node_modules/zip-stream/index.js:166:49)
    at ~/test.js:32:13
    at new Promise (<anonymous>)
[... many more of these] 

$ node -v
v16.19.1

Just pointing out that this library does not work with AWS Lambda & S3 in a full streaming setup. It seems to die in the middle of the read stream and the whole process just quits.

I've had luck with another zip streaming library, though.

Description of problem and suggested solution can be found here: archiverjs/node-archiver#274 (comment)

Hi there,

Note that I've created this same issue on the "archiver" project here: archiverjs/node-archiver#602

I strongly suspect that the issue discussed below is an issue with this library as opposed to "archiver". I have tried using its "TAR" output and am not experiencing any issues.

I've been troubleshooting an issue where archiver appears to be generating corrupt archives. Using version 5.3.1 on node v16.13.1.

We're streaming in files that have been retrieved from ssh2-sftp-client.

This library seems to work fine with very large archives built from a few large files. It also seems to work fine for archives up to 199 files. However, when I have 200 files or more, the archive gets corrupted. By diffing the hexdumps of one archive that has 199 files and another that has 200, I can see the archive with 200 files is missing the "End of central directory record" (EOCD). See below for the bytes that are missing from my archive with 200 files (note the first four bytes below are the last part of the last filename in the archive).

0039b9b0 65 2e 70 70 50 4b 05 06 00 00 00 00 c6 00 c6 00 |e.ppPK..........|
0039b9c0 f4 3c 00 00 c0 7c 39 00 00 00 |.<...|9...|
0039b9ca
Otherwise, the generated files are pretty much identical (except for one less file being present in the "good" archive).

Are you aware of any file count limit for this library?

Many thanks,
Nick

I want to write to some sort of buffer rather than file system, and then send it of to the server in binary format, how would one do that?

There is an underlying issue affecting compress-commons. It can cause some streams to have an incorrect checksum value calculated.

After the issue is fixed in compress-commons, this package should be updated to use the newer version of compress-commons.

Hello,

Is there anyway to get the content-length?

prob should be looked into.

Description

When trying to add a string of considerable size such as >7 MB, the archive.entry exits abruptly without any exception and exit code - 0.

Below is sample code

const ZipStream = require('zip-stream');
const archive = new ZipStream(); 

function arrayToCsv(dataArray) {
    const output = dataArray.map((val) => {
        if (val === null || val === undefined) {
            return '';
        }
        const modVal = `${val}`; // Convert everything to string
        return `"${modVal.replace(/"/g, '""')}"`;
    }).join(separator);
    return output;
}

function getCSV(rows, cols = 10) {

    let csv = '';
    for (let i = 0; i < rows; i++) { // files

        const row = new Array(cols);
        row.fill(`randomString${i + 1}`);
        csv = csv + arrayToCsv(row) + '\n';
    }
    return csv;
}

process.on('exit', (code) => {
    console.log(`Process exit with code - ${code}`)
});

console.log(`Before adding csv`);
csv = getCSV(20000);

archive.entry(csv, { name: 'csv1.csv', zlib: { level: 9 } }, (err, entry) => {
    if (err) {
        throw err
    }
    console.log(`After adding csv`);
    archive.finish();
})

Output

Before adding csv
Process exit with code 0

The After adding csv is not printed to console.

I have used object-sizeof to calculate size of csv
size of csv - 7777880 OR 7.42 MB

For lower sizes, it works fine.

Please let me know if there is anything wrong with the code above.

I have three jpg files, and they have same name.
I deal with them like this:
archive.entry(filestream, { name: ${folderName + item.info.name} }, (err, entry) => { ... }
And the result is, I get a zip file, unarchive it, there is only one jpg.
Is there any options to avoid this?

BTW, the api documentation page is 404.

Is the generation of a zip with password supported ?
If so where can i find the options ?

Thank you !

A function to calculate the final size of the archive would be possible if all files were of known sizes and compression was set to STORE.

the time has come. only node v4+ will be supported in next release.

 error TypeError: Cannot read property 'pipesCount' of undefined
    at module.exports.Readable.pipe (_stream_readable.js:545:16)
    at module.exports.ZipArchiveOutputStream._smartStream (/path/to/node_modules/compress-commons/lib/archivers/zip/zip-archive-output-stream.js:184:11)
    at module.exports.ZipArchiveOutputStream._appendStream (/path/to/node_modules/compress-commons/lib/archivers/zip/zip-archive-output-stream.js:96:20)
    at module.exports.ArchiveOutputStream.entry (/path/to/node_modules/compress-commons/lib/archivers/archive-output-stream.js:88:10)
    at module.exports.ZipStream.entry (/path/to/node_modules/zip-stream/index.js:138:49)

my code that's calling it is pretty strait forward

function write (zip, stream, name) {
  return new Promise((resolve, reject) => {
    zip.entry(stream, {
      name: path.basename(name)
    }, function (err) {
      if (err) {
        return reject(err)
      }
      resolve()
    })
  })
}

from what I can tell DeflateCRC32Stream is not a real stream for some reason and I'm pretty stumped, ideas ?

registered archiverjs org to contain all the modules related to archiver and future plugins.

need to look into allowing a type of directory and what the proper zip header is for an empty directory.

文件夹里面文件很多的时候，会出现只解压缩处理部分文件，使用WinRAR 进行压缩成zip

Append file entry to an exist zip file instead of creating a new zip file.

I'm trying to write a ZIP file in chunks, by separate independent processes running at different times (but in the right serialized order). Ideally each process should append one single file, and the last process, after writing the last file, also finalizes the archive.

It would be great if it was possible to serialize the state of the archiver, so that each process can pick up where the previous one left.

I quickly read the code and it looks like there's really just a handful of internal variables that would need to be serialized, and probably JSON should suffice.

Would that be possible or is there some show stopper I'm not currently seeing?
If possible, what would be the internal variables to take care of? That would give me a head start for a PR.

Thanks for the great library! A work of art.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update actions/checkout action to v4.1.7
• Update actions/setup-node action to v4.0.3
• Update dependency rimraf to v5.0.10
• Update dependency chai to v4.5.0
• Update dependency mocha to v10.7.3
• Update dependency chai to v5
• Update dependency rimraf to v6
• Lock file maintenance
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Is it possible to specific no compression, so it will create a zip file, but not compress?

e.g. the pack test seems to fail against node 10, but succeeds against node 8

     TypeError: Cannot read property 'objectMode' of undefined
      at WriteHashStream.Writable.write (_stream_writable.js:271:22)

Hi , I am taking the output of this library and sending it over http, on the other side i have a test that needs to inflate the stream and run a couple of operations against the output. I am having a hard time decompressing the output using standard node libraries (zlib, pako). I keep running into the following error: "incorrect header check". Have been through the source of this as well as compress-commons for clues but haven't been able to figure it out.

this has been reported numerous times would be good to figure out whats causing it.

as reported at archiverjs/node-archiver#74. i cant personally verify this since no access.

@gconaty moving issue here as its the underlying lib that would need any adjustments.

I want to set file encoding as GKB, How to file encoding?

im considering bumping the requirements to node 4.0+ for v2.0.0 which will likely be the next release given an API change for symlink.

Can I get the size of the zip file so I can send a content-length header before piping it to res?
Otherwise, the user doesn't see how big the file is.
Something like that:

async.series(items, function (err, result) {
    archive.finish();

    // I would like to set the content-length header here and then pipe it

    archive
        .pipe(res)
        .on('end', end)
        .on('finish', end);
});

Is it possible? Or do I really have to write it to disk fs.createReadStream, read the size, and then pipe it to the user?

Hi, another packaging request here. Sorry I keep doing this to you. Here's the deal:

zip-stream depends on lodash.defaults which means to create a Debian package for zip-stream, I have to package up all of these things:

│  │  ├─ lodash.defaults (~2.4.1)                 None
│  │  │  ├─ lodash.keys (~2.4.1)                  None
│  │  │  │  ├─ lodash.isobject (~2.4.1)           None
│  │  │  │  │  └─ lodash._objecttypes (~2.4.1)    None
│  │  │  │  ├─ lodash._shimkeys (~2.4.1)          None
│  │  │  │  │  └─ lodash._objecttypes (~2.4.1)    None
│  │  │  │  └─ lodash._isnative (~2.4.1)          None
│  │  │  └─ lodash._objecttypes (~2.4.1)          None

I looked into it, and if you used utils-merge instead, it would work the same way, and utils-merge is already packaged up for Debian, so it would require no work on my part.

What do you think?

as far as i can tell, this project does not use lodash, and yet it is stated as a dependence in package.json. lodash is a quite large module (4.8MB), and so removing it would help to reduce node_module weight

Hi!

There seems to be an issue with the default Windows unzipper where a .zip file seems to be empty when the user double clicks the file:

https://answers.microsoft.com/en-us/ie/forum/ie11-windows_7/zip-file-download-appears-empty/7e9abb94-f52b-4e04-b1ec-5b5bd1504a21
https://answers.microsoft.com/en-us/windows/forum/windows_7-files/the-compressed-zipped-folder-is-empty/efb1a0db-73a3-479a-855d-f34ade79d2b9
https://answers.microsoft.com/en-us/windows/forum/windows_7-files/i-cant-extract-zipped-files-all-i-get-is-empty/4784eb7f-8adb-4654-81e3-c7f408f1ce52

An easy way around it seems to be to use 7zip instead, however, the users of my app who sometimes download zip files generated using zip-stream and are on Windows do not know this (and I would rather have the default Windows unzipper work rather than having to teach them 7zip).

So my question is: can I do something when using zip-stream to make the generated zip files more likely to be read by the default Windows unzipper? Are there any flags/options to set?