[Debug] Exhausted heap space about archive HOT 13 OPEN

Looks like I'll need to rework the cache for the zlib decompression a bit. The file seeking it needs to do is going outside the bounds of the in-memory cache, and causing file io thrashing. I had previously improved it for smaller compressed blocks, like in that Audios zip, but the other zips in your collection have much larger compressed blocks. Hopefully I can get that fixed up soon.

from archive.

If it's possible, maybe you could try the 4.0 branch I've been working on. You'd have to change your pubspec to the git path:

dependencies:
  archive:
    git:
      url: https://github.com/brendan-duncan/archive
      ref: '4.0'

I did a bunch of work on memory and file io performance in that branch, it would be great to know if it's enough to get past the iOS memory issues. It should also be considerably faster to extract that zip too. I tested the previous version and it took over a minute, and on the 4.0 branch it took a couple seconds.

from archive.

Thank you for your reply!
I will try this today in the evening, but this does not only happen on iOS but also on Android.

from archive.

Mobile is much more restrictive with memory.
The 4.0 branch is now in the main branch, it will be the next version update. So you can test without the git branch:

dependencies:
  archive:
    git:
      url: https://github.com/brendan-duncan/archive

from archive.

Is there a migration guide available?
Hitting this

The method 'decodeBuffer' isn't defined for the type 'ZipDecoder'.
Try correcting the name to the name of an existing method, or defining a method named 'decodeBuffer'.

from archive.

I assume it should be decodeStream but I am not sure.

from archive.

Yeah, decodeStream. There's a migration guide link in the README.

from archive.

I tried the new version in debug mode on Android and it is significantly slower, even so much that I cannot use this version.

v3.6.1

I/flutter (18876): assets/dict/dictionary.zip
I/flutter (18876): 0:00:01.509784
I/flutter (18876): assets/dict/examples.zip
I/flutter (18876): 0:00:01.104178
I/flutter (18876): assets/dict/krad.zip
I/flutter (18876): 0:00:00.250422
I/flutter (18876): assets/dict/radk.zip
I/flutter (18876): 0:00:00.170664
I/flutter (18876): assets/ipadic.zip
I/flutter (18876): 0:00:00.384508

master

I/flutter (14027): assets/dict/dictionary.zip
I/flutter (14027): 0:02:35.098774
I/flutter (14027): assets/dict/examples.zip
I/flutter (14027): 0:01:21.271891
I/flutter (14027): assets/dict/krad.zip
I/flutter (14027): 0:00:00.988176
I/flutter (14027): assets/dict/radk.zip
I/flutter (14027): 0:00:00.564761
I/flutter (14027): assets/ipadic.zip
I/flutter (14027): 0:00:26.827676

You can find the used files here.

from archive.

But on a positive note, the unpacking of the initially mentioned zip does now work on android in debug mode.
Thanks!

from archive.

I could add decodeBuffer back with a deprecated label if you think that would be easier.

No, I think it is fine. I just did not know where to find the migration guide

from archive.

Well shoot, back to the drawing board then. Sometimes Dart drives me a bit nuts. Must be some extra file io, probably file position seeking, causing the slowdown. Strange the tests I did were faster. Thanks for the report, I'll get it sorted out soon.

from archive.

Decoding a zip on dart:io platforms should be significantly faster with 4.0 now.

It will use native executed zlib for decompressing zip data on platforms that support it now. The dictionary.zip file was taking >1minute before, takes <1second now.

If you could run your tests on the update in git, that would be appreciated.

from archive.

Same problem here using latest 3.x or 4.0. I'm trying to unzip a 2.5 GB archive containing a 1 GB file, it fails on iOS (working on Android). Here's the code:

final input = InputFileStream("/path/to/my-archive.zip");
final archive = ZipDecoder().decodeStream(input);

for (final file in archive) {
  final filename = file.name;
  if (file.isFile) {
    final output = OutputFileStream('/path/to/$filename');
    output.writeStream(file.getContent()!);
    await file.close();
  } else {
    await Directory('${datasetsFolder.path}/$filename').create(recursive: true);
  }
}

And the log:

Exhausted heap space, trying to allocate 1044090912 bytes.
[ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Out of Memory
#0 new Uint8List (dart:typed_data-patch/typed_data_patch.dart:2236:3)
#1 new OutputMemoryStream (package:archive/src/util/output_memory_stream.dart:17:19)
#2 new Inflate.stream (package:archive/src/codecs/zlib/inflate.dart:37:29)
#3 ZipFile.getStream (package:archive/src/codecs/zip/zip_file.dart:232:21)
#4 ArchiveFile.decompress (package:archive/src/archive/archive_file.dart:208:40)
#5 ArchiveFile.getContent (package:archive/src/archive/archive_file.dart:139:7)
#6 ApplicationState.test (package:atlas/application.dart:77:33)
#7 _FutureListener.handleValue (dart:async/future_impl.dart:162:18)

Are you planning to support this use case or do I need to find a workaround?

from archive.