instantiations / es_compression Goto Github PK

I added the esbrotli-mac64.dylib in Pod file

s.vendored_libraries = 'esbrotli-mac64.dylib'

The following error will throw when building

ld: library not found for -lesbrotli-mac64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

But libesbrotli-mac64.dylib is working.

Its not entirely clear what the min/max values for compression levels are in the lz4 lib.

Running for command line, using Dart

• Dart version: Dart SDK version: 3.3.3 (stable) (Tue Mar 26 14:21:33 2024 +0000) on "macos_arm64"
• es_compression version:

    dependency: "direct main"
    description:
      name: es_compression
      sha256: "554d47db69c40eb8647e4c1f29e68223c834cc3d0602976252642aa50afb2faa"
      url: "https://pub.dev"
    source: hosted
    version: "2.0.11"

Full strack trace:

Invalid argument(s): Failed to load dynamic library '/Users/[anon]/.pub-cache/hosted/pub.dev/es_compression-2.0.11/lib/src/lz4/blobs/eslz4-mac64.dylib': dlopen(/Users/[anon]/.pub-cache/hosted/pub.dev/es_compression-2.0.11/lib/src/lz4/blobs/eslz4-mac64.dylib, 0x0001): tried: '/Users/[anon]/.pub-cache/hosted/pub.dev/es_compression-2.0.11/lib/src/lz4/blobs/eslz4-mac64.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/[anon]/.pub-cache/hosted/pub.dev/es_compression-2.0.11/lib/src/lz4/blobs/eslz4-mac64.dylib' (no such file), '/Users/[anon]/.pub-cache/hosted/pub.dev/es_compression-2.0.11/lib/src/lz4/blobs/eslz4-mac64.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64'))
#0      _open (dart:ffi-patch/ffi_dynamic_library_patch.dart:11:43)
#1      new DynamicLibrary.open (dart:ffi-patch/ffi_dynamic_library_patch.dart:22:12)
#2      OpenLibraryStrategy.open (package:es_compression/src/framework/native/library/open_library_strategy.dart:38:28)
#3      OpenViaPackageRelativeStrategy.openFor (package:es_compression/src/framework/native/library/package_relative_strategy.dart:46:12)
#4      OpenLibrary.openLibrary (package:es_compression/src/framework/native/library/open_library.dart:44:32)
#5      new Lz4Library._ (package:es_compression/src/lz4/ffi/library.dart:54:20)
#6      Lz4Library._instance (package:es_compression/src/lz4/ffi/library.dart:34:50)
#7      Lz4Library._instance (package:es_compression/src/lz4/ffi/library.dart)
#8      new Lz4Library (package:es_compression/src/lz4/ffi/library.dart:49:27)
#9      new Lz4Dispatcher (package:es_compression/src/lz4/ffi/dispatcher.dart:47:31)
#10     new Lz4DecompressFilter (package:es_compression/src/lz4/ffi/decompress_filter.dart:22:37)
#11     _makeLz4DecompressFilter (package:es_compression/src/lz4/decoder.dart:57:5)
#12     new _Lz4DecoderSink._ (package:es_compression/src/lz4/decoder.dart:47:13)
#13     Lz4Decoder.startChunkedConversion (package:es_compression/src/lz4/decoder.dart:38:28)
#14     CodecConverter.convert (package:es_compression/src/framework/converters.dart:20:7)

es_compression was always designed with the intent of using ffi for algorithm implementations.
However, it may be good to keep the door open for algorithm implementations in pure dart so it can be used in other contexts.

This means a slight refactor to CodecBuffer

• Its _bytes is a Pointer. Not a big change to parameterize this

The current lz4 decoder/encoder/options refer to the lz4 c library constants and dispatcher.

Hello, i'm trying to decompress .pak file, but i'm getting this error

Unhandled Exception: FormatException: ERROR_frameType_unknown

var decompressedData = Lz4Codec().decode(pakBytes);

[ +129 ms] E/flutter (29516): [ERROR:flutter/lib/ui/ui_dart_state.cc(186)] Unhandled Exception: Invalid argument(s): Failed to load dynamic library 'esbrotli-android64.so': dlopen failed: library "esbrotli-android64.so" not found
[        ] E/flutter (29516): #0      _open (dart:ffi-patch/ffi_dynamic_library_patch.dart:11:55)
[        ] E/flutter (29516): #1      new DynamicLibrary.open (dart:ffi-patch/ffi_dynamic_library_patch.dart:20:12)
[        ] E/flutter (29516): #2      OpenLibraryStrategy.open (package:es_compression/src/framework/native/library/open_library_strategy.dart:35:29)
[        ] E/flutter (29516): #3      OpenViaOsResolutionStrategy.openFor (package:es_compression/src/framework/native/library/os_resolution_strategy.dart:26:12)
[        ] E/flutter (29516): #4      OpenLibrary.openLibrary (package:es_compression/src/framework/native/library/open_library.dart:43:32)
[        ] E/flutter (29516): #5      new BrotliLibrary._ (package:es_compression/src/brotli/ffi/library.dart:58:20)
[        ] E/flutter (29516): #6      BrotliLibrary._instance (package:es_compression/src/brotli/ffi/library.dart:36:21)
[        ] E/flutter (29516): #7      BrotliLibrary._instance (package:es_compression/src/brotli/ffi/library.dart)
[        ] E/flutter (29516): #8      new BrotliLibrary (package:es_compression/src/brotli/ffi/library.dart:52:12)
[        ] E/flutter (29516): #9      new BrotliDispatcher (package:es_compression/src/brotli/ffi/dispatcher.dart:65:34)
[        ] E/flutter (29516): #10     new BrotliCompressFilter (package:es_compression/src/brotli/ffi/compress_filter.dart:20:40)
[        ] E/flutter (29516): #11     _makeBrotliCompressFilter (package:es_compression/src/brotli/encoder.dart:179:10)
[        ] E/flutter (29516): #12     new _BrotliEncoderSink._ (package:es_compression/src/brotli/encoder.dart:148:13)
[        ] E/flutter (29516): #13     BrotliEncoder.startChunkedConversion (package:es_compression/src/brotli/encoder.dart:115:31)
[        ] E/flutter (29516): #14     CodecConverter.convert (package:es_compression/src/framework/converters.dart:20:7)
[        ] E/flutter (29516): #15     Codec.encode (dart:convert/codec.dart:21:32)
[        ] E/flutter (29516): #16     _LbWhiteboardState.initState.<anonymous closure> (package:pathon/home/ui/whiteboard_widget.dart:36:31)
[        ] E/flutter (29516): #17     _rootRunUnary (dart:async/zone.dart:1362:47)
[        ] E/flutter (29516): #18     _CustomZone.runUnary (dart:async/zone.dart:1265:19)
[        ] E/flutter (29516): #19     _CustomZone.runUnaryGuarded (dart:async/zone.dart:1170:7)
[        ] E/flutter (29516): #20     _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:341:11)
[        ] E/flutter (29516): #21     _DelayedData.perform (dart:async/stream_impl.dart:591:14)
[        ] E/flutter (29516): #22     _StreamImplEvents.handleNext (dart:async/stream_impl.dart:706:11)
[        ] E/flutter (29516): #23     _PendingEvents.schedule.<anonymous closure> (dart:async/stream_impl.dart:663:7)
[        ] E/flutter (29516): #24     _rootRun (dart:async/zone.dart:1346:47)
[        ] E/flutter (29516): #25     _CustomZone.run (dart:async/zone.dart:1258:19)
[        ] E/flutter (29516): #26     _CustomZone.runGuarded (dart:async/zone.dart:1162:7)
[        ] E/flutter (29516): #27     _CustomZone.bindCallbackGuarded.<anonymous closure> (dart:async/zone.dart:1202:23)
[        ] E/flutter (29516): #28     _rootRun (dart:async/zone.dart:1354:13)
[        ] E/flutter (29516): #29     _CustomZone.run (dart:async/zone.dart:1258:19)
[        ] E/flutter (29516): #30     _CustomZone.runGuarded (dart:async/zone.dart:1162:7)
[        ] E/flutter (29516): #31     _CustomZone.bindCallbackGuarded.<anonymous closure> (dart:async/zone.dart:1202:23)
[        ] E/flutter (29516): #32     _microtaskLoop (dart:async/schedule_microtask.dart:40:21)
[        ] E/flutter (29516): #33     _startMicrotaskLoop (dart:async/schedule_microtask.dart:49:5)

I have installed it in pubscpec.yaml
es_compression: ^2.0.2

$ flutter doctor
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel beta, 2.1.0-12.2.pre, on Linux, locale en_US.UTF-8)
[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
[✓] Chrome - develop for the web
[✓] Android Studio (version 4.1)
[✓] Android Studio
[✓] Connected device (2 available)

• No issues found!

The streaming buffers version that we have right now is pretty performant in Dart.
However, if you have all the bytes up front...then it certainly could have the potential to be even faster (at the expense of larger in-memory buffers).

The VAST Platform implements an optimized version by extensions on ByteArray/String like lz4Compress/lz4Decompress.
In a 455MB example (linux-3.3.tar), the lz4Compress was 1.2 seconds in-memory and the streaming was 2.2 seconds.

I believe we can do something similar here whenever encode()/decode() is called.

How can I use gzip decode in flutter?

websocket connected success, and received data, snapshot.data like [31, 42, 42,12.......] data is compressed by zstd, how to decompress it?

   channel = new IOWebSocketChannel.connect('ws://$SOCKET_URL?token=$userToken');

   // socket connected success

   StreamBuilder(
      stream: channel.stream,
      builder: (context, snapshot) {
      
      print(snapshot.data);  // recieved data [31, 42, 42,12.......]

      final result = compress(snapshot.data);  // here how to compress

      return new Padding(
        child: new Text(''),
      );
    },
  )

Invalid argument(s): Failed to lookup symbol 'BrotliDecoderCreateInstance': dlsym(RTLD_DEFAULT, BrotliDecoderCreateInstance): symbol not found

this library does not support iOS operations. An error was reported. I don't know if Android is the same.
so，What should I do to solve this problem? Do you have a plan?
It looks great and should support more platforms.

my code:

final codec = Lz4Codec();
points = codec.decode(data);

and throws exception: Unsupported operation: No Decoder Implementation

Hi,

I have used the following dart code to use es_compression .

final codec = BrotliCodec();
bytesStream = asStream(bytes);
bytesStream
.transform(codec.encoder)
.transform(codec.decoder)
.fold<List>([], (buffer, data) {
buffer.addAll(data);
return buffer;
}).then((decoded) {
assert(bytes == decoded)
});

but when i build project it gives following error.
Failed to load dynamic library 'libeslz4.android32.so': dlopen failed : library "libeslz4.android32.so" not found.

Please suggest any solution to load ".so" library using android emulator.

So far I've just been using the prebuilt lz4 dll we ship with the VAST Platform (VA Smalltalk).
However, as I was looking at how to prepare for pub.dev distribution, I wondered how this would work.
I came across the following:
dart-lang/sdk#36712

So since this is designed to be a generic dart library, I guess the options are to just

1. Provide build instructions and put the responsibility on the user
2. Distribute the binaries and internally do the DynamicLibrary.open on them

I've got pretty nice CMakeScripts from the VAST Platform build I can distribute, that's fine by me.
But I would like to have users just to be able to run this out of the box on their platform and not have to mess around with CMake builds / c compilers and so on.

Therefore, I will do as they suggest (until a better solution comes along) and model what was done in tflite_native and have a blobs directory for each codec that contains prebuilt binaries.

I will also provide the CMake scripts used to build them in the tool directory along with a dart script to perform the builds if someone wants to do that.

Command:
$ escompress -i"input.txt" -o"output.lz4"

Error message:

type 'Null' is not a subtype of type 'String' in type cast
#0      main (file:///home/xxx/.pub-cache/hosted/pub.dev/es_compression-2.0.9/bin/es_compress.dart:57:40)
#1      _delayEntrypointInvocation.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:295:33)
#2      _RawReceivePort._handleMessage (dart:isolate-patch/isolate_patch.dart:192:26)

Env:
Tested on both Windows 10 and Ubuntu
Dart SDK version: 2.19.2 (stable) (Unknown timestamp) on "linux_x64"
es_compression-2.0.9

We will be distributing the shared libs so shouldn't be a problem.
But would like to have something flexible enough to handle unsupported platforms and custom locations.

This means exposing custom hooks that the user can control to force where and how a library is loaded

Please forgive my ignorance. Does the es_compression:lz4 library support compress/decompress block format?

use zstd encryption then program output "unsupported platform"

i have pub get this package. i want to compress list/bytes data
List<int> compressedBytes = brotli.encode(bytes);
i got below error
I/ViewRootImpl@f09b504MainActivity: MSG_WINDOW_FOCUS_CHANGED 1 0
I/ViewRootImpl@f09b504MainActivity: mThreadedRenderer.initializeIfNeeded()#2 mSurface={isValid=true 0xb400007a36422000}
D/InputMethodManager(10294): startInputInner - Id : 0
I/InputMethodManager(10294): startInputInner - mService.startInputOrWindowGainedFocus
I/ViewRootImpl@f09b504MainActivity: ViewPostIme pointer 0
I/ViewRootImpl@f09b504MainActivity: ViewPostIme pointer 1
E/flutter (10294): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Invalid argument(s): Failed to load dynamic library 'libesbrotli-android64.so': dlopen failed: library "libesbrotli-android64.so" not found
E/flutter (10294): #0 open (dart:ffi-patch/ffi_dynamic_library_patch.dart:11:43)
E/flutter (10294): #1 new DynamicLibrary.open (dart:ffi-patch/ffi_dynamic_library_patch.dart:22:12)
E/flutter (10294): #2 OpenLibraryStrategy.open (package:es_compression/src/framework/native/library/open_library_strategy.dart:38:28)
E/flutter (10294): #3 OpenViaOsResolutionStrategy.openFor (package:es_compression/src/framework/native/library/os_resolution_strategy.dart:26:12)
E/flutter (10294): #4 OpenLibrary.openLibrary (package:es_compression/src/framework/native/library/open_library.dart:44:32)
E/flutter (10294): #5 new BrotliLibrary. (package:es_compression/src/brotli/ffi/library.dart:56:20)
E/flutter (10294): #6 BrotliLibrary._instance (package:es_compression/src/brotli/ffi/library.dart:36:21)
E/flutter (10294): #7 BrotliLibrary._instance (package:es_compression/src/brotli/ffi/library.dart)
E/flutter (10294): #8 new BrotliLibrary (package:es_compression/src/brotli/ffi/library.dart:51:30)
E/flutter (10294): #9 new BrotliDispatcher (package:es_compression/src/brotli/ffi/dispatcher.dart:65:34)
E/flutter (10294): #10 new BrotliCompressFilter (package:es_compression/src/brotli/ffi/compress_filter.dart:20:40)
E/flutter (10294): #11 _makeBrotliCompressFilter (package:es_compression/src/brotli/encoder.dart:178:5)
E/flutter (10294): #12 new BrotliEncoderSink. (package:es_compression/src/brotli/encoder.dart:147:13)
E/flutter (10294): #13 BrotliEncoder.startChunkedConversion (package:es_compression/src/brotli/encoder.dart:114:31)
E/flutter (10294): #14 CodecConverter.convert (package:es_compression/src/framework/converters.dart:20:7)
E/flutter (10294): #15 Codec.encode (dart:convert/codec.dart:21:32)
E/flutter (10294): #16 MergePDFFileController.CompressPDf (package:choicepdfeditor/controllers/MergePDFController.dart:120:40)
E/flutter (10294):

I need to understand this a bit more.
I see mention of this in the tflite-native project.
I've also read about FFI characteristics with regards to isolates and since FFI is still technically Beta...things may change here.

Right now just need more information and use cases

Finish implementing the escompress program which will be built from the es_compress.dart program in bin

I'm trying to use this package in windows flutter app and i get this error

[ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Invalid argument(s): Failed to load dynamic library 'eslz4-win64.dll': The specified module could not be found.

code from flutter

    final codec = Lz4Codec();

    print('Lz4 Encoder Version: ${codec.libraryVersion}');

    // One-shot encode/decode
    final encoded = codec.encode(randomBytes);
    var decoded = codec.decode(encoded);
    final oneShotResult =
        verifyEquality(randomBytes, decoded, header: 'One-shot: ');

    // Streaming encode/decode
    // Split the random bytes into 10 buckets
    final chunks = splitIntoChunks(randomBytes, 10);
    final randomStream = Stream.fromIterable(chunks);
    decoded = await randomStream
        .transform(codec.encoder)
        .transform(codec.decoder)
        .fold<List<int>>(<int>[], (buffer, data) {
      buffer.addAll(data);
      return buffer;
    });
    final streamResult =
        verifyEquality(randomBytes, decoded, header: 'Streaming: ');
    return (oneShotResult == true && streamResult == true) ? 0 : -1;
  }```

Hello, I have already the packed so the file
But I don't know where to put the so file, cause the program can't read to him, and ask you for help, thank you

Right now the input/output buffer sizes are a sort of 'educated guess' based on what we did in VAST.
I think building some options into the benchmarks where we can look at various timings as we independently adjust input and output buffers (on a per algo basis) would be beneficial.

I've reviewed other bindings and concluded what we have already is not too bad, but I've played with some different sizes and seen some decent gains. Its just a question of desired memory usage in some cases.

Invalid argument(s): Failed to load dynamic library (dlopen failed: library "eslz4-android64.so" not found)
NoSuchMethodError: The getter 'lz4VersionNumber' was called on null.

I tried adding your library in flutter but when the app is run the app closes immediately, I've tried making the package dart:cli / changing some of the other code which I don't think is useful, but the result is still the same, Can you make your library work in flutter app? I'm trying to make a realtime chat app with lz4 because it's faster

In some of the dispatchers, we allocate a few bytes of memory for use with compression calls so we don't have to keep allocating/freeing tiny bits of memory on each call.
Technically speaking, these don't get free'd. Realistically, the dispatcher is a singleton and these cached pointers absorb may 16 or so bytes. The impact is minimal.
But, if there is a cleaner mechanism around like Arena, I'm glad to use it.

Can you decompress? IOS

Have done several sweeps, but code was continuing to change.
This should be done once everything is in place for public release

Right now we have the encode()/decode() one-shot examples.
We also need codec examples using transform()

Hello, I am developing a Flutter app and I just want to use lz4 to decompress a large amount of data, that's all.
I have checked the issues in the repository and would like to ask you again if anyone has implemented integration with iOS or if there are any examples that can be provided as a reference.

As far as I can see, there are already examples of integration with Android.

Can it zip with password?

Hello, I'm trying to use the lz4 package in an iOS app I'm developing using flutter. I see that you have the blob builder tool to build the dynamic libraries. I can generate the eslz4-mac64.dylib library, but how do I generate a eslz4-ios64.dylib?

Then just to clarify, I can do the following in my flutter app:

import 'dart:convert';
import 'package:es_compression/lz4.dart';

final lz4codec = Lz4Codec.libraryPath = <path_to_es_compression>/eslz4-ios64.dylib

and then

print('Lz4 library version: ${lz4.libraryVersion}');

would print the lz4 version of the library? Is this how I can confirm that I've loaded the library?

If I direct the libraryPath to the mac64 library I get the following error:

Invalid argument(s): Failed to lookup symbol 'LZ4_versionNumber': dlsym(RTLD_DEFAULT, LZ4_versionNumber): symbol not found

Hey, thanks for the great package, I had the following problem when I tried this package with Android for the first time:
Invalid argument(s): Failed to load dynamic library 'eszstd-android32.so': dlopen failed: library "eszstd-android32.so" not found
adding the .so files into jniLibs folder fixed the problem when I tried with debug mode and it works perfectly
now when I made a release apk or even when i run on release mode the exact same error appears again like it can't see the files anymore
i have looked inside the apk and all the files are there inside the lib folder

please let me know I have any idea why this is happening cause I have tried everything
Thanks again

请问这个so文件从哪里获得

i got an error "ArgumentError (Invalid argument(s): Failed to load dynamic library 'libeszstd-android64.so': dlopen failed: library "libeszstd-android64.so" not found)" when try to run the encode and decode of this lib on flutter.

Required changed are available in nullsafety branch