Comments (5)
Hi,
first of all, thank you for a informative description.
As for 'configure the decompression so it uses less memory' - yes, it's possible to set the minimum size of memory buffers for IO before decoder creation.
Example:
PLzmaSDK.decoderReadSize = 1024
PLzmaSDK.decoderWriteSize = 1024
PLzmaSDK.streamReadSize = 1024
PLzmaSDK.streamWriteSize = 1024
Also, you can disable thread-safety by placing #define LIBPLZMA_THREAD_UNSAFE 1
at the top of libplzma.h
file or globaly for POD.
But, all of this will not fully help you. Cause the biggest ammount of memory is archive's dictionary. And to reduce the size of dictionary, you need to create/use archive with compression level <= 3.
Example: 7za a -mx3 test.7z install-arm64-minimal-20230910T213208Z.iso
Please refer to: https://documentation.help/7-Zip/method.htm 'x=[0 | 1 | 5 | 7 | 9 ] Sets level of compression of 7z. There you'll find dictionary sizes.
from plzmasdk.
Thanks for detailed context @OlehKulykov 🙏 Just so you know, I have very limited knowledge about compression, so some of my questions may be off.
As for 'configure the decompression so it uses less memory' - yes, it's possible to set the minimum size of memory buffers for IO before decoder creation.
Is there any calculation I can use to understand what is contributing to the memory consumption? Example:
- For an archive of size X, using dictionary size of Y, the memory consumption during decompression can be estimated as
<some formula> = Z MB
Also, you can disable thread-safety by placing #define LIBPLZMA_THREAD_UNSAFE 1 at the top of libplzma.h file or globaly for POD.
How does thread safety affect memory consumption? How are those two related?
But, all of this will not fully help you. Cause the biggest ammount of memory is archive's dictionary. And to reduce the size of dictionary, you need to create/use archive with compression level <= 3.
Example: 7za a -mx3 test.7z install-arm64-minimal-20230910T213208Z.iso
Please refer to: https://documentation.help/7-Zip/method.htm 'x=[0 | 1 | 5 | 7 | 9 ] Sets level of compression of 7z. There you'll find dictionary sizes.
Thanks! I am currently using level 9, as I need to have the smallest size possible. Will play with the compression level value (the main problem I see is that reducing the compression level results in a bigger archive)
Couple more questions if you can 🙏
- Would the usage of a file OutStream help in this case? (I already tried to set it up, but did not manage yet because I could not find any example of it. All examples of streams I could find in the repository are using memory streams, not file streams)
- For my usecase, I would like to maintain the compression level so the archive size is as small as possible. I could accept a slower decompression on exchange for less memory consumption, is there a way to configure this?
from plzmasdk.
Hi,
Q/A:
- Is there any calculation I can use to understand what is contributing to the memory consumption? Example:
- Almost no. You should take the ammount of memory for dictionary depending on selected compression level plus avarange memory for your UI code, ext.
- How does thread safety affect memory consumption? How are those two related?
- Less code -> less memory. Mutexes are class fields, fileds/mutexes & processing code could be ignored/disabled by preprocessor definition & not compiled into the app binary.
No class field -> no extra memory for the whole class. No processing code -> less size of the app binary -> less memory to run the app.
-
Yes, lower compression level -> smaller dictionary -> less memory to decompress -> bigger size of the archive, no matter which type of archive.
-
OutStream. Out'File'Stream -> write decoded/decompressed parts directly to physical file. Out'Memory'Stream -> write decoded/decompressed parts to heap memory.
'Public interface' for streams is same. The difference in a way of storing the content of stream.
P.S.: The design of 'public interface' allows even to write decoded/decompressed parts to HTTPs, FTP, etc. via client code & custom implementation. -
'I would like to maintain the compression level so the archive size is as small as possible'.
- You can Google a bit: https://www.google.com/search?q=compression+algorithm+without+dictionary
- Or try to update your task/needs to a single file inside an 'arhive', something like tar.xz or your own format based on on a single compressed data/stream (NOT an arhive with files, dirs inside).
Good luck.
from plzmasdk.
P.S.: Some info about this SDK.
The main programming language is C++, both patched original 7-zip SDK and code 'cross platform way how to use it' - this SDK.
Since C++ libraries are a huge pain ..., there is a 'C language bindings' - all functionality available in pure C via extra code, i.e. C bindings.
Reminder from README: the structure of SDK for iOS looks like:
- Main C++ implementation -> C bindings -> Swift
- Main C++ implementation -> Objective-C
In case of a very limited memory, depending on your skills(I don't know), for your specific needs, you can implement next:
- Decide what do you need. From the comments you have to decode/extract some archive items by prefix from a big archive which contains 'something interesting/needed for you' & rest.
- As I see, the app's language is Swift.
- Make a local version of Cocoapod or local package for Apple's Package Manger. It's MIT - everything is fine.
- Remove from local version of POD | APM all swift and Obj-C code. Keep in mind reference.
4.1. !!! At this point you are using the only 'libplzma.a'. My recommendation is: use static lib to avoid unused code!!!. - Disable TAR & thread-safety via preprocessor definition 'LIBPLZMA_NO_TAR=1' -> '#define LIBPLZMA_NO_TAR 1', 'LIBPLZMA_THREAD_UNSAFE=1' -> '#define LIBPLZMA_THREAD_UNSAFE 1'
5.1. Keep in mind: less code -> less memory.
5.2. You are good in C. Create your own Swift class/func for your specific needs & use directly lib's C bindings(take a look in Swift reference).
5.3. You are good in C++. Create your own Obj-C class/func(*.mm)+Bridging, disable C bindings by preprocessor definition 'LIBPLZMA_NO_C_BINDINGS=1' -> '#define LIBPLZMA_NO_C_BINDINGS 1' (take a look in Obj-C reference). - In case of issues with SDK, please create an issue here in GitHub.
6.1. In case of some specific/weird using of the SDK, please use email from public headers.
from plzmasdk.
Thank you so much for this great amount of context. It is very much enlightening, appreciate it a lot
4.1. !!! At this point you are using the only 'libplzma.a'. My recommendation is: use static lib to avoid unused code!!!.
Thanks! I am already doing this and linking PLzmaSDK statically 👍
I will try the alternatives you mentioned and write here the results when I have them 🙏
from plzmasdk.
Related Issues (20)
- add LIBPLZMA_OPT_NO_CRYPTO to disable crypto HOT 1
- how to add empty directory without real fs? HOT 4
- Why PLzmaSDK is much more slower than LzmaSDKObjC HOT 1
- Crash decoding an archive in String function (Swift 1.1.3) HOT 3
- Can't use it in android kotlin HOT 1
- Detect password protected files HOT 3
- Duplicated files HOT 1
- android ndk-build failed HOT 1
- Expect more convenience than before
- 7z extract progress error
- Error reporting using source code compilation
- Why is the decompression speed so slow?
- Missing Streaming Interface for Swift HOT 1
- File's creation and modified dates are set to decompressed date
- Support for Zstandard HOT 1
- How to use multivolume archives HOT 2
- Including the Apple Privacy manifest file
- Build issues in Catalina with Xcode 12 HOT 5
- set LIBPLZMA_NO_PROGRESS compile failed in Xcode12 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from plzmasdk.