Comments (12)
SevenZipFile.extract() calls Worker class to extract target. The core logic inside py7zr is as follows:
- walk through file lists and detect directories and symbolic links, and then memory target output filepath into worker class.
Line 723 in 98ac851
Line 763 in 98ac851
- making directories
Lines 765 to 767 in 98ac851
- calling worker extractor
Line 776 in 98ac851
- making symbolic links
Lines 778 to 785 in 98ac851
We should add a callback to step3 worker extractor.
Here is an extractor main part.
Lines 201 to 207 in 98ac851
Processes are run in spawned another process concurrently. We need to add some shared memory or pipe to communicate extract worker process and main process.
Here is a core part of worker extractor
Lines 229 to 235 in 98ac851
Here line 231 get a output file path for target compressed chunk which register in step1.
We can add a code to report progress to main process here.
from py7zr.
Yes, I think just report IsDirectory
, FileName
and PercentDone
.
from py7zr.
py7zr extract files concurrently using threading. This may cause report become such as
[(IsDirectory, FileName, DoneItems, SubTotalItems), (IsDirectory, FileName, DoneItems, SubTotalItems).... ]
For example, test data 'tests/data/mblock_1.7z' has 3 blocks, when running '7z l mblock_1.7z' we can see
Listing archive: ./mblock_1.7z
--
Path = ./mblock_1.7z
Type = 7z
Physical Size = 631690
Headers Size = 2305
Method = LZMA2:1536k BCJ
Solid = +
Blocks = 3
So py7zr will run with 4 threads, first one extract directories at first,
second one extract following files;
2015-11-18 18:04:24 .R..A 5263 94014 C/7z.h
2015-11-09 18:41:08 .R..A 1548 C/7zAlloc.c
2015-03-26 01:07:58 .R..A 403 C/7zAlloc.h
.etc...
and third one extract concurrently with second thread following files;
2014-12-03 23:35:44 ....A 5522 25204 DOC/7zC.txt
2010-09-16 21:57:16 ....A 7573 DOC/7zFormat.txt
2016-09-27 19:42:33 ....A 3031 DOC/Methods.txt
etc...
and last extract concurrently
2016-10-05 00:13:34 ....A 35328 510167 bin/7zS2.sfx
2016-10-05 00:13:32 ....A 35328 bin/7zS2con.sfx
2016-10-05 00:12:31 ....A 113152 bin/7zSD.sfx
...etc
from py7zr.
All extract thread concurrently submit maybe cause issue, I think need add synchronize lock, but maybe cause extract efficiency.
from py7zr.
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days
from py7zr.
@MiyamuraMiyako now #130 implement a primitive progress callback mechanism.
Could you try it? You can see how to use it on test_basic.py:446 test_extract_callback test case.
from py7zr.
Current API does not provide percentage, but provide filename and extracted bytes.
User can get list of filename, its property and file size from list() method, then API user can calculate a percentage, display whether it is a directory.
from py7zr.
I tested this version, It can achieve I need.
from py7zr.
Now you can find a CLI option --verbose
at #130 which produce such as
$ python -m py7zr x --verbose tests/data/test_1.7z
- scripts (0%)
- scripts/py7zr (15%)
- setup.cfg (23%)
- setup.py (100%)
You can find a sample code in py7zr/cli.py
class CliExtractCallback(py7zr.callbacks.ExtractCallback):
def __init__(self, total_bytes, ofd=sys.stdout):
self.ofd = ofd
self.archive_total = total_bytes
self.total_bytes = 0
self.columns, _ = shutil.get_terminal_size(fallback=(80, 24))
self.pwidth = 0
def report_start(self, processing_file_path, processing_bytes):
self.ofd.write('- {}'.format(processing_file_path))
self.pwidth += len(processing_file_path) + 2
def report_end(self, processing_file_path, wrote_bytes):
self.total_bytes += int(wrote_bytes)
plest = self.columns - self.pwidth
progress = self.total_bytes / self.archive_total
msg = '({:.0%})\n'.format(progress)
if plest - len(msg) > 0:
self.ofd.write(msg.rjust(plest))
else:
self.ofd.write(msg)
self.pwidth = 0
if __name__ == "__main__":
with py7zr.SevenZipFile('target.7z', 'r') as a:
archive_info = a.archiveinfo()
cb = CliExtractCallback(total_bytes=archive_info.uncompressed)
a.extractall(callback=cb)
from py7zr.
v0.7.0b3 released with callback.
from py7zr.
hello
please help me add file to archive, and printing callback in percents
from py7zr.
@gmankab please raise another ticket to request callback feature for archive/write functions.
from py7zr.
Related Issues (20)
- Please consider removing 0.9.5 -> 0.11.2 packages from pypi.org (malware test cases) HOT 3
- IndexError / Worker initialization sets wrong last_file_index HOT 2
- Code cannot handle Error which are presented as "There are some data after the end of the payload data" in 7zip HOT 4
- Some test cases use deprecated functions
- Unable to create 7z/zip archive. HOT 3
- AttributeError when adding empty string with writestr() HOT 2
- Python "zipfile" compatibility HOT 2
- Is there an option to split to volumes? HOT 1
- Memory leaks when uncompressing multi-volume archives HOT 2
- File extraction fails when give filename str as a `targets` argument in v0.21.0 HOT 8
- read/readall dumps the decompressed files to memory, instead of streaming them HOT 2
- Extraction is not completing and dosen't give any error
- Add support for `mode="x"` in `SevenZipFile()` constructor
- py7zr fails to extract a specific encrypted archive
- bug
- Python 3.7 is not supported in version 0.21.0 HOT 1
- Extracting a corrupt 7z file and try to delete it will cause the PermissionError in windows
- Give version number to be 1.0.0 and change status to be production HOT 1
- `mode="a"` errors if the file doesn't already exist
- Add `getinfo()` method to get a single `FileInfo` by name HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from py7zr.