Extract archive callback. about py7zr HOT 12 CLOSED

SevenZipFile.extract() calls Worker class to extract target. The core logic inside py7zr is as follows:

1. walk through file lists and detect directories and symbolic links, and then memory target output filepath into worker class.

2. making directories

3. calling worker extractor

4. making symbolic links

We should add a callback to step3 worker extractor.

Here is an extractor main part.

Processes are run in spawned another process concurrently. We need to add some shared memory or pipe to communicate extract worker process and main process.

Here is a core part of worker extractor

Here line 231 get a output file path for target compressed chunk which register in step1.

We can add a code to report progress to main process here.

from py7zr.

Yes, I think just report IsDirectory, FileName and PercentDone.

from py7zr.

py7zr extract files concurrently using threading. This may cause report become such as
[(IsDirectory, FileName, DoneItems, SubTotalItems), (IsDirectory, FileName, DoneItems, SubTotalItems).... ]

For example, test data 'tests/data/mblock_1.7z' has 3 blocks, when running '7z l mblock_1.7z' we can see

Listing archive: ./mblock_1.7z

--
Path = ./mblock_1.7z
Type = 7z
Physical Size = 631690
Headers Size = 2305
Method = LZMA2:1536k BCJ
Solid = +
Blocks = 3

So py7zr will run with 4 threads, first one extract directories at first,
second one extract following files;

2015-11-18 18:04:24 .R..A         5263        94014  C/7z.h
2015-11-09 18:41:08 .R..A         1548               C/7zAlloc.c
2015-03-26 01:07:58 .R..A          403               C/7zAlloc.h
.etc...

and third one extract concurrently with second thread following files;

2014-12-03 23:35:44 ....A         5522        25204  DOC/7zC.txt
2010-09-16 21:57:16 ....A         7573               DOC/7zFormat.txt
2016-09-27 19:42:33 ....A         3031               DOC/Methods.txt
etc...

and last extract concurrently

2016-10-05 00:13:34 ....A        35328       510167  bin/7zS2.sfx
2016-10-05 00:13:32 ....A        35328               bin/7zS2con.sfx
2016-10-05 00:12:31 ....A       113152               bin/7zSD.sfx
...etc

from py7zr.

All extract thread concurrently submit maybe cause issue, I think need add synchronize lock, but maybe cause extract efficiency.

from py7zr.

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

from py7zr.

@MiyamuraMiyako now #130 implement a primitive progress callback mechanism.
Could you try it? You can see how to use it on test_basic.py:446 test_extract_callback test case.

from py7zr.

Current API does not provide percentage, but provide filename and extracted bytes.
User can get list of filename, its property and file size from list() method, then API user can calculate a percentage, display whether it is a directory.

from py7zr.

I tested this version, It can achieve I need.

from py7zr.

Now you can find a CLI option --verbose at #130 which produce such as

$ python -m py7zr x --verbose tests/data/test_1.7z 
- scripts                                                                  (0%)
- scripts/py7zr                                                           (15%)
- setup.cfg                                                               (23%)
- setup.py                                                               (100%)

You can find a sample code in py7zr/cli.py

class CliExtractCallback(py7zr.callbacks.ExtractCallback):

    def __init__(self, total_bytes, ofd=sys.stdout):
        self.ofd = ofd
        self.archive_total = total_bytes
        self.total_bytes = 0
        self.columns, _ = shutil.get_terminal_size(fallback=(80, 24))
        self.pwidth = 0

    def report_start(self, processing_file_path, processing_bytes):
        self.ofd.write('- {}'.format(processing_file_path))
        self.pwidth += len(processing_file_path) + 2

    def report_end(self, processing_file_path, wrote_bytes):
        self.total_bytes += int(wrote_bytes)
        plest = self.columns - self.pwidth
        progress = self.total_bytes / self.archive_total
        msg = '({:.0%})\n'.format(progress)
        if plest - len(msg) > 0:
            self.ofd.write(msg.rjust(plest))
        else:
            self.ofd.write(msg)
        self.pwidth = 0


if __name__ == "__main__":
    with py7zr.SevenZipFile('target.7z', 'r') as a:
        archive_info = a.archiveinfo()
        cb = CliExtractCallback(total_bytes=archive_info.uncompressed)
        a.extractall(callback=cb)

from py7zr.

v0.7.0b3 released with callback.

from py7zr.

hello

please help me add file to archive, and printing callback in percents

from py7zr.

@gmankab please raise another ticket to request callback feature for archive/write functions.

from py7zr.