pmqs / io-compress Goto Github PK
View Code? Open in Web Editor NEWIO-Compress - Perl5 module to read/write compressed data in multiple formats
io-compress's People
Contributors
Stargazers
Watchers
Forkers
mstock fabiensanglard clayne atoomic fastmailops sigurthor annamorphism surfndez khwilliamson mbeijen jackdeguest demerphq haarg rwp0 ranwitterio-compress's Issues
011-streamzip.t fails on Strawberry Perl
http://www.cpantesters.org/cpan/report/a49dc938-6e61-1014-8b4a-600c032c038b
t/010examples-zlib.t ............ ok
# Failed test (t/011-streamzip.t at line 72)
# got: '65280'
# expected: '0'
# Failed test (t/011-streamzip.t at line 74)
# got: 'Undefined subroutine &IO::Compress::Adapter::Lzma::mkRawZipCompObject called at C:\home\tennis\.cpanm\work\1649587452.7800\IO-Compress-2.105\blib\lib/IO/Compress/Zip.pm line 158.
# '
# expected: ''
# Command line: "C:\Strawberry\perl\bin\perl.exe" "-MExtUtils::testlib" -w ./bin//streamzip -method lzma <tst1876X00025.tmp >tst1876X00026.tmp 2>tst1876X00027.tmp
# Test called from t/011-streamzip.t, line 152
# Failed test (t/011-streamzip.t at line 156)
# got: ''
# expected: 'hello
# this is
# a test
# message
# x ttttt
# xuuuuuu
# the end
# '
# Failed test (t/011-streamzip.t at line 72)
# got: '512'
# expected: '0'
# Failed test (t/011-streamzip.t at line 74)
# got: 'Undefined subroutine &IO::Compress::Adapter::Lzma::mkRawZipCompObject called at C:\home\tennis\.cpanm\work\1649587452.7800\IO-Compress-2.105\blib\lib/IO/Compress/Zip.pm line 158.
# '
# expected: ''
# Command line: "C:\Strawberry\perl\bin\perl.exe" "-MExtUtils::testlib" -w ./bin//streamzip -zipfile tst1876X00029.tmp -method lzma <tst1876X00028.tmp 2>tst1876X00030.tmp
# Test called from t/011-streamzip.t, line 166
# Failed test (t/011-streamzip.t at line 170)
# got: undef
# expected: 'hello
# this is
# a test
# message
# x ttttt
# xuuuuuu
# the end
# '
# Looks like you failed 6 tests of 137.
t/011-streamzip.t ...............
Dubious, test returned 6 (wstat 1536, 0x600)
Failed 6/137 subtests
(less 8 skipped subtests: 123 okay)
cz-14gzopen.t fails on AFS filesystem
Test fails for perl-5.36.3 and perl-5.38.2 (no other versions tested) when compiling in an AFS-part of the filesystem.
Compiling in e.g. /var/tmp works fine, though. ./Configure detects that the sources reside in AFS: AFS may be running... I'll be extra cautious then...
The errors detected are:
cpan/IO-Compress/t/cz-14gzopen ................................... # Failed test ' gzopen returns undef'
# at t/cz-14gzopen.t line 506.
# Failed test ' gzerrno ok'
# at t/cz-14gzopen.t line 507.
# gzerrno 0
# Failed test ' gzopen returns undef'
# at t/cz-14gzopen.t line 528.
# Failed test ' gzerrno ok'
# at t/cz-14gzopen.t line 529.
# Looks like you failed 4 tests of 264.
FAILED at test 182
They correspond directly to tests using chmod (line numbers at the beginning):
488 title 'read/write a non-readable/writable file';
...
497 chmod 0444, $name
...
506 ok !$fil, " gzopen returns undef" ;
507 ok $gzerrno, " gzerrno ok" or
...
520 chmod 0222, $name ;
...
528 ok !$fil, " gzopen returns undef" ;
529 ok $gzerrno, " gzerrno ok";
The tests must fail as there is no such thing as file-based ACLs in AFS; see https://docs.openafs.org/UserGuide/HDRWQ46.html
I assume that these tests should be skipped when compiled in an AFS-directory as a workaround.
Just for completeness:
% pwd
/afs/CELL/user/X/XXXXX
% /usr/bin/fs version
auristor 2021.05
Platform:
osname=linux
osvers=5.14.0-362.13.1.el9_3.x86_64
archname=x86_64-linux
uname='linux SOME-HOST.XXX.XXX 5.14.0-362.13.1.el9_3.x86_64 #1 smp preempt_dynamic thu dec 21 07:12:43 est 2023 x86_64 x86_64 x86_64 gnulinux '
config_args='-Dafs=true'
hint=recommended
useposix=true
d_sigaction=define
useithreads=undef
usemultiplicity=undef
use64bitint=define
use64bitall=define
uselongdouble=undef
usemymalloc=n
default_inc_excludes_dot=define
Compiler:
cc='cc'
ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'
optimize='-O2'
cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
ccversion=''
gccversion='11.4.1 20230605 (Red Hat 11.4.1-2)'
gccosandvers=''
intsize=4
longsize=8
ptrsize=8
doublesize=8
byteorder=12345678
doublekind=3
d_longlong=define
longlongsize=8
d_longdbl=define
longdblsize=16
longdblkind=3
ivtype='long'
ivsize=8
nvtype='double'
nvsize=8
Off_t='off_t'
lseeksize=8
alignbytes=8
prototype=
Linker and Libraries:
ld='cc'
ldflags =' -fstack-protector-strong -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib /usr/lib64 /usr/local/lib64
libs=-lpthread -ldl -lm -lcrypt -lutil -lc
perllibs=-lpthread -ldl -lm -lcrypt -lutil -lc
libc=/lib/../lib64/libc.so.6
so=so
useshrplib=false
libperl=libperl.a
gnulibc_version='2.34'
Dynamic Linking:
dlsrc=dl_dlopen.xs
dlext=so
d_dlsymun=undef
ccdlflags='-Wl,-E'
cccdlflags='-fPIC'
lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'
IO-Compress and several other dists uploaded under Perl6 directory
The latest release of this dist, as well as several related compression dists, have been uploaded under the Perl6 subdirectory. This directory is ignored by a number of tools in the ecosystem. PAUSE is also supposed to be ignoring the Perl6 directory, but there appears to be a bug, and the releases were successfully indexed.
The dists should be re-uploaded to somewhere outside the Perl6 directory.
Problematic uploads:
$ curl http://cpan.mirror.local/CPAN/modules/02packages.details.txt | grep 'Perl6/'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
Bundle::Compress::Zlib 2.094 P/PM/PMQS/Perl6/Bundle-Compress-Zlib-2.094.tar.gz
Bundle::IO::Compress::Bzip2 2.094 P/PM/PMQS/Perl6/Bundle-IO-Compress-Bzip2-2.094.tar.gz
Compress::Zlib 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
File::GlobMapper 1.001 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Compress::Adapter::Bzip2 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Compress::Adapter::Deflate 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Compress::Adapter::Identity 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Compress::Adapter::Lzf 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzf-2.094.tar.gz
IO::Compress::Adapter::Lzip 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzma-2.094.tar.gz
IO::Compress::Adapter::Lzma 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzma-2.094.tar.gz
IO::Compress::Adapter::LZO 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzop-2.094.tar.gz
IO::Compress::Adapter::Xz 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzma-2.094.tar.gz
IO::Compress::Base 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Compress::Base::Common 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Compress::Bzip2 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Compress::Deflate 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Compress::Gzip 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Compress::Gzip::Constants 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Compress::Lzf 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzf-2.094.tar.gz
IO::Compress::Lzip 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzma-2.094.tar.gz
IO::Compress::Lzma 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzma-2.094.tar.gz
IO::Compress::Lzop 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzop-2.094.tar.gz
IO::Compress::Lzop::Constants 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzop-2.094.tar.gz
IO::Compress::RawDeflate 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Compress::Xz 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzma-2.094.tar.gz
IO::Compress::Zip 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Compress::Zip::Constants 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Compress::Zlib::Constants 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Compress::Zlib::Extra 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Uncompress::Adapter::Bunzip2 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Uncompress::Adapter::Identity 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Uncompress::Adapter::Inflate 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Uncompress::Adapter::Lzf 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzf-2.094.tar.gz
IO::Uncompress::Adapter::LZO 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzop-2.094.tar.gz
IO::Uncompress::Adapter::UnLzip 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzma-2.094.tar.gz
IO::Uncompress::Adapter::UnLzma 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzma-2.094.tar.gz
IO::Uncompress::Adapter::UnXz 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzma-2.094.tar.gz
IO::Uncompress::AnyInflate 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Uncompress::AnyUncompress 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Uncompress::Base 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Uncompress::Bunzip2 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Uncompress::Gunzip 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Uncompress::Inflate 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Uncompress::RawInflate 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
IO::Uncompress::UnLzf 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzf-2.094.tar.gz
IO::Uncompress::UnLzip 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzma-2.094.tar.gz
IO::Uncompress::UnLzma 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzma-2.094.tar.gz
IO::Uncompress::UnLzop 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzop-2.094.tar.gz
IO::Uncompress::UnXz 2.094 P/PM/PMQS/Perl6/IO-Compress-Lzma-2.094.tar.gz
IO::Uncompress::Unzip 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
U64 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
Zlib::OldDeflate 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
Zlib::OldInflate 2.094 P/PM/PMQS/Perl6/IO-Compress-2.094.tar.gz
Streaming issue
When writing in streaming mode, force a Zip64 extra field in the local header if the file size >0xFFFFFFFF
Summary of issue
my $size = 0x100000000 ;
my $zipfile = "large-streamed.zip";
my $zip = new IO::Compress::Zip $zipfile,
Name => "one",
Stream => 1,
Minimal => 1,
Sparse => $size,
Method => ZIP_CM_STORE,
Zip64 => 1,
;
# File size now >0xFFFFFFFF, but the next entry doesn't enable Zip64
$zip->newStream(Name => "two", Zip64 => 0, Sparse => 0, Stream => 1, Method => 8);
print $zip "12345";
$zip->close();
This results in a file that unzip doesn't like
$ unzip -t large-streamed.zip
Archive: large-streamed.zip
testing: one OK
testing: two OK
error: invalid zip file with overlapped components (possible zip bomb)
$ zipdetails large-streamed.zip
000000000 LOCAL HEADER #1 04034B50
000000004 Extract Zip Spec 2D '4.5'
000000005 Extract OS 00 'MS-DOS'
000000006 General Purpose Flag 0008
[Bit 3] 1 'Streamed'
000000008 Compression Method 0000 'Stored'
00000000A Last Mod Time 527D4876 'Mon Mar 29 09:03:44 2021'
00000000E CRC 00000000
000000012 Compressed Length FFFFFFFF
000000016 Uncompressed Length FFFFFFFF
00000001A Filename Length 0003
00000001C Extra Length 0014
00000001E Filename 'one'
000000021 Extra ID #0001 0001 'ZIP64'
000000023 Length 0010
000000025 Uncompressed Size 0000000000000000
00000002D Compressed Size 0000000000000000
000000035 PAYLOAD
100000035 STREAMING DATA HEADER 08074B50
100000039 CRC D202EF8D
10000003D Compressed Length 0000000100000000
100000045 Uncompressed Length 0000000100000000
10000004D LOCAL HEADER #2 04034B50
100000051 Extract Zip Spec 14 '2.0'
100000052 Extract OS 00 'MS-DOS'
100000053 General Purpose Flag 0008
[Bit 3] 1 'Streamed'
100000055 Compression Method 0000 'Stored'
100000057 Last Mod Time 527D487B 'Mon Mar 29 09:03:54 2021'
10000005B CRC 00000000
10000005F Compressed Length 00000000
100000063 Uncompressed Length 00000000
100000067 Filename Length 0003
100000069 Extra Length 0000
10000006B Filename 'two'
10000006E PAYLOAD 12345
100000073 STREAMING DATA HEADER 08074B50
100000077 CRC CBF53A1C
10000007B Compressed Length 00000005
10000007F Uncompressed Length 00000005
100000083 CENTRAL HEADER #1 02014B50
100000087 Created Zip Spec 2D '4.5'
100000088 Created OS 03 'Unix'
100000089 Extract Zip Spec 2D '4.5'
10000008A Extract OS 00 'MS-DOS'
10000008B General Purpose Flag 0008
[Bit 3] 1 'Streamed'
10000008D Compression Method 0000 'Stored'
10000008F Last Mod Time 527D4876 'Mon Mar 29 09:03:44 2021'
100000093 CRC D202EF8D
100000097 Compressed Length FFFFFFFF
10000009B Uncompressed Length FFFFFFFF
10000009F Filename Length 0003
1000000A1 Extra Length 0014
1000000A3 Comment Length 0000
1000000A5 Disk Start 0000
1000000A7 Int File Attributes 0000
[Bit 0] 0 'Binary Data'
1000000A9 Ext File Attributes 81A40000
1000000AD Local Header Offset 00000000
1000000B1 Filename 'one'
1000000B4 Extra ID #0001 0001 'ZIP64'
1000000B6 Length 0010
1000000B8 Uncompressed Size 0000000100000000
1000000C0 Compressed Size 0000000100000000
1000000C8 CENTRAL HEADER #2 02014B50
1000000CC Created Zip Spec 14 '2.0'
1000000CD Created OS 03 'Unix'
1000000CE Extract Zip Spec 14 '2.0'
1000000CF Extract OS 00 'MS-DOS'
1000000D0 General Purpose Flag 0008
[Bit 3] 1 'Streamed'
1000000D2 Compression Method 0000 'Stored'
1000000D4 Last Mod Time 527D487B 'Mon Mar 29 09:03:54 2021'
1000000D8 CRC CBF53A1C
1000000DC Compressed Length 00000005
1000000E0 Uncompressed Length 00000005
1000000E4 Filename Length 0003
1000000E6 Extra Length 000C
1000000E8 Comment Length 0000
1000000EA Disk Start 0000
1000000EC Int File Attributes 0000
[Bit 0] 0 'Binary Data'
1000000EE Ext File Attributes 81A40000
1000000F2 Local Header Offset FFFFFFFF
1000000F6 Filename 'two'
1000000F9 Extra ID #0001 0001 'ZIP64'
1000000FB Length 0008
1000000FD Offset to Local Dir 000000010000004D
100000105 ZIP64 END CENTRAL DIR 06064B50
RECORD
100000109 Size of record 000000000000002C
100000111 Created Zip Spec 14 '2.0'
100000112 Created OS 03 'Unix'
100000113 Extract Zip Spec 14 '2.0'
100000114 Extract OS 00 'MS-DOS'
100000115 Number of this disk 00000000
100000119 Central Dir Disk no 00000000
10000011D Entries in this disk 0000000000000002
100000125 Total Entries 0000000000000002
10000012D Size of Central Dir 0000000000000082
100000135 Offset to Central dir 0000000100000083
10000013D ZIP64 END CENTRAL DIR 07064B50
LOCATOR
100000141 Central Dir Disk no 00000000
100000145 Offset to Central dir 0000000100000105
10000014D Total no of Disks 00000001
100000151 END CENTRAL HEADER 06054B50
100000155 Number of this disk 0000
100000157 Central Dir Disk no 0000
100000159 Entries in this disk 0002
10000015B Total Entries 0002
10000015D Size of Central Dir 00000082
100000161 Offset to Central Dir FFFFFFFF
100000165 Comment Length 0000
Done
Windows failure in t/105oneshot-zip-only.t
ODS streaming issue fails
https://github.com/pmqs/IO-Compress/runs/507927871?check_suite_focus=true
# Failed test (t/105oneshot-zip-only.t at line 320)
# got: '<?xml version="1.0" encoding="UTF-8"?>
# <office:document-meta xmlns:office="urn:oasis:names:tc:opendocument:xmlns:office:1.0" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:meta="urn:oasis:names:tc:opendocument:xmlns:meta:1.0" xmlns:ooo="http://openoffice.org/2004/office" xmlns:grddl="http://www.w3.org/2003/g/data-view#" office:version="1.2"><office:meta><meta:creation-date>2018-12-25T11:36:11.437260543</meta:creation-date><dc:date>2018-12-25T11:36:55.657945697</dc:date><meta:editing-duration>PT54S</meta:editing-duration><meta:editing-cycles>1</meta:editing-cycles><meta:document-statistic meta:table-count="1" meta:cell-count="3" meta:object-count="0"/><meta:generator>LibreOffice/6.0.7.3$Linux_X86_64 LibreOffice_project/00m0$Build-3</meta:generator></office:meta></office:document-meta>'
# expected: '<?xml version="1.0" encoding="UTF-8"?>
# <office:document-meta xmlns:office="urn:oasis:names:tc:opendocument:xmlns:office:1.0" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:meta="urn:oasis:names:tc:opendocument:xmlns:meta:1.0" xmlns:ooo="http://openoffice.org/2004/office" xmlns:grddl="http://www.w3.org/2003/g/data-view#" office:version="1.2"><office:meta><meta:creation-date>2018-12-25T11:36:11.437260543</meta:creation-date><dc:date>2018-12-25T11:36:55.657945697</dc:date><meta:editing-duration>PT54S</meta:editing-duration><meta:editing-cycles>1</meta:editing-cycles><meta:document-statistic meta:table-count="1" meta:cell-count="3" meta:object-count="0"/><meta:generator>LibreOffice/6.0.7.3$Linux_X86_64 LibreOffice_project/00m0$Build-3</meta:generator></office:meta></office:document-meta>'
# Looks like you failed 1 test of 230.
t/105oneshot-zip-only.t .........
Tests hanging on Windows Quadmath Perl build
Issue report copied from Perl Monks by @sisyphus
Not sure that anyone here is particularly interested in this, but I'll mention it anyway that, as of the release (on 20.01.2021) of perl-5.33.6, it's possible to build perl on MS Windows with an NV type of __float128.
And it has become a little easier in 5.33.7 with a change to GNUmakefile that removed the need for a somewhat cumbersome configuration argument.
With the release of perl-5.22.0, the NV type of long double was enabled on Windows, and now we finally get the additional option of having the __float128 NV type.
All of this does, of course, rely on using mingw-w64 ports of gcc to build your Windows perl.I think that many of us, accustomed to simply using Strawberry Perl, don't realize just how easy it is to build perl from source on Windows.
Strawberry Perl, itself, does provide us with the toolchain that's capable of building any recent version of perl (including blead releases) from source.
The latest blead source can be obtained with:
$ git clone https://github.com/sisyphus/perl5.git blead_cloneor the perl-5.33.7 development tarball is available from:
https://cpan.metacpan.org/authors/id/R/RE/RENEEB/perl-5.33.7.tar.gzThen, I place my C:_64\strawberry-5.32.0\c\bin folder at the beginning of my PATH, and I have all that I need to build perl.
Simply cd to the perl source's win32 folder and run:
gmake INST_TOP=C:\bleadstraw CCHOME=C:\_64\strawberry-5.32.0\c CCTYPE=GCC I_QUADMATH=define USE_QUADMATH=define testThen, to install perl into the location specified by INST_TOP, just run:
gmake INST_TOP=C:\bleadstraw CCHOME=C:\_64\strawberry-5.32.0\c CCTYPE=GCC I_QUADMATH=define USE_QUADMATH=define installAnd that's it - in C:\bleadstraw I now have a Windows perl that has __float128 as its NV.
You may want to alter INST_TOP, and you probably need to alter CCHOME which should specify the full path to gcc's bin directory.
NOTE that CCHOME does not include "\bin". (Failure to get CCHOME right will cause some test failures, but will not affect the perl that has been built.)If you want to do a 'long double' build instead, you would just remove the 2 gmake arguments that include the string "QUADMATH", and insert the USE_LONG_DOUBLE=define argument.
If you don't request either a 'long double' or a 'quadmath' build, then the nvtype will be 'double'.If you're using a 32-bit compiler, you'll also need to insert the argument WIN64=undef, and you might also want to add the argument USE_64_BIT_INT=define if you want 64-bit integers and pointers.
With 64-bit compilers, perl's integer and pointer will inevitably be 64-bit.I think that covers the most commonly exercised options ... but you'll find additional options laid out in the GNUmakefile.
I don't know if it's a general issue, but (on 64-bit builds only) there's usually a couple of test scripts in cpan/IO-Compress/t that hang for me .
It's generally (but not always) the same test scripts that hang, and it has been happening for a few years. In perl-5.33.7, I'm finding that the problem has moved to a different couple of files.
The Strawberry project have never, to my knowledge, complained about such an issue - so I'm hoping that it's just something in my particular environment.
It would be nice to find out if it is "just me".
It's quite annoying - I have to kill these hangs using process explorer in order to get the test suite to run to completion. (Killing them with Ctrl-C kills the entire 'gmake test' process.)
These test scripts have always passed when run outside of the testing harness, and they've also passed even when I've run make test in the cpan/IO-Compress folder.
There's also the same thing happening in the cpan/IPC-Cmd tests with t/01_IPC-Cmd.t.
If I can find any evidence that it's also an issue for others then I'll try to get it fixed.Oh, one last thing - if you build a 32-bit perl with I_QUADMATH=define USE_QUADMATH=define but without USE_64_BIT_INT=define, then expect some failing tests in Math::BigFloat due to a Math::BigInt::Calc bug.
This issue has been reported, with a proposed simple fix to Calc.pm.Cheers,
Rob
zipdetails breaks on zip with single directory in zip64 mode
the -fz option forces zip64
$ mkdir fred
$ zip -fz fred.zip fred
adding: fred/ (stored 0%)
$ zipdetails fred.zip
zzz at /media/paul/Linux-Shared/base/perl/ext/Gzip/IO-Compress/bin/zipdetails line 1711.
LICENSE File is missing
Automated SCA Tools (such a Blackduck) and Code Repositories (such a GitHub) can not identify;
- Permissions (Commercial use, Modification, Distribution, Patent use, Private use)
- Limitations (Liability,Trademark use,Warranty)
- Conditions (License and copyright notice, State changes)
of Using this Module without including a LICENSE File in residing within the Code Basis.
'operlap' should be 'overlap'
Line 2284 in b7c4500
see: pmqs/zipdetails#6
Problems with Term::Readline on AppVevor build
Using Term::ReadLine to read prompts from user. Code is along these lines
use Term::ReadLine;
my $term = Term::ReadLine->new("");
$got = $term->readline("replace '$filename'? [y]es, [n]o, [A]ll, [N]one, [r]ename: ")
This was checked in under 1b4864a
The CI builds on Linux, MacOS & Windows (git github CI) all work fine but the AppVeyor Win32 build fails (https://ci.appveyor.com/project/pmqs/io-compress/builds/33060246)
# Locale Info
# Derived Locale: 'cp437'
# chcp output: 'Active code page: 437'
# Failed test (t/010examples-nested-unzip.t at line 122)
# got: 'Unable to get Terminal Size. The Win32 GetConsoleScreenBufferInfo call didn't work. The COLUMNS and LINES environment variables didn't work. at C:/strawberry/perl/vendor/lib/Term/ReadLine/readline.pm line 410.
# Cannot create second readline interface, falling back to dumb.
# Cannot create second readline interface, falling back to dumb.
# Cannot create second readline interface, falling back to dumb.
# Cannot create second readline interface, falling back to dumb.
# Cannot create second readline interface, falling back to dumb.
# Cannot create second readline interface, falling back to dumb.
# Cannot create second readline interface, falling back to dumb.
# Cannot create second readline interface, falling back to dumb.
# '
# expected: ''
# Command line: "C:\strawberry\perl\bin\perl.exe" "-MExtUtils::testlib" -w C:/projects/io-compress/./bin/nested-unzip C:/projects/io-compress/tmpdir1652X00004.tmp/zip1.zip 2>tst1652X00021.tmp
# Test called from t/010examples-nested-unzip.t, line 143
# Test called from t/010examples-nested-unzip.t, line 551
# Failed test (t/010examples-nested-unzip.t at line 122)
2.096: test suite is failing
2.095 was OK.
+ /usr/bin/make -O -j48 V=1 VERBOSE=1 test COMPRESS_ZLIB_RUN_ALL=1
PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/000prereq.t ................... ok
t/001bzip2.t .................... ok
t/001zlib-generic-deflate.t ..... ok
t/001zlib-generic-gzip.t ........ ok
t/001zlib-generic-rawdeflate.t .. ok
t/001zlib-generic-zip.t ......... ok
t/002any-deflate.t .............. ok
t/002any-gzip.t ................. ok
t/002any-rawdeflate.t ........... ok
t/002any-transparent.t .......... ok
t/002any-zip.t .................. ok
t/004gziphdr.t .................. ok
t/005defhdr.t ................... ok
t/006zip.t ...................... ok
t/010examples-bzip2.t ........... ok
t/010examples-zlib.t ............ ok
t/011-streamzip.t ............... ok
t/01misc.t ...................... ok
t/020isize.t .................... ok
Cannot detect source of 't/050interop-gzip.t'! at /usr/share/perl5/vendor_perl/TAP/Parser/IteratorFactory.pm line 256.
TAP::Parser::IteratorFactory::detect_source(TAP::Parser::IteratorFactory=HASH(0x5595d6557260), TAP::Parser::Source=HASH(0x5595d6535140)) called at /usr/share/perl5/vendor_perl/TAP/Parser/IteratorFactory.pm line 211
TAP::Parser::IteratorFactory::make_iterator(TAP::Parser::IteratorFactory=HASH(0x5595d6557260), TAP::Parser::Source=HASH(0x5595d6535140)) called at /usr/share/perl5/vendor_perl/TAP/Parser.pm line 472
TAP::Parser::_initialize(TAP::Parser=HASH(0x5595d6567008), HASH(0x5595d638e9a8)) called at /usr/share/perl5/vendor_perl/TAP/Object.pm line 55
TAP::Object::new("TAP::Parser", HASH(0x5595d638e9a8)) called at /usr/share/perl5/vendor_perl/TAP/Object.pm line 130
TAP::Object::_construct(TAP::Harness=HASH(0x5595d60a4a08), "TAP::Parser", HASH(0x5595d638e9a8)) called at /usr/share/perl5/vendor_perl/TAP/Harness.pm line 852
TAP::Harness::make_parser(TAP::Harness=HASH(0x5595d60a4a08), TAP::Parser::Scheduler::Job=HASH(0x5595d6388118)) called at /usr/share/perl5/vendor_perl/TAP/Harness.pm line 651
TAP::Harness::_aggregate_single(TAP::Harness=HASH(0x5595d60a4a08), TAP::Parser::Aggregator=HASH(0x5595d60a4930), TAP::Parser::Scheduler=HASH(0x5595d6354f80)) called at /usr/share/perl5/vendor_perl/TAP/Harness.pm line 743
TAP::Harness::aggregate_tests(TAP::Harness=HASH(0x5595d60a4a08), TAP::Parser::Aggregator=HASH(0x5595d60a4930), "t/000prereq.t", "t/001bzip2.t", "t/001zlib-generic-deflate.t", "t/001zlib-generic-gzip.t", "t/001zlib-generic-rawdeflate.t", "t/001zlib-generic-zip.t", ...) called at /usr/share/perl5/vendor_perl/Test/Harness.pm line 136
Test::Harness::_aggregate_tests(TAP::Harness=HASH(0x5595d60a4a08), TAP::Parser::Aggregator=HASH(0x5595d60a4930), "t/000prereq.t", "t/001bzip2.t", "t/001zlib-generic-deflate.t", "t/001zlib-generic-gzip.t", "t/001zlib-generic-rawdeflate.t", "t/001zlib-generic-zip.t", ...) called at /usr/share/perl5/vendor_perl/Test/Harness.pm line 118
Test::Harness::_aggregate(TAP::Harness=HASH(0x5595d60a4a08), TAP::Parser::Aggregator=HASH(0x5595d60a4930), "t/000prereq.t", "t/001bzip2.t", "t/001zlib-generic-deflate.t", "t/001zlib-generic-gzip.t", "t/001zlib-generic-rawdeflate.t", "t/001zlib-generic-zip.t", ...) called at /usr/share/perl5/vendor_perl/Test/Harness.pm line 151
Test::Harness::runtests("t/000prereq.t", "t/001bzip2.t", "t/001zlib-generic-deflate.t", "t/001zlib-generic-gzip.t", "t/001zlib-generic-rawdeflate.t", "t/001zlib-generic-zip.t", "t/002any-deflate.t", "t/002any-gzip.t", ...) called at /usr/share/perl5/vendor_perl/ExtUtils/Command/MM.pm line 72
ExtUtils::Command::MM::test_harness(0, "blib/lib", "blib/arch") called at -e line 1
make: *** [Makefile:948: test_dynamic] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.Euuavl (%check)
IO::Compress::Zip::zip( '/some/where/file.txt' => '/some/where/else.zip' ) creates an archive of the input file
The documentation from IO::Compress::Zip states If the $input_filename_or_reference parameter is a simple scalar, it is assumed to be a filename. This file will be opened for reading and the input data will be read from it.
However, this is not true. if one does IO::Compress::Zip::zip( '/some/where/file.txt' => '/some/where/output.zip' ) this will actually create an archive of /some/where/file.txt rather than reading data from it as the documentation advertise.
Calling the command line utility unzip on the resulting file else.zip would result in a folder hierarchy some/where/file.txt which shows the zip file keeps a record of the original file
However, if one use IO::Uncompress::Zip::unzip to decompress output.zip, we do get back our original content.
For example:
use IO::Compress::Zip;
use IO::Uncompress::Unzip;
open( my $fh, ">", "/tmp/file.txt" ) || die( "$!" );
select((select( $fh ), $| = 1)[0]);
print( $fh "Hello world\n" ) || die( $! );
close( $fh );
my $rv = IO::Compress::Zip::zip( '/tmp/file.txt' => '/tmp/output.zip' );
die( "Error: $IO::Compress::Zip::ZipError\n" ) if( !defined( $rv ) );
print( "ok, source file compressed -> /tmp/output.zip\n" );
IO::Uncompress::Unzip::unzip( '/tmp/file.txt' => '/tmp/decompressed.txt' );
open( my $in, "<", "/tmp/decompressed.txt" ) || die( $! );
my $txt = '';
read( $in, $txt, 1024 );
close( $in );
print( "Decompressed result is: '$txt'\n" ); # "Hello world" <-- ok
A look at /tmp/output.zip using perl -MDevel::Hexdump=xd -lE 'say xd <>' /tmp/output.zip proves that:
[0000] 50 4B 03 04 14 00 08 00 08 00 CA A5 83 54 00 00 PK.. .... .... .T..
[0010] 00 00 00 00 00 00 00 00 00 00 0D 00 1C 00 2F 74 .... .... .... ../t
[0020] 6D 70 2F 66 69 6C 65 2E 74 78 74 55 54 09 00 03 mp/f ile. txtU T...
[0030] 0C 89 49 62 7D 88 49 62 75 78 0B 00 01 04 E8 03 ..Ib }.Ib ux.. ....
[0040] 00 00 04 E8 03 00 00 F3 48 CD C9 C9 57 28 CF 2F .... .... H... W(./
[0050] CA 49 E1 02 00 50 4B 07 08 D5 E0 39 B7 0E 00 00 .I.. .PK. ...9 ....
[0060] 00 0C 00 00 00 50 4B 01 02 14 03 14 00 08 00 08 .... .PK. .... ....
[0070] 00 CA A5 83 54 D5 E0 39 B7 0E 00 00 00 0C 00 00 .... T..9 .... ....
[0080] 00 0D 00 18 00 00 00 00 00 01 00 00 00 B4 81 00 .... .... .... ....
[0090] 00 00 00 2F 74 6D 70 2F 66 69 6C 65 2E 74 78 74 .../ tmp/ file .txt
[00a0] 55 54 05 00 01 0C 89 49 62 75 78 0B 00 01 04 E8 UT.. ...I bux. ....
[00b0] 03 00 00 04 E8 03 00 00 50 4B 05 06 00 00 00 00 .... .... PK.. ....
[00c0] 01 00 01 00 53 00 00 00 65 00 00 00 00 00 .... S... e... ..
The expected behaviour would have been to have the actual content of file.txt zipped and not the file also.
LibreOffice incorrectly creates entries marked as Streamed
Some programs (older versions of LibreOffice) mark entries as Streamed (bit 3 of the General Purpose Bit Flags field is set) , but still fill out the Compressed Length, Uncompressed Length & CRC32 fields in the local file header
Example below is taken from attached file testfile1.zip. (Note - This file originally had an "odt" extension. renamed to allow uploading to github)
0248 LOCAL HEADER #B 04034B50
024C Extract Zip Spec 14 '2.0'
024D Extract OS 00 'MS-DOS'
024E General Purpose Flag 0808
[Bits 1-2] 0 'Normal Compression'
[Bit 3] 1 'Streamed'
[Bit 11] 1 'Language Encoding'
0250 Compression Method 0008 'Deflated'
0252 Last Mod Time 50477166 'Fri Feb 7 14:11:12 2020'
0256 CRC F1E42324
025A Compressed Length 000000BB
025E Uncompressed Length 00000104
0262 Filename Length 000C
0264 Extra Length 0000
0266 Filename 'manifest.rdf'
APPNOTE.TXT version 6.3.6. section 4.4.4 says this
Bit 3: If this bit is set, the fields crc-32, compressed size and uncompressed size are set to zero in the local header. The correct values are put in the data descriptor immediately following the compressed data. (Note: PKZIP version 2.04g for DOS only recognizes this bit for method 8 compression, newer versions of PKZIP recognize this bit for any compression method.)
That suggests that LibreOffice is creating a non-compliant zip container.
The code in IO::Uncompress::Unzip::readHeader currently assumes that zip files do the right thing with the Streamed bit. Can work around this issue by checking for non-zero entries in the Compressed Length, Uncompressed Length & CRC32 fields
Information on zipdetails
Hi,
this distribution comes with its own version of zipdetails, which is already delivered with perl core.
I stumbled upon this while maintaining the perl modules repository for openSUSE.
The problem is that when another module depends on perl and IO::Compress (like IO::Compress::Lzma), there is a conflict between the script delivered with perl and the one from this module.
One solution could be renaming it to /usr/bin/zipdetails-IO-Compress in our spec file.
What do you think?
'bin/streamzip' wasn't added to EXE_FILES in Makefile.PL
Issue noted in 76d2795#commitcomment-35780457
No module matching dist name
This dist is named IO-Compress, but there is no matching IO::Compress module. While this is technically acceptable, it confuses various systems, including parts of perl core.
It would be more convenient to have an actual IO::Compress module. All it would need to contain would be a version number and some pod linking to the real modules contained in the dist.
Remove Term::ReadLine and charnames from dependencies
See 94d9272#commitcomment-40595745
Issue fixed in git repo, but not on CPAN.
t/111const-deflate.t tests fail with Compress::Raw::Zlib 2.200
Hello,
This may well be a duplicate of #46
Compress::Raw::Zlib 2.200 has been released, and the changes mention zlib_ng
Installing IO-Compress against that causes test failures in t/111const-deflate.t
root@6b6b6136e997:~/.cpanm/work/1655863178.1/IO-Compress-2.106# prove -lv t/111const-deflate.t
t/111const-deflate.t ..
1..355
Failed 355/355 subtests
Test Summary Report
-------------------
t/111const-deflate.t (Wstat: 139 Tests: 0 Failed: 0)
Non-zero wait status: 139
Parse errors: Bad plan. You planned 355 tests but ran 0.
Files=1, Tests=0, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.07 cusr 0.00 csys = 0.09 CPU)
Result: FAIL
An easy repro is:
docker run --rm perl:5.34 cpanm IO::Compress::Base
Pinning Compress::Raw::Zlib to the previous version "fixes" it:
docker run --rm perl:5.34 cpanm Compress::Raw::[email protected] IO::Compress::Base
IO::Uncompress::Unzip::getHeaderInfo() should provide (more) information in transparent mode
While I'm reporting issues with IO::Uncompress::Unzip:
When processing files that may turn out not to be compressed, I try to have a common code path.
getHeaderInfo() simply returns undef. I use the following code, which really belongs in getHeaderInfo.
my $member = $in->getHeaderInfo();
if( !defined $member ) {
my $size = ( stat $filename )[7] || 0;
$size = U64->new( int( $size / U64::HI_1 ), int( $size % U64::HI_1 ) );
$member = { Name => $filename,
Time => ( ( stat _ )[9] ) || 0,
Type => $type,
UncompressedLength => $size,
MethodName => 'uncompressed',
};
}Providing this pseudo-header metadata allows the following code to use @member->{*} to report filename, size, time, etc. (If this looks odd, in real life it's a subroutine...and $type is 'text', or if binmode has been called, data)
Note that the code in Uncompress::Base sets Name to undef and Type to 'plain', expecting a caller of getHeaderInfo to receive that - even though getHeaderInfo in fact returns undef.
I'm not especially happy about using the U64 method, but that's what getHeaderInfo returns for compressed files.
Note that what's returned for compressed files is not well documented - it should be.
This method returns either a hash reference (in scalar context) or a
list or hash references (in array context) that contains information
about each of the header fields in the compressed data stream(s).
requires a dive into the internals to figure out how to turn "information about" into specific usable data.
And it would be nice if u64::new did the math to split a 64-bit int (or on 32-bit systems, double) into the required upper and lower 1/2s.
Documentation Issues
There's no example showing how to use this as a stream filter (glaring omission, this being the purpose of the module!).
There's no example showing how to use [,OPTS] either - very confusing.
How does a stream get a filename? How are normal everyday users supposed to unzip files that have no name?
You probably want to add something like this, after you modify the below to put a filename (I suggest "output.csv") into the zip:-
echo mydata,tostore | perl -MIO::Compress::Zip -e 'IO::Compress::Zip::zip \*STDIN=>\*STDOUT,AutoClose=>1; ' >output.zip
The above, FYI, does:-
# unzip -l output.zip
Archive: output.zip
Length Date Time Name
-------- ---- ---- ----
15 07-06-19 21:56
-------- -------
15 1 file
(Note the missing name)
How to set encoding utf8 layer?
This code don't work for me:
$fh = IO::Uncompress::Gunzip->new($fname);
$fh->binmode(":encoding(utf8)");
The same code for IO::File works fine. What I have to do, that this work for IO::Uncompress::Gunzip?
Now I am forced to utf8::decode() the read data. Can you help me please?
Error when writing a zip container to a pipe
$ echo abc | perl -Mblib ./IO-Compress/bin/streamzip | cat > /tmp/xc.zip
Use of uninitialized value in concatenation (.) or string at /media/paul/Linux-Shared/base/perl/ext/Gzip/blib/lib/IO/Compress/Base.pm line 806.
Can't use an undefined value as an ARRAY reference at /media/paul/Linux-Shared/base/perl/ext/Gzip/blib/lib/IO/Compress/Zip.pm line 547.
$ xdump /tmp/xc.zip
00000000 50 4b 03 04 14 00 00 00 08 00 55 80 8b 54 00 00 PK........U..T..
00000010 00 00 00 00 00 00 00 00 00 00 01 00 00 00 2d 4b ..............-K
00000020 4c 4a e6 02 00 LJ...
$ perl -Mblib -MIO::Compress::Zip -e 'print "$IO::Compress::Zip::VERSION\n"'
2.104
silently stops reading into buffer after exactly 64MB
I do have a big gz file
gzip -cd ~/tmp/file.gz | wc -c
452058742
I am processing the file with the t/bigfile.t test script
#<<<
use strict; use warnings;
#>>>
use Test::More import => [ qw( BAIL_OUT is note use_ok ) ], tests => 4;
use Config qw( %Config );
use File::Temp qw( tempdir tempfile );
my $module;
BEGIN {
my $module = 'IO::Uncompress::Gunzip';
use_ok $module, 2.206, '$GunzipError' or BAIL_OUT "Cannot load module '$module'!";
}
note "Perl $] at $^X";
note "OS $Config{ osname } $Config{ osvers }";
my $input = shift;
note 'Input filename: ', $input;
note 'Input size in bytes: ', ( stat( $input ) )[ 7 ];
my $z = IO::Uncompress::Gunzip->new( $input, { Append => 1 } )
or BAIL_OUT "Cannot create $module object: $GunzipError!";
my $buffer;
my $status;
1 while $status = $z->read( $buffer ) > 0;
is $status, '', '$status is empty should be 0 (eof expected)';
is $GunzipError, '', '$GunzipError is empty';
my ( $fh, $output ) = tempfile( DIR => tempdir( CLEANUP => 1 ) );
note 'Output filename: ', $output;
print $fh $buffer;
my $size = ( stat( $fh ) )[ 7 ];
is $size / 1024 / 1024, 64, 'Output size is 64MB';
The output is this
prove --verbose t/bigfile.t :: ~/tmp/file.gz
t/bigfile.t ..
1..4
ok 1 - use IO::Uncompress::Gunzip;
# Perl 5.014004 at /opt/perlbrew/perls/perl-5.14.4/bin/perl
# OS aix 7.2.0.0
# Input filename: /home/micsw/tmp/file.gz
# Input size in bytes: 7324449
ok 2 - $status is empty should be 0 (eof expected)
ok 3 - $GunzipError is empty
# Output filename: /tmp/DMXW8Uh9rG/p4EPsfa2vT
ok 4 - Output size is 64MB
ok
All tests successful.
Files=1, Tests=4, 1 wallclock secs ( 0.02 usr 0.01 sys + 0.13 cusr 0.07 csys = 0.23 CPU)
Result: PASS
My expectation is that at least one of the two assertions
ok 2 - $status is empty should be 0 (eof expected)
ok 3 - $GunzipError is empty
should fail because the unzip action stops prematurely after 64MB without raising a warning or an exception.
I have used PerlIO::gzip as an alternative method to uncompress. The phenomenon is the same. Reading and uncompressing the data stops after 64MB. I do not understand why!
Support encoding of filenames
Issue is mostly from zip file create don Windows using a code page encoding. Unzipping on Linux doesn't work.
Various modules of 2.105 distribution still declared version 2.104
In the past it seems like the versions of the various modules have reflected that of the distribution itself. However, the latest release has mixed versions:
IO-Compress/lib/IO/Compress/Base.pm
Line 23 in 42e028f
Delete GZIP environment variable before running interop tests
Gzip allows default options to be set with the GZIP environment variable.
get the test harness to delete this env variable in CompTestUtils.pm to prevent unexpected results.
Same goes for BZIP, BZIP2 for Bzip2
ZSTD_CLEVEL for zstd.
XZ_DEFAULTS for xz
2.106: test suite is failing when `IO-Compress` is build against zlib-ng
Looks like there are few issues in test suite
+ cd IO-Compress-2.106
+ /usr/bin/make -O -j48 V=1 VERBOSE=1 test COMPRESS_ZLIB_RUN_ALL=1
PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/000prereq.t ................... ok
t/001bzip2.t .................... ok
t/001zlib-generic-deflate.t ..... ok
t/001zlib-generic-gzip.t ........ ok
t/001zlib-generic-rawdeflate.t .. ok
t/001zlib-generic-zip.t ......... ok
t/002any-deflate.t .............. ok
t/002any-gzip.t ................. ok
t/002any-rawdeflate.t ........... ok
t/002any-transparent.t .......... ok
t/002any-zip.t .................. ok
t/004gziphdr.t .................. ok
# Failed test (t/005defhdr.t at line 184)
# got: '7'
# expected: '5'
# Failed test (t/005defhdr.t at line 187)
# got: '1'
# expected: '9'
# Failed test (t/005defhdr.t at line 184)
# got: '7'
# expected: '5'
# Failed test (t/005defhdr.t at line 187)
# got: '1'
# expected: '9'
# Looks like you failed 4 tests of 596.
t/005defhdr.t ...................
Dubious, test returned 4 (wstat 1024, 0x400)
Failed 4/596 subtests
t/006zip.t ...................... ok
t/010examples-bzip2.t ........... ok
t/010examples-zlib.t ............ ok
t/011-streamzip.t ............... ok
t/01misc.t ...................... ok
t/020isize.t .................... ok
t/050interop-gzip.t ............. ok
t/100generic-bzip2.t ............ ok
t/100generic-deflate.t .......... ok
t/100generic-gzip.t ............. ok
t/100generic-rawdeflate.t ....... ok
t/100generic-zip.t .............. ok
t/101truncate-bzip2.t ........... ok
t/101truncate-deflate.t ......... ok
t/101truncate-gzip.t ............ ok
# Looks like you planned 626 tests but only ran 616.
t/101truncate-rawdeflate.t ......
Dubious, test returned 10 (wstat 2560, 0xa00)
Failed 10/626 subtests
t/101truncate-zip.t ............. ok
t/102tied-bzip2.t ............... ok
t/102tied-deflate.t ............. ok
t/102tied-gzip.t ................ ok
t/102tied-rawdeflate.t .......... ok
t/102tied-zip.t ................. ok
t/103newtied-bzip2.t ............ ok
t/103newtied-deflate.t .......... ok
t/103newtied-gzip.t ............. ok
t/103newtied-rawdeflate.t ....... ok
t/103newtied-zip.t .............. ok
t/104destroy-bzip2.t ............ ok
t/104destroy-deflate.t .......... ok
t/104destroy-gzip.t ............. ok
t/104destroy-rawdeflate.t ....... ok
t/104destroy-zip.t .............. ok
t/105oneshot-bzip2.t ............ ok
t/105oneshot-deflate.t .......... ok
t/105oneshot-gzip-only.t ........ ok
t/105oneshot-gzip.t ............. ok
t/105oneshot-rawdeflate.t ....... ok
t/105oneshot-zip-bzip2-only.t ... ok
t/105oneshot-zip-only.t ......... ok
t/105oneshot-zip-store-only.t ... ok
t/105oneshot-zip.t .............. ok
t/106prime-bzip2.t .............. ok
t/106prime-deflate.t ............ ok
t/106prime-gzip.t ............... ok
t/106prime-rawdeflate.t ......... ok
t/106prime-zip.t ................ ok
t/107multi-bzip2.t .............. ok
t/107multi-deflate.t ............ ok
t/107multi-gzip.t ............... ok
t/107multi-rawdeflate.t ......... ok
t/107multi-zip-only.t ........... ok
t/107multi-zip.t ................ ok
t/108anyunc-bzip2.t ............. ok
t/108anyunc-deflate.t ........... ok
t/108anyunc-gzip.t .............. ok
t/108anyunc-rawdeflate.t ........ ok
t/108anyunc-transparent.t ....... ok
t/108anyunc-zip.t ............... ok
t/109merge-deflate.t ............ ok
t/109merge-gzip.t ............... ok
t/109merge-rawdeflate.t ......... ok
t/109merge-zip.t ................ skipped: not implemented yet
t/110encode-bzip2.t ............. ok
t/110encode-deflate.t ........... ok
t/110encode-gzip.t .............. ok
t/110encode-rawdeflate.t ........ ok
t/110encode-zip.t ............... ok
t/111const-deflate.t ............ ok
t/112utf8-zip.t ................. ok
t/113issues.t ................... ok
t/999meta-json.t ................ skipped: Test::CPAN::Meta::JSON required for testing META.json
t/999meta-yml.t ................. skipped: Test::CPAN::Meta required for testing META.yml
t/999pod.t ...................... ok
# Failed test (t/cz-01version.t at line 34)
# got: '1.2.11'
# expected: '1.2.11.zlib-ng'
#
# The version of zlib.h does not match the version of libz
#
# You have zlib.h version 1.2.11
# and libz version 1.2.11.zlib-ng
#
# You probably have two versions of zlib installed on your system.
# Try removing the one you don't want to use and rebuild.
# Looks like you failed 1 test of 3.
t/cz-01version.t ................
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/3 subtests
# Failed test (t/cz-03zlib-v1.t at line 55)
# got: '1.2.11.zlib-ng'
# expected: '1.2.11'
# Failed test (t/cz-03zlib-v1.t at line 712)
# got: 1
# expected: -3
# Looks like you failed 2 tests of 472.
t/cz-03zlib-v1.t ................
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/472 subtests
t/cz-05examples.t ............... ok
# Failed test (t/cz-06gzsetp.t at line 48)
# got: '1.2.11.zlib-ng'
# expected: '1.2.11'
# Looks like you failed 1 test of 52.
t/cz-06gzsetp.t .................
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/52 subtests
# Failed test (t/cz-08encoding.t at line 43)
# got: '1.2.11.zlib-ng'
# expected: '1.2.11'
# Looks like you failed 1 test of 30.
t/cz-08encoding.t ...............
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/30 subtests
# Failed test (t/cz-14gzopen.t at line 34)
# got: '1.2.11.zlib-ng'
# expected: '1.2.11'
# Looks like you failed 1 test of 265.
t/cz-14gzopen.t .................
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/265 subtests
t/globmapper.t .................. ok
Test Summary Report
-------------------
t/005defhdr.t (Wstat: 1024 Tests: 596 Failed: 4)
Failed tests: 75, 78, 375, 378
Non-zero exit status: 4
t/101truncate-rawdeflate.t (Wstat: 2560 Tests: 616 Failed: 0)
Non-zero exit status: 10
Parse errors: Bad plan. You planned 626 tests but ran 616.
t/cz-01version.t (Wstat: 256 Tests: 3 Failed: 1)
Failed test: 2
Non-zero exit status: 1
t/cz-03zlib-v1.t (Wstat: 512 Tests: 472 Failed: 2)
Failed tests: 4, 196
Non-zero exit status: 2
t/cz-06gzsetp.t (Wstat: 256 Tests: 52 Failed: 1)
Failed test: 1
Non-zero exit status: 1
t/cz-08encoding.t (Wstat: 256 Tests: 30 Failed: 1)
Failed test: 2
Non-zero exit status: 1
t/cz-14gzopen.t (Wstat: 256 Tests: 265 Failed: 1)
Failed test: 3
Non-zero exit status: 1
Files=93, Tests=69154, 992 wallclock secs ( 5.82 usr 0.47 sys + 968.18 cusr 2.37 csys = 976.84 CPU)
Result: FAIL
Failed 7/93 test programs. 10/69154 subtests failed.
make: *** [Makefile:851: test_dynamic] Error 255
nextStream not updating filehandle correctly
We are using IO::Uncompress::Unzip to iterate through zip files for various processes. We initially look for a specific filename (case-insensitive match). The files in the archive are CSV.
We've hit a case where the filehandle doesn't seem to be "positioned" correctly when nextStream is called. In the attached sample.zip, the first file in the archive has one line. The code below is attempting to dump the content of the second file (metadata.csv) in the archive.
Sample code:
#!/usr/bin/env perl
use strict;
use warnings;
use IO::Uncompress::Unzip;
use Data::Dumper;
my $file = 'sample.zip';
my $unzip = IO::Uncompress::Unzip->new($file) or die "Couldn't open ZIP\n";
do {{
my $header = $unzip->getHeaderInfo or die "Couldn't read ZIP";
if ( $header->{Name} =~ m/^(metadata)\.csv$/i ) {
my @content = <$unzip>;
warn Dumper \@content;
}
}} while ($unzip->nextStream > 0);
Sample input:
sample.zip
Script output:
$VAR1 = [
'"organisation_path","collection_occasion_key","episode_key"
',
'"key","value"
',
'"version","2"
',
'"type","PMHC"
'
];
where the first line is from the first file in the archive, not the file we are slurping content from, which is incorrect.
If the content of both files are read, the output is as expected. The single line of the first file that seems to cause issues.
cannot reproduce "unknown key value(s)" croaking
perl 5.14.2
IO::Compress 2.206
I am using this function
sub _gunzip_file {
my ( $filename ) = @_;
my $gunzip = IO::Uncompress::Gunzip->new( $filename, { Append => 1, MultiStream => 1 } );
my $buffer = '';
1 while $gunzip->read( $buffer ) > 0;
die "gunzip failed: $GunzipError\n" if $GunzipError;
return $buffer;
}
to gunzip files. Recently the function croaks with the message
Compress::Raw::Zlib::Inflate::new: unknown key value(s) AppendOutput CRC32 ADLER32 at perl5/IO/Uncompress/Adapter/Inflate.pm line 34
Unfortunately until today this happens only once and I was not able to reproduce it even with the same .gz input file that has caused the croaking.
Any idea why it could randomly happen that the key values AppendOutput CRC32 ADLER32 are treated as unknown?
Missing dependency Compress::Raw::Zlib v2.095
The latest version of IO::Compress expects (per Makefile.PL) to have Compress::Raw::Zlib at the same version, but at the time of this writing, the latest version for Compress::Raw::Zlib is v2.094 dated July 13th.
Thanks for your work on these modules.
MANIFEST error for streamzip
Should be t/011-streamzip.t, not t/011streamzip.t
Please document (POD, README, etc.) that IO::Compress also supports xz and lzma compression
Besides the historic gzip, xz in the meanwhile seems the most often used compression method in the Linux. It definitely has superseeded bzip2 and I've never seen lzop really gaining a relevant market share.
Yet xz seems not yet supported by IO::Compress, especially not by IO::Uncompress::AnyUncompress whose name is a bit exaggerated given that it doesn't support the second most used compression.
IO::Uncompress::Unzip stream can't be read by Storable (PerlIO) - Demo showing how to fix
I have a small reproducer demonstrating that IO::Uncompress::Unzip's file descriptor won't work with Storable. Presumably there are other cases...
If I had to guess, it would be that the FH, which is passed to Storable's XS code, may not be positioned (for XS) correctly after header detection...Clearly, it works for ordinary Perl consumers.
Here is the sequence:
- Write a small hash to a file with Storable.
- Using system zip, create a zip file containing the Storable file.
- Read the Storable file from
\*STDIN& verify that it's what we wrote. (passes) - Read the Storable file from *STDIN with IUU, passing the file handle to Storable (fails)
- Read the zip archive of the Storable file from *STDIN with IUU, passing file handle to Storable (fails)
- Delete the file, extract it from the Zip archive, and read it from *STDIN with Storable (passes
From this, we can see that the file is written correctly by Storable, and can be read after a trip thru the Zip archive. Further, Storable is happy reading from a file handle.
BUT, neither the raw file nor the Zip member can be read correctly by Storable if IUU is in the middle.
IO::Uncompress::Unzip 2.102
In real life, the work-around is to write all 300MB of uncompressed data to a tempfile, and pass that handle to Storable. Which is suboptimal...
Here is the test program:
#!/usr/bin/perl
use warnings;
use strict;
use Storable( qw/store fd_retrieve/ );
use IO::Uncompress::Unzip;
exit unless( @ARGV);
my $cmd = shift;
if( $cmd eq "write" ) {
my $file = shift or die( "no file\n" );
my $hash = { key => 'value' };
store( $hash, $file );
system( "zip $file.zip $file" );
exit;
}
if( $cmd eq "storable" ) {
binmode( \*STDIN );
my $read = fd_retrieve( \*STDIN );
if( !$read || !exists $read->{key} ){
print "Failed\n";
} else {
print "OK\n";
}
exit;
}
if( $cmd eq 'uncompress' ) {
my $in = IO::Uncompress::Unzip->new( \*STDIN, { } ) or die( "IUU\n" );
binmode( $in );
my $read = fd_retrieve( $in );
if( !$read || !exists $read->{key} ){
print "Failed\n";
} else {
print "OK\n";
}
exit;
}
if( $cmd eq 'extract' ) {
my $file = shift or die( "no file" );
system( "rm $file; unzip $file.zip;" );
system( "$0 storable <$file" );
exit;
}
die( "Bad command $cmd\n" );
exit;And here are the commands (Yes, it fails with newer versions of Perl, and yes in real life there's error checking):
# ./IUZ-Storable.pl write foo # Write hash to file "foo" and zip
adding: foo (stored 0%)
# ./IUZ-Storable.pl storable <foo # Read with Storable and verify
OK
# ./IUZ-Storable.pl uncompress <foo # Read foo (transparent mode) via IUU fails
Magic number checking on storable string failed at /usr/lib/perl5/5.8.8/i386-linux-thread-multi/Storable.pm line 443, at ./IUZ-Storable.pl line 38.
# ./IUZ-Storable.pl uncompress <foo.zip # Read Zip archive via IUU fails
Magic number checking on storable string failed at /usr/lib/perl5/5.8.8/i386-linux-thread-multi/Storable.pm line 443, at ./IUZ-Storable.pl line 38.
# ./IUZ-Storable.pl extract foo # Extract from zip file and read/verify with Storable
Archive: foo.zip
extracting: foo
OK
#FWIW, strace doesn't show anything odd on the system side of the pipe.
strace -f ./IUZ-Storable.pl uncompress <foo
(...)
read(0, "pst0\4\t\0041234\4\4\4\10\3\1\0\0\0\n\5value\3\0\0\0k"..., 4096) = 34
write(2, "Magic number checking on storabl"..., 148Magic number checking on storable string failed at /usr/lib/perl5/5.8.8/i386-linu (...)
# And here's the entire Storable file (34 bytes);
## od -tao1 foo
0000000 p s t 0 eot ht eot 1 2 3 4 eot eot eot bs etx
160 163 164 060 004 011 004 061 062 063 064 004 004 004 010 003
0000020 soh nul nul nul nl enq v a l u e etx nul nul nul k
001 000 000 000 012 005 166 141 154 165 145 003 000 000 000 153
0000040 e y
145 171
0000042
How to use the OO interface?
Although I have read the documentation of IO::Compress::Gzip, it is unclear to me how the following functional call
gzip($inputFile => $outputFilehandle, {Strict => 0}) or die "gzip failed: $GzipError\n";
could be translated into an OO approach. $inputFile is a filename. $outputFilehandle is a raw read+write filehandle.
Furthermore I would like to know which instance(object) method I could use to access the header information of the zip archive. I have seen IO::Uncompress::Gunzip::getHeaderInfo() but I cannot believe that no such method exists for a gzip object itself because such an object should known how it has created the zip archive including its header.
WeakDecrypt should not be listed in MANIFEST
The release tarball on the CPAN for 2.103 contains a new file lib/IO/Compress/Zip/WeakDecrypt.pm which
- is not in git
- has an interesting license comment at the top
- doesn't seem to be used by the other modules
- needs Data::Peek which is not mentioned in the metadata
I'm a bit confused :)
IO-Compress/bin/zipdetails 2.105 fails porting/utils.t in core on i386 (32 bit integer) builds.
See Perl/perl5#19617
zipdetails includes 64 bit hex constants, when run on a 32 bit perl it will produce warnings like:
# Failed test 83 - utils/zipdetails compiles at porting/utils.t line 85
# got "Integer overflow in hexadecimal number at utils/zipdetails line 1432.
Integer overflow in hexadecimal number at utils/zipdetails line 2247.
Integer overflow in hexadecimal number at utils/zipdetails line 2248.
Integer overflow in hexadecimal number at utils/zipdetails line 2249.
Integer overflow in hexadecimal number at utils/zipdetails line 2250.
Integer overflow in hexadecimal number at utils/zipdetails line 2251.
Integer overflow in hexadecimal number at utils/zipdetails line 2252.
Integer overflow in hexadecimal number at utils/zipdetails line 2253.
Integer overflow in hexadecimal number at utils/zipdetails line 2254.
utils/zipdetails syntax OK
"
# expected "utils/zipdetails syntax OK
"
t/porting/utils .................................................. FAILED at test 83
Related to this I have created:
pmqs/zipdetails#8
pmqs/zipdetails#7
Perl/perl5#19618
Perl/perl5#19617
Calling nextStream on an IO::Uncompress::Zip object in Transparent mode dies when input is uncompressed
The recommended mechanism for processing all members of a Zip file is to call nextStream at EOF untiil it returns 0 (no more data).
If an uncompressed file is passed to such an application, the application doesn't know that the data is uncompressed. So at EOF it calls nextStream to see if there's another member.
IO::Compress::Unzip isa IO::Uncompress::RawInflate, which isa IO::Uncompress::Base, where the nextStream method call lands. In turn, it calls IO::Unompress::Base::gotoNextStream.
When IO::Unompress::Base::gotoNextStream calls $self->reset(), reset's subsequent call to *$self->{Uncomp}->reset() dies, since *$self->{Uncomp} is undefined in this case.
Demo:
$ cat >foo.txt <<EOF
1234
5678
EOF
$ ./nextstream
1234
5678
Can't call method "reset" on an undefined value at .../IO/Uncompress/Base.pm line 991, <GEN0> line 2.
$ modver IO::Uncompress::Unzip IO::Uncompress::Base
IO::Uncompress::Unzip 2.102
IO::Uncompress::Base 2.102#!/usr/bin/perl
use warnings;
use strict;
use IO::Uncompress::Unzip( qw/$UnzipError/);
my $in = IO::Uncompress::Unzip->new( "foo.txt",
{ AutoClose => 1, Transparent => 1,} ) or
die( "foo.txt: $UnzipError\n" );
my $status;
for( $status = 1; $status > 0; $status = $in->nextStream ) {
print $_ while( <$in> );
}
die( "error: $!\n" ) if( $status < 0 );
exit;This patch seems to work, at least for this case.
--- IO/Uncompress/Base.pm.orig 2021-02-28 03:31:51.000000000 -0500
+++ IO/Uncompress/Base.pm 2021-10-27 10:36:58.000000000 -0400
@@ -1004,10 +1004,12 @@
sub nextStream
{
my $self = shift ;
+ return 0 if( *$self->{Plain} );
+
my $status = $self->gotoNextStream();
$status == 1
or return $status ;
*$self->{Pending} = ''streamzip TODO
- Add compression level option
- filename encoding option
- minimal -- disable extra attributes
- select compression method by number
- zip member comment
- zip file comment
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.