juliaio / codeczlib.jl Goto Github PK
View Code? Open in Web Editor NEWzlib codecs for TranscodingStreams.jl.
License: Other
zlib codecs for TranscodingStreams.jl.
License: Other
I'm getting a build error on FreeBSD for v0.5.1. The error looks to be related to bicycle1885/ZlibBuilder#11 .
Installed CodecZlib ─ v0.5.1
Building CodecZlib → `~/.julia/packages/CodecZlib/DAjXH/deps/build.log`
┌ Error: Error building `CodecZlib`:
│ [ Info: Downloading https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.3/Zlib.v1.2.11.x86_64-unknown-freebsd11.1.tar.gz to /home/username/.julia/packages/CodecZlib/DAjXH/deps/usr/downloads/Zlib.v1.2.11.x86_64-unknown-freebsd11.1.tar.gz...
│ ERROR: LoadError: LibraryProduct(nothing, ["libz"], :libz, "Prefix(/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│ [1] error(::String) at ./error.jl:33
│ [2] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /home/username/.julia/packages/BinaryProvider/4F5Hq/src/Products.jl:414
│ [3] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0
│ [4] top-level scope at logging.jl:310
│ [5] top-level scope at /home/username/.julia/packages/CodecZlib/DAjXH/deps/build.jl:82
│ [6] include at ./boot.jl:317 [inlined]
│ [7] include_relative(::Module, ::String) at ./loading.jl:1041
│ [8] include(::Module, ::String) at ./sysimg.jl:29
│ [9] include(::String) at ./client.jl:388
│ [10] top-level scope at none:0
│ in expression starting at /home/username/.julia/packages/CodecZlib/DAjXH/deps/build.jl:78
[00:03:57] ######################################################################## 100.0%
└ @ Pkg.Operations /home/vagrant/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1069
username@hal ~/.j/p/C/D/deps julia build.jl --verbose Wed Dec 12 00:11:07 2018
Executing ~/.julia/config/startup.jl
[ Info: Directory /usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr/lib does not exist!
[ Info: Destination file /usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr/downloads/Zlib.v1.2.11.x86_64-unknown-freebsd11.1.tar.gz already exists, verifying...
[ Info: Hash cache is consistent, returning true
[ Info: Installing /usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr/downloads/Zlib.v1.2.11.x86_64-unknown-freebsd11.1.tar.gz into /usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr
[ Info: Removing files installed by manifests/Zlib.v1.2.11.x86_64-unknown-freebsd11.1.list
[ Info: share/man/man3/zlib.3 removed
[ Info: Culling empty directory share/man/man3
[ Info: include/zlib.h removed
[ Info: include/zconf.h removed
[ Info: Culling empty directory include
[ Info: logs/Zlib.log removed
[ Info: Culling empty directory logs
[ Info: /usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr/lib/libz.a does not exist, but ignoring
[ Info: /usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr/lib/pkgconfig/zlib.pc does not exist, but ignoring
[ Info: manifests/Zlib.v1.2.11.x86_64-unknown-freebsd11.1.list removed
[ Info: Could not locate libz inside /usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr/lib
ERROR: LoadError: LibraryProduct(nothing, ["libz"], :libz, "Prefix(/usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr)") is not satisfied, cannot generate deps.jl!
Stacktrace:
[1] error(::String) at ./error.jl:33
[2] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /home/username/.julia/packages/BinaryProvider/4F5Hq/src/Products.jl:414
[3] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0
[4] top-level scope at logging.jl:310
[5] top-level scope at /usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/build.jl:82
[6] include at ./boot.jl:317 [inlined]
[7] include_relative(::Module, ::String) at ./loading.jl:1041
[8] include(::Module, ::String) at ./sysimg.jl:29
[9] exec_options(::Base.JLOptions) at ./client.jl:229
[10] _start() at ./client.jl:421
in expression starting at /usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/build.jl:78
This is bad for corporate users. I'll be opening similar issues on affected important packages.
Building CodecZlib → `D:\Users\FNORO\.julia\packages\CodecZlib\9jDi1\deps\build.log`
┌ Error: Error building `CodecZlib`:
│ ┌ Warning: On Windows, creating file symlinks requires Administrator privileges
│ └ @ Base.Filesystem file.jl:794
│ [ Info: Downloading https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.4/Zlib.v1.2.11.x86_64-w64-mingw32.tar.gz to D:\Users\FNORO\.julia\packages\CodecZlib\9jDi1\deps\usr\downloads\Zlib.v1.2.11.x86_64-w64-mingw32.tar.gz...
│ ERROR: LoadError: Could not download https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.4/Zlib.v1.2.11.x86_64-w64-mingw32.tar.gz to D:\Users\FNORO\.julia\packages\CodecZlib\9jDi1\deps\usr\downloads\Zlib.v1.2.11.x86_64-w64-mingw32.tar.gz:
│ ErrorException("")
│ Stacktrace:
│ [1] error(::String) at .\error.jl:33
│ [2] #download#93(::Bool, ::Function, ::String, ::String) at D:\Users\FNORO\.julia\packages\BinaryProvider\4F5Hq\src\PlatformEngines.jl:498
│ [3] #download at .\none:0 [inlined]
│ [4] #download_verify#94(::Bool, ::Bool, ::Bool, ::Function, ::String, ::String, ::String) at D:\Users\FNORO\.julia\packages\BinaryProvider\4F5Hq\src\PlatformEngines.jl:567
│ [5] #download_verify at .\none:0 [inlined]
│ [6] #install#133(::Prefix, ::String, ::Bool, ::Bool, ::Bool, ::Function, ::String, ::String) at D:\Users\FNORO\.julia\packages\BinaryProvider\4F5Hq\src\Prefix.jl:314
│ [7] (::getfield(BinaryProvider, Symbol("#kw##install")))(::NamedTuple{(:prefix, :force, :verbose),Tuple{Prefix,Bool,Bool}}, ::typeof(install), ::String, ::String) at .\none:0
│ [8] top-level scope at D:\Users\FNORO\.julia\packages\CodecZlib\9jDi1\deps\build.jl:89
│ [9] include at .\boot.jl:326 [inlined]
│ [10] include_relative(::Module, ::String) at .\loading.jl:1038
│ [11] include(::Module, ::String) at .\sysimg.jl:29
│ [12] include(::String) at .\client.jl:403
│ [13] top-level scope at none:0
│ in expression starting at D:\Users\FNORO\.julia\packages\CodecZlib\9jDi1\deps\build.jl:78
│ Exce��o ao chamar "DownloadFile" com "2" argumento(s): "O servidor remoto retornou um erro: (407) Autentica��o de
│ Proxy Obrigat�ria."
│ No linha:5 caractere:1
│ + $webclient.DownloadFile("https://github.com/bicycle1885/ZlibBuilder/r ...
│ + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
│ + CategoryInfo : NotSpecified: (:) [], MethodInvocationException
│ + FullyQualifiedErrorId : WebException
│
└ @ Pkg.Operations C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Pkg\src\Operations.jl:1075
Ref:
JuliaIO/EzXML.jl#103
I am trying to install but I get a build error. Running on M1 Mac.
julia> Pkg.build("CodecZlib")
Building CodecZlib → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/05916673a2627dd91b4969ff8ba6941bc85a960e/build.log`
ERROR: Error building `CodecZlib`:
┌ Warning: Platform `arm64-apple-darwin21.1.0` is not an officially supported platform
└ @ BinaryProvider ~/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:450
ERROR: LoadError: KeyError: key "unknown" not found
They are both owned by JuliaIO
, both (de)compress gzip, and both are up-to-date?
Hi, I'm trying to build the package Pkg.build("CodecZlib")
but, I get the following error.
julia> Pkg.build("CodecZlib")
INFO: Building CodecZlib
Info: Downloading https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.0/Zlib.x86_64-apple-darwin14.tar.gz to /Users/ladino/.julia/v0.6/CodecZlib/deps/usr/downloads/Zlib.x86_64-apple-darwin14.tar.gz...
[14:50:57] curl: (48) An unknown option was passed in to libcurl
=====================================================================[ ERROR: CodecZlib ]======================================================================
LoadError: Could not download https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.0/Zlib.x86_64-apple-darwin14.tar.gz to /Users/ladino/.julia/v0.6/CodecZlib/deps/usr/downloads/Zlib.x86_64-apple-darwin14.tar.gz
while loading /Users/ladino/.julia/v0.6/CodecZlib/deps/build.jl, in expression starting on line 92
===============================================================================================================================================================
=======================================================================[ BUILD ERRORS ]========================================================================
WARNING: CodecZlib had build errors.
- packages with build errors remain installed in /Users/ladino/.julia/v0.6
- build the package(s) and all dependencies with `Pkg.build("CodecZlib")`
- build a single package by running its `deps/build.jl` script
===============================================================================================================================================================
julia>
Any suggestions on how to proceed?
Thanks in advance,
At build CodecZlib
(after doing add CodecZlib#master
) , I get the following error message (if I am logged in as a User)
ERROR: Can not create symbolic link : Access is denied. : C:\Users\Me\.julia\packages\CodecZlib\POfe\deps\usr\bin\libz.dll
ERROR: Can not create symbolic link : Access is denied. : C:\Users\Me\.julia\packages\CodecZlib\POfe\deps\usr\bin\libz-1.dll
Instead, when logged in as an Admin, then it works.
Most zipped MMTF files are unzipped fine by CodecZlib but a few seem to give strange behaviour. For example http://mmtf.rcsb.org/v1.0/full/1XXJ.mmtf.gz gives different behaviour using CodecZlib to using my system gunzip
.
Using gunzip
to get 1XXJ.mmtf and then unpacking:
julia> using MsgPack
julia> unpack(open("1XXJ.mmtf"))
Dict{Any,Any} with 39 entries:
"numGroups" => 1211
"chainNameList" => UInt8[0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x0d, 0x00, 0x00 … 0x00, 0x00, 0x43, 0x00, 0x00, 0x0…
"bondAtomList" => UInt8[0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x09, 0x30, 0x00, 0x00 … 0x24, 0xbb, 0x00, 0x00, 0x24, 0xc…
"bFactorList" => UInt8[0x00, 0x00, 0x00, 0x0a, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00 … 0xfb, 0x01, 0x03, 0x13, 0xfe, 0xd…
"groupIdList" => UInt8[0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x04, 0xbb, 0x00, 0x00 … 0x00, 0x03, 0x00, 0x00, 0x00, 0x0…
"structureId" => "1XXJ"
"numAtoms" => 9501
"unitCell" => Any[126.245, 142.265, 81.322, 90.0, 90.0, 90.0]
"groupList" => Any[Dict{Any,Any}("groupName"=>"VAL","bondAtomList"=>Any[1, 0, 2, 1, 3, 2, 4, 1, 5, 4, 6, 4],"elementL…
"numChains" => 13
"groupTypeList" => UInt8[0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x04, 0xbb, 0x00, 0x00 … 0x00, 0x05, 0x00, 0x00, 0x00, 0x0…
"numModels" => 1
"altLocList" => UInt8[0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, …
"rWork" => 0.189
"rFree" => 0.248
"chainIdList" => UInt8[0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x0d, 0x00, 0x00 … 0x00, 0x00, 0x4c, 0x00, 0x00, 0x0…
"bondOrderList" => UInt8[0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x04, 0x98, 0x00, 0x00 … 0x01, 0x01, 0x01, 0x01, 0x01, 0x0…
"resolution" => 2.8
"chainsPerModel" => Any[13]
"ncsOperatorList" => Any[]
"xCoordList" => UInt8[0x00, 0x00, 0x00, 0x0a, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00 … 0xf6, 0xba, 0x73, 0x1a, 0xb9, 0xd…
"yCoordList" => UInt8[0x00, 0x00, 0x00, 0x0a, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00 … 0xb3, 0xa4, 0x7d, 0x4e, 0xa5, 0x4…
"numBonds" => 9687
"secStructList" => UInt8[0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x04, 0xbb, 0x00, 0x00 … 0xff, 0xff, 0xff, 0xff, 0xff, 0xf…
"occupancyList" => UInt8[0x00, 0x00, 0x00, 0x09, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00 … 0x00, 0x06, 0x00, 0x00, 0x00, 0x6…
"groupsPerChain" => Any[295, 295, 295, 295, 1, 1, 1, 1, 1, 8, 6, 6, 6]
"insCodeList" => UInt8[0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x04, 0xbb, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, …
"releaseDate" => "2005-03-22"
⋮ => ⋮
Using CodecZlib and then unpacking:
julia> using MsgPack, CodecZlib
julia> stream = GzipDecompressorStream(open("1XXJ.mmtf.gz"))
julia> open("1XXJ.mmtf", "w") do output
for line in eachline(stream)
println(output, line)
end
end
julia> close(stream)
julia> unpack(open("1XXJ.mmtf"))
Dict{Any,Any} with 38 entries:
"sequenceIndexList" => UInt8[0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x04, 0xbb, 0x00, 0x00 … 0x00, 0x01, 0x00, 0x00, 0x00, 0…
"numGroups" => 1211
"chainNameList" => UInt8[0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x0d, 0x00, 0x00 … 0x00, 0x00, 0x43, 0x00, 0x00, 0…
"bondAtomList" => UInt8[0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x09, 0x30, 0x00, 0x00 … 0x24, 0xbb, 0x00, 0x00, 0x24, 0…
98 => 70
114 => 76
"groupIdList" => UInt8[0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x04, 0xbb, 0x00, 0x00 … 0x00, 0x03, 0x00, 0x00, 0x00, 0…
"structureId" => "1XXJ"
"numAtoms" => 9501
"unitCell" => Any[126.245, 142.265, 81.322, 90.0, 90.0, 90.0]
"groupList" => Any[Dict{Any,Any}("groupName"=>"VAL","bondAtomList"=>Any[1, 0, 2, 1, 3, 2, 4, 1, 5, 4, 6, 4],"elemen…
"numChains" => 13
"groupTypeList" => UInt8[0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x04, 0xbb, 0x00, 0x00 … 0x00, 0x05, 0x00, 0x00, 0x00, 0…
"numModels" => 1
"altLocList" => UInt8[0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00…
"rFree" => 0.248
"chainIdList" => UInt8[0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x0d, 0x00, 0x00 … 0x00, 0x00, 0x4c, 0x00, 0x00, 0…
"bondOrderList" => UInt8[0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x04, 0x98, 0x00, 0x00 … 0x01, 0x01, 0x01, 0x01, 0x01, 0…
"resolution" => 2.8
"chainsPerModel" => Any[13]
116 => UInt8[0x00, 0x00, 0x00, 0x0a, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00 … 0xfb, 0x01, 0x03, 0x13, 0xfe, 0…
"xCoordList" => UInt8[0x00, 0x00, 0x00, 0x0a, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00 … 0xf6, 0xba, 0x73, 0x1a, 0xb9, 0…
"yCoordList" => UInt8[0x00, 0x00, 0x00, 0x0a, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00 … 0xb3, 0xa4, 0x7d, 0x4e, 0xa5, 0…
"numBonds" => 9687
"secStructList" => UInt8[0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x04, 0xbb, 0x00, 0x00 … 0xff, 0xff, 0xff, 0xff, 0xff, 0…
"occupancyList" => UInt8[0x00, 0x00, 0x00, 0x09, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00 … 0x00, 0x06, 0x00, 0x00, 0x00, 0…
"groupsPerChain" => Any[295, 295, 295, 295, 1, 1, 1, 1, 1, 8, 6, 6, 6]
"insCodeList" => UInt8[0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x04, 0xbb, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00…
⋮ => ⋮
Some of the dictionary keys are Int64
in this case, which is incorrect. Any idea what's going on? Other examples are 2ILI and 4X4T.
The names seem synomous to me,
and the code for each looks identical.
Am I missing something?
I have been running some data processing in which I compress the output array of bytes (FlatBuffer built bytes) using the transcode
API, however sometimes some of the workers die due to this crash inside zlib.
From worker 2:
From worker 2: signal (11): Segmentation fault
From worker 2: in expression starting at no file:0
From worker 2: unknown function (ip: 0x7f85e663d04d)
From worker 2: read_buf at /root/.julia/packages/CodecZlib/DAjXH/deps/usr/lib/libz.so (unknown line)
From worker 2: fill_window at /root/.julia/packages/CodecZlib/DAjXH/deps/usr/lib/libz.so (unknown line)
From worker 2: deflate_slow at /root/.julia/packages/CodecZlib/DAjXH/deps/usr/lib/libz.so (unknown line)
From worker 2: deflate at /root/.julia/packages/CodecZlib/DAjXH/deps/usr/lib/libz.so (unknown line)
From worker 2: deflate! at /root/.julia/packages/CodecZlib/DAjXH/src/libz.jl:77 [inlined]
From worker 2: process at /root/.julia/packages/CodecZlib/DAjXH/src/compression.jl:175 [inlined]
From worker 2: callprocess at /root/.julia/packages/TranscodingStreams/SaPZ8/src/stream.jl:603
From worker 2: flushbuffer at /root/.julia/packages/TranscodingStreams/SaPZ8/src/stream.jl:560
From worker 2: flushbufferall at /root/.julia/packages/TranscodingStreams/SaPZ8/src/stream.jl:567 [inlined]
From worker 2: write at /root/.julia/packages/TranscodingStreams/SaPZ8/src/stream.jl:450 [inlined]
From worker 2: transcode at /root/.julia/packages/TranscodingStreams/SaPZ8/src/transcode.jl:85
From worker 2: transcode at /root/.julia/packages/TranscodingStreams/SaPZ8/src/transcode.jl:34
From worker 2: loadPartAndStore at ./REPL[24]:6
From worker 2: unknown function (ip: 0x7f85d42848d7)
From worker 2: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
From worker 2: #28 at ./REPL[27]:7
From worker 2: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
From worker 2: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1538 [inlined]
From worker 2: jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:563
From worker 2: #112 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/process_messages.jl:269
From worker 2: run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/process_messages.jl:56
From worker 2: macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/process_messages.jl:269 [inlined]
From worker 2: #111 at ./task.jl:262
From worker 2: jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
From worker 2: jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1538 [inlined]
From worker 2: start_task at /buildworker/worker/package_linux64/build/src/task.c:268
From worker 2: unknown function (ip: 0xffffffffffffffff)
From worker 2: Allocations: 2455881495 (Pool: 2455857282; Big: 24213); GC: 5298
Worker 2 terminated.
I could not find any reproducible example, it just happens every so often. The code I have been using is somewhat reminiscent to this one.
using CodecZlib
using AWSS3
files = # files to process
@everywhere function loadPartAndStore(file)
part = # download file
outputBytes = #process part
AWSS3.s3_put(..., CodecZlib.transcode(CodecZlib.GzipCompressor, outputBytes))
end
pmap(loadPartAndStore, files)
I am aware of the fact that this is more of zlib
issue, however it might be possible, that the way I use it is somewhat non-standard. For example I know, that the code above allocates the working space of GzipCompressor
every iteration, that is mainly because I am not sure how to treat the preallocation (described here) in distributed environment.
Julia v0.7.0
CodecZlib v0.5.1
TranscodingStreams v0.8.1
I was studying why FASTX.jl on gzipped FASTQ files was so slow in the benchmark at https://github.com/lh3/biofast. It appears the unzipping is at least 2x slower than whatever he uses for his C code.
I confirmed this by timing this code:
function bar(path)
stream = GzipDecompressorStream(open(path))
v = read(stream)
close(stream)
return length(v)
end
On a gzipped FASTQ file and comparing it with $ gunzip -dc my_file.fq.gz > test
. Whereas the Julia code took around 2.1 seconds, the Bash code took 0.6 seconds. Profiling confirms that nearly 100% of the time is spent in the ccall
to libz
. Hence, it appears to be a problem with the binary itself.
Jaakko Ruohio on the Julia Slack suggested this might have something to do with how the Libz artifact is compiled. Perhaps it's missing some compile flags which makes it poorly optimized?
Update: JuliaPackaging/Yggdrasil#1051 sped up the Julia code by 2x, but it remains around 2x slower than both gunzip
and C code that calls into zlib.h
.
I'm unable to build CodecZlib
and am experiencing the following build error:
julia> Pkg.build("CodecZlib")
INFO: Building CodecZlib
Info: Downloading https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.0/Zlib.x86_64-w64-mingw32.tar.gz to C:\Users\jcarey\.julia\v0.6\CodecZlib\deps\usr\downloads
\Zlib.x86_64-w64-mingw32.tar.gz...
Exception setting "SecurityProtocol": "Cannot convert null to type "System.Net.SecurityProtocolType" due to invalid enumeration values. Specify one of the following enumeratio
n values and try again. The possible enumeration values are "Ssl3, Tls"."
At line:1 char:35
+ [System.Net.ServicePointManager]:: <<<< SecurityProtocol =
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : PropertyAssignmentException
Exception calling "DownloadFile" with "2" argument(s): "The underlying connection was closed: An unexpected error occurred on a send."
At line:4 char:24
+ $webclient.DownloadFile <<<< ("https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.0/Zlib.x86_64-w64-mingw32.tar.gz", "C:\Users\jcarey\.julia\v0.6\CodecZlib\de
ps\usr\downloads\Zlib.x86_64-w64-mingw32.tar.gz")
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : DotNetMethodException
==============================================================================[ ERROR: CodecZlib ]==============================================================================
LoadError: Could not download https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.0/Zlib.x86_64-w64-mingw32.tar.gz to C:\Users\jcarey\.julia\v0.6\CodecZlib\deps\u
sr\downloads\Zlib.x86_64-w64-mingw32.tar.gz
while loading C:\Users\jcarey\.julia\v0.6\CodecZlib\deps\build.jl, in expression starting on line 92
================================================================================================================================================================================
================================================================================[ BUILD ERRORS ]================================================================================
WARNING: CodecZlib had build errors.
- packages with build errors remain installed in C:\Users\jcarey\.julia\v0.6
- build the package(s) and all dependencies with `Pkg.build("CodecZlib")`
- build a single package by running its `deps/build.jl` script
================================================================================================================================================================================
Is there something that I need to change with my setup, or is there a problem with CodecZlib
?
I am reading and processing a large gzip-compressed file and want to show some progress information, preferably with ProgressMeter.jl, which has the following idiom:
seekend(io)
fileSize = position(io)
seekstart(io)
p = Progress(fileSize, 1) # minimum update interval: 1 second
while !eof(io)
line = readline(io)
# Here's where you do all the hard, slow work
update!(p, position(io))
end
In the example above, io
would come from a GzipDecompressorStream
. But
julia> position(io)
ERROR: MethodError: no method matching position(::TranscodingStreams.TranscodingStream{CodecZlib.GzipDecompressor,IOStream})
I understand that the position may not be clearly defined because of buffering, but since seekend
and seekstart
are supported, is it possible to get eg the position of the original stream?
I ran into the issue below when trying to build (I came across this as I tried using JLD2
)
I do not know much about binary dependincies; it is interesting to see that the error refers to a file of BinaryProvider though.
any ideas?
julia> Pkg.checkout("CodecZlib")
WARNING: Base.Pkg is deprecated, run `using Pkg` instead
in module Main
[ Info: Checking out CodecZlib master...
[ Info: Pulling CodecZlib latest master...
[ Info: No packages to install, update or remove
julia> Pkg.upWARNING: Base.Pkg is deprecated, run `using Pkg` instead
in module Main
julia> Pkg.update()
WARNING: Base.Pkg is deprecated, run `using Pkg` instead
in module Main
[ Info: Updating METADATA...
[ Info: Updating CodecZlib master...
┌ Warning: Package FileIO: skipping update (dirty)...
└ @ Pkg.Entry entry.jl:431
[ Info: Updating WinRPM master...
[ Info: Updating JLD2 master...
[ Info: Updating OnlineStatsBase master...
[ Info: Updating PyCall master...
[ Info: Computing changes...
[ Info: No packages to install, update or remove
julia> Pkg.build("CodecZlib")
WARNING: Base.Pkg is deprecated, run `using Pkg` instead
in module Main
[ Info: Building CodecZlib
┌ Warning: `rsearch(s::AbstractString, c::Char)` is deprecated, use `coalesce(findlast(isequal(c), s), 0)` instead.
│ caller = extract_platform_key(::String) at Prefix.jl:210
└ @ BinaryProvider Prefix.jl:210
┌ Error: ------------------------------------------------------------
│ # Build failed for CodecZlib
│ exception =
│ LoadError: stat: invalid argument (EINVAL)
│ Stacktrace:
│ [1] stat(::String) at .\stat.jl:68
│ [2] isfile at .\stat.jl:303 [inlined]
│ [3] #install#106(::BinaryProvider.Prefix, ::Bool, ::Bool, ::Bool, ::Function, ::String, ::String) at C:\Users\bernhard.konig\.julia\v0.7\BinaryProvider\src\Prefix.jl:273
│ [4] (::getfield(BinaryProvider, Symbol("#kw##install")))(::NamedTuple{(:prefix, :force, :verbose),Tuple{BinaryProvider.Prefix,Bool,Bool}}, ::typeof(BinaryProvider.install), ::String, ::String) at .\<missing>:0
│ [5] top-level scope at C:\Users\bernhard.konig\.julia\v0.7\CodecZlib\deps\build.jl:92
│ [6] include(::Module, ::String) at .\boot.jl:306
│ [7] include_relative(::Module, ::String) at .\loading.jl:1067
│ [8] include at .\sysimg.jl:29 [inlined]
│ [9] include(::String) at .\loading.jl:1101
│ [10] top-level scope
│ [11] eval at .\boot.jl:309 [inlined]
│ [12] eval at .\sysimg.jl:74 [inlined]
│ [13] evalfile(::String, ::Array{String,1}) at .\loading.jl:1096 (repeats 2 times)
│ [14] #6 at .\none:13 [inlined]
│ [15] cd(::getfield(Main, Symbol("##6#8")){String}, ::String) at .\file.jl:61
│ [16] (::getfield(Main, Symbol("##5#7")))(::IOStream) at .\none:12
│ [17] #open#321(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(Main, Symbol("##5#7")), ::String, ::Vararg{String,N} where N) at .\iostream.jl:369
│ [18] open(::Function, ::String, ::String) at .\iostream.jl:367
│ [19] top-level scope
│ [20] eval at .\boot.jl:309 [inlined]
│ [21] eval(::Module, ::Expr) at .\sysimg.jl:74
│ [22] exec_options(::Base.JLOptions) at .\client.jl:304
│ [23] _start() at .\client.jl:455
│ in expression starting at C:\Users\bernhard.konig\.julia\v0.7\CodecZlib\deps\build.jl:92
└ @ Main none:16
┌ Warning: ------------------------------------------------------------
│ # Build error summary
│
│ CodecZlib had build errors.
│
│ - packages with build errors remain installed in C:\Users\bernhard.konig\.julia\v0.7
│ - build the package(s) and all dependencies with `Pkg.build("CodecZlib")`
│ - build a single package by running its `deps/build.jl` script
└ @ Pkg.Entry entry.jl:649
julia> versioninfo()
Julia Version 0.7.0-DEV.4690
Commit 78c7d87369* (2018-03-23 22:25 UTC)
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-3.9.1 (ORCJIT, skylake)
Environment:
JULIA_BINDIR = C:\Julia-0.7.X\bin\
JULIA_HOME = C:\Julia-0.6.X\bin\
julia>
How should one proceed to generate a valid empty gzip file using CodecZlib?
Here is what I tried:
import CodecZlib
gz = CodecZlib
fh = gz.GzipCompressorStream(open("test_out.gz", "w"))
close(fh)
However, the following results in an empty file (zero bytes) that gzip
cannot handle properly:
$ zcat test_out.gz
gzip: test_out.gz: unexpected end of file
$
I can generate valid gzipped file using Python:
>>> from gzip import open as gzopen
>>> with gzopen("test_out.gz", "w") as fh:
... pass
...
The resulting file is not really empty (a few bytes), and gzip
does not complain when I try to zcat it:
$ zcat test_out.gz
$
It would be nice to be able to copy a Codec like in python https://docs.python.org/3/library/zlib.html#zlib.Compress.copy
This can be useful when repeating compression of data with a long common prefix like in JuliaIO/TranscodingStreams.jl#92
trying to load the RDatasets
package I get the following error: CodecZlib.jl is not installed properly, run Pkg.build("CodecZlib") and restart Julia
. However, when I run Pkg.build("CodecZlib")
I get:
Building CodecZlib → `~/.julia/packages/CodecZlib/5t9zO/deps/build.log`
┌ Error: Error building `CodecZlib`:
│ ERROR: LoadError: LibraryProduct(nothing, ["libz"], :libz, "Prefix(/home/sam/.julia/packages/CodecZlib/5t9zO/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│ [1] error(::String) at ./error.jl:33
│ [2] write_deps_file(::String, ::Array{LibraryProduct,1}; verbose::Bool, isolate::Bool) at /home/sam/.julia/packages/BinaryProvider/GeAtj/src/Products.jl:419
│ [3] top-level scope at /home/sam/.julia/packages/CodecZlib/5t9zO/deps/build.jl:93
│ [4] include(::String) at ./client.jl:439
│ [5] top-level scope at none:5
│ in expression starting at /home/sam/.julia/packages/CodecZlib/5t9zO/deps/build.jl:78
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Operations.jl:892
This happens on Ubuntu 20.04 with Julia version:
versioninfo()
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-8.0.1 (ORCJIT, ivybridge)
Environment:
JULIA_NUM_THREADS = 8
https://stackoverflow.com/q/52590792/992687
Just wondering :)
Hi,
I am parsing through a zlib compressed XML file and get the below error during the decompression. The package that I am testing can be found here: https://bitbucket.org/SSamanipour/ms_import.jl/src/master/.
Any help is highly appreciated.
Cheers,
Saer
Got exception outside of a @test
zlib error: (code: -5)
Stacktrace:
[1] transcode(codec::CodecZlib.ZlibDecompressor, data::Vector{UInt8})
@ TranscodingStreams ~/.julia/packages/TranscodingStreams/IVlnc/src/transcode.jl:121
[2] transcode(#unused#::Type{CodecZlib.ZlibDecompressor}, data::Vector{UInt8})
@ TranscodingStreams ~/.julia/packages/TranscodingStreams/IVlnc/src/transcode.jl:34
[3] read_scan_info(line::LightXML.XMLElement, mz_thresh::Vector{Int64}, Int_thresh::Int64)
@ MS_Import ~/Desktop/dev/pkgs/ms_import.jl/src/MS_Import.jl:398
[4] read_scan(msRun::LightXML.XMLElement, mz_thresh::Vector{Int64}, Int_thresh::Int64)
@ MS_Import ~/Desktop/dev/pkgs/ms_import.jl/src/MS_Import.jl:330
[5] mzxml_read(path2mzxml::String, mz_thresh::Vector{Int64}, Int_thresh::Int64)
@ MS_Import ~/Desktop/dev/pkgs/ms_import.jl/src/MS_Import.jl:295
[6] import_files(pathin::String, filenames::Vector{String}, mz_thresh::Vector{Int64}, Int_thresh::Int64)
@ MS_Import ~/Desktop/dev/pkgs/ms_import.jl/src/MS_Import.jl:116
[7] import_files(pathin::String, filenames::Vector{String}, mz_thresh::Vector{Int64})
@ MS_Import ~/Desktop/dev/pkgs/ms_import.jl/src/MS_Import.jl:100
[8] macro expansion
@ ~/Desktop/dev/pkgs/ms_import.jl/test/runtests.jl:49 [inlined]
[9] macro expansion
@ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
[10] top-level scope
@ ~/Desktop/dev/pkgs/ms_import.jl/test/runtests.jl:34
[11] include(fname::String)
@ Base.MainInclude ./client.jl:444
[12] top-level scope
@ none:6
[13] eval
@ ./boot.jl:360 [inlined]
[14] exec_options(opts::Base.JLOptions)
@ Base ./client.jl:261
[15] _start()
@ Base ./client.jl:485
Sorry to bother you. I was trying to install this package and found that I needed to install the development libraries for llvm (this was on fedora 28). Should this be mentioned somewhere in the documentation?
Any chance anyone knows what this is?
julia> Pkg.build("CodecZlib")
Building CodecZlib → `~/.julia/packages/CodecZlib/9jDi1/deps/build.log`
┌ Error: Error building `CodecZlib`:
│ [ Info: Downloading https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.4/Zlib.v1.2.11.x86_64-apple-darwin14.tar.gz to /Users/Nick/.julia/packages/CodecZlib/9jDi1/deps/usr/downloads/Zlib.v1.2.11.x86_64-apple-darwin14.tar.gz...
│ ERROR: LoadError: Could not download https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.4/Zlib.v1.2.11.x86_64-apple-darwin14.tar.gz to /Users/Nick/.julia/packages/CodecZlib/9jDi1/deps/usr/downloads/Zlib.v1.2.11.x86_64-apple-darwin14.tar.gz:
│ ErrorException("")
│ Stacktrace:
│ [1] error(::String) at ./error.jl:33
│ [2] #download#89(::Bool, ::Function, ::String, ::String) at /Users/Nick/.julia/packages/BinaryProvider/4F5Hq/src/PlatformEngines.jl:498
│ [3] #download at ./none:0 [inlined]
│ [4] #download_verify#90(::Bool, ::Bool, ::Bool, ::Function, ::String, ::String, ::String) at /Users/Nick/.julia/packages/BinaryProvider/4F5Hq/src/PlatformEngines.jl:567
│ [5] #download_verify at ./none:0 [inlined]
│ [6] #install#129(::Prefix, ::String, ::Bool, ::Bool, ::Bool, ::Function, ::String, ::String) at /Users/Nick/.julia/packages/BinaryProvider/4F5Hq/src/Prefix.jl:314
│ [7] (::getfield(BinaryProvider, Symbol("#kw##install")))(::NamedTuple{(:prefix, :force, :verbose),Tuple{Prefix,Bool,Bool}}, ::typeof(install), ::String, ::String) at ./none:0
│ [8] top-level scope at /Users/Nick/.julia/packages/CodecZlib/9jDi1/deps/build.jl:89
│ [9] include at ./boot.jl:317 [inlined]
│ [10] include_relative(::Module, ::String) at ./loading.jl:1044
│ [11] include(::Module, ::String) at ./sysimg.jl:29
│ [12] include(::String) at ./client.jl:392
│ [13] top-level scope at none:0
│ in expression starting at /Users/Nick/.julia/packages/CodecZlib/9jDi1/deps/build.jl:78
│ [17:59:44] dyld: Library not loaded: @rpath/libssl.1.0.0.dylib
│ [17:59:44] Referenced from: /Users/Nick/anaconda/lib/libssh2.1.dylib
│ [17:59:44] Reason: image not found
└ @ Pkg.Operations /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1097
The example from the README
, compressed = transcode(GzipCompressor, "test string")
, gives the following error:
MethodError: transcode(::Type{GzipCompressor}, ::String) is ambiguous. Candidates:
transcode(T, src::String) in Base at c.jl:306
transcode(::Type{C}, args...) where C<:TranscodingStreams.Codec in TranscodingStreams at ...\.julia\packages\TranscodingStreams\5yQuA\src\transcode.jl:33
Possible fix, define
transcode(::Type{C}, ::String) where C<:TranscodingStreams.Codec
This seems to be a new issue with the recently released version of TranscodingStreams (v0.9.12).
Would this be a good package to implement the crc calls to zlib?
I am getting this error:
julia> Pkg.build("CodecZlib")
Building CodecZlib → `~/.julia/packages/CodecZlib/DAjXH/deps/build.log`
┌ Error: Error building `CodecZlib`:
│ ERROR: LoadError: IOError: could not spawn `/software/packages/julia/julia-1.0.2/bin/julia -Cnative -J/software/packages/julia/julia-1.0.2/lib/julia/sys.so --compile=yes --depwarn=yes -O0 --output-ji /home/trecakov/.julia/compiled/v1.0/BinaryProvider/ek6VZ.ji --output-incremental=yes --startup-file=no --history-file=no --warn-overwrite=yes --color=no --eval "while !eof(stdin)
│ code = readuntil(stdin, '\0')
│ eval(Meta.parse(code))
│ end
│ "`: resource temporarily unavailable (EAGAIN)
│ Stacktrace:
│ [1] _jl_spawn(::String, ::Array{String,1}, ::Cmd, ::Tuple{Base.PipeEndpoint,RawFD,RawFD}) at ./process.jl:367
│ [2] (::getfield(Base, Symbol("##494#495")){Cmd})(::Tuple{Base.PipeEndpoint,RawFD,RawFD}) at ./process.jl:509
│ [3] setup_stdio(::getfield(Base, Symbol("##494#495")){Cmd}, ::Tuple{Pipe,IOStream,IOStream}) at ./process.jl:490
│ [4] #_spawn#493(::Nothing, ::Function, ::Cmd, ::Tuple{Pipe,IOStream,IOStream}) at ./process.jl:508
│ [5] (::getfield(Base, Symbol("#kw##_spawn")))(::NamedTuple{(:chain,),Tuple{Nothing}}, ::typeof(Base._spawn), ::Cmd, ::Tuple{Pipe,IOStream,IOStream}) at ./none:0
│ [6] #_spawn#490(::Nothing, ::Function, ::Base.CmdRedirect, ::Tuple{Pipe,IOStream,IOStream}) at ./process.jl:398
│ [7] _spawn(::Base.CmdRedirect, ::Tuple{Pipe,IOStream,IOStream}) at ./process.jl:398
│ [8] #open#503(::Bool, ::Bool, ::Function, ::Base.CmdRedirect, ::IOStream) at ./process.jl:599
│ [9] #open at ./none:0 [inlined]
│ [10] open(::Base.CmdRedirect, ::String, ::IOStream) at ./process.jl:567
│ [11] create_expr_cache(::String, ::String, ::Array{Pair{Base.PkgId,UInt64},1}, ::Base.UUID) at ./loading.jl:1111
│ [12] macro expansion at ./logging.jl:306 [inlined]
│ [13] compilecache(::Base.PkgId, ::String) at ./loading.jl:1190
│ [14] _require(::Base.PkgId) at ./logging.jl:309
│ [15] require(::Base.PkgId) at ./loading.jl:858
│ [16] macro expansion at ./logging.jl:309 [inlined]
│ [17] require(::Module, ::Symbol) at ./loading.jl:840
│ [18] include at ./boot.jl:317 [inlined]
│ [19] include_relative(::Module, ::String) at ./loading.jl:1044
│ [20] include(::Module, ::String) at ./sysimg.jl:29
│ [21] include(::String) at ./client.jl:392
│ [22] top-level scope at none:0
│ in expression starting at /home/trecakov/.julia/packages/CodecZlib/DAjXH/deps/build.jl:1
└ @ Pkg.Operations /software/packages/julia/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1097
Any suggestions?
I'm trying to decompress a 60GB gzippe'd file (this is the compressed size, not sure what the uncompressed is but I expect a high compression ratio), and I hit this error:
ERROR: InexactError: trunc(UInt32, 4294986918)
Stacktrace:
[1] throw_inexacterror(f::Symbol, #unused#::Type{UInt32}, val::UInt64)
@ Core ./boot.jl:614
[2] checked_trunc_uint
@ ./boot.jl:644 [inlined]
[3] toUInt32
@ ./boot.jl:733 [inlined]
[4] UInt32
@ ./boot.jl:768 [inlined]
[5] convert
@ ./number.jl:7 [inlined]
[6] setproperty!
@ ./Base.jl:39 [inlined]
[7] process(codec::CodecZlib.GzipDecompressor, input::TranscodingStreams.Memory, output::TranscodingStreams.Memory, error::TranscodingStreams.Error)
@ CodecZlib ~/.julia/packages/CodecZlib/ruMLE/src/decompression.jl:160
It would seem that the issue lies in this line
CodecZlib.jl/src/decompression.jl
Line 160 in f9fddaa
avail_in
is a UInt32
(Cuint
) while size
is a UInt64
(UInt
, see TranscodingStreams.jl/src/memory.jl#L11)
I imagine that either Zstream.avail_in
needs to change to a Culong
or max(input.size, typemax(Cuint))
needs to be applied.
I got this warning:
Warning: Package CodecZlib does not have Libdl in its dependencies:
Please consider to update the code for CSV package.
If I enable DEBUG messages (by setting level = Logging.Debug
) then my output is completely flooded by TranscodingStreams | [Debug] called process()
. Is this message necessary to call as often as it is?
It seems that one can only decompress 0xffffff
(uncomressedbytes) at a time and the above python library is doing the whole thing in multiple shots -- can we not automatically handle this? (I'm hitting Header error at the moment in Julia if I attempt it)
This issue is used to trigger TagBot; feel free to unsubscribe.
If you haven't already, you should update your TagBot.yml
to include issue comment triggers.
Please see this post on Discourse for instructions and more details.
If you'd like for me to do this for you, comment TagBot fix
on this issue.
I'll open a PR within a few hours, please be patient!
Build error on FreeBSD 12.0. Likely related to bicycle1885/ZlibBuilder#11.
(v1.0) pkg> add CodecZlib
Updating registry at `~/.julia/registries/General`
Updating git-repo `https://github.com/JuliaRegistries/General.git`
Resolving package versions...
Installed StatsBase ─ v0.27.0
Updating `/usr/home/username/.julia/environments/v1.0/Project.toml`
[no changes]
Updating `/usr/home/username/.julia/environments/v1.0/Manifest.toml`
[2913bbd2] ↑ StatsBase v0.26.0 ⇒ v0.27.0
(v1.0) pkg> build CodecZlib
Building CodecZlib → `~/.julia/packages/CodecZlib/DAjXH/deps/build.log`
┌ Error: Error building `CodecZlib`:
│ ERROR: LoadError: LibraryProduct(nothing, ["libz"], :libz, "Prefix(/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│ [1] error(::String) at ./error.jl:33
│ [2] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /home/username/.julia/packages/BinaryProvider/4F5Hq/src/Products.jl:414
│ [3] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0
│ [4] top-level scope at /home/username/.julia/packages/CodecZlib/DAjXH/deps/build.jl:93
│ [5] include at ./boot.jl:317 [inlined]
│ [6] include_relative(::Module, ::String) at ./loading.jl:1044
│ [7] include(::Module, ::String) at ./sysimg.jl:29
│ [8] include(::String) at ./client.jl:392
│ [9] top-level scope at none:0
│ in expression starting at /home/username/.julia/packages/CodecZlib/DAjXH/deps/build.jl:78
└ @ Pkg.Operations /home/vagrant/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1097
(v1.0) pkg>
julia> versioninfo()
Julia Version 1.0.3
Commit 099e826241 (2018-12-18 01:34 UTC)
Platform Info:
OS: FreeBSD (x86_64-unknown-freebsd11.1)
CPU: AMD Opteron 63xx class CPU
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.0 (ORCJIT, bdver1)
uname -a
FreeBSD hostname.novalocal 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC amd64
Would it be possible to add the other IOStream
API methods, e.g. position()
, seek()/seekstart()/seekend()
to a generic TranscodingStream
?
Would it be possible to have 7z support?
This looks current in the doc, but no longer (I believe this is a pattern I used before) works for me:
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.3.0 (2019-11-26)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using Pkg; Pkg.status()
Project HH v0.1.0
Status `~/src/bitbucket.org/garborg/HH/Project.toml`
[4f18b42c] BusinessDays v0.9.6
[944b1d66] CodecZlib v0.6.0
[34da2185] Compat v3.2.0
[864edb3b] DataStructures v0.17.9
[cd3eb016] HTTP v0.8.8
[682c06a0] JSON v0.21.0
[6ac74813] MD5 v0.2.1
[295af30f] Revise v2.5.0
[ade2ca70] Dates
julia> using CodecZlib: transcode, GzipCompressor
julia> bytes = codeunits("asfjkl")
6-element Base.CodeUnits{UInt8,String}:
0x61
0x73
0x66
0x6a
0x6b
0x6c
julia> transcode(GzipCompressor, bytes)
26-element Array{UInt8,1}:
0x1f
0x8b
0x08
0x00
0x00
0x00
0x00
0x00
0x00
0x03
0x4b
0x2c
0x4e
0xcb
0xca
0xce
0x01
0x00
0x75
0x9e
0x18
0x55
0x06
0x00
0x00
0x00
julia> transcode(GzipCompressor(), bytes)
ERROR: zlib error: <no message> (code: -2)
Stacktrace:
[1] transcode(::GzipCompressor, ::Base.CodeUnits{UInt8,String}) at /home/sean/.julia/packages/TranscodingStreams/MsN8d/src/transcode.jl:121
[2] top-level scope at REPL[5]:1
I'm trying to read git packfiles, which contain chunks of Zlib-compressed data, so I want to do something like:
f = open(packfile)
nobjs = readheader(f)
for i = 1:nobjs
nbytes = readchunkheader(f) # number of bytes in _decompressed_ data
data = read(ZlibDecompressionStream(f), nbytes)
# do something with `data`
end
However the problem is that ZlibDecompressionStream
will buffer past the end of the relevant end of the data in f
(so the position in the second loop will be in the wrong place). Is it possible to set the position of f
back to the end of the zlib data?
Building CodecZlib → `~/.julia/packages/CodecZlib/3RErA/deps/build.log`
┌ Error: Error building `CodecZlib`:
│ ERROR: LoadError: LibraryProduct(nothing, ["libz"], :libz, "Prefix(/home/seth/.julia/packages/CodecZlib/3RErA/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│ [1] error(::String) at ./error.jl:33
│ [2] #write_deps_file#134(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /home/seth/.julia/packages/BinaryProvider/ZbFxn/src/Products.jl:393
│ [3] write_deps_file(::String, ::Array{LibraryProduct,1}) at /home/seth/.julia/packages/BinaryProvider/ZbFxn/src/Products.jl:380
│ [4] top-level scope at none:0
│ [5] include at ./boot.jl:317 [inlined]
│ [6] include_relative(::Module, ::String) at ./loading.jl:1038
│ [7] include(::Module, ::String) at ./sysimg.jl:29
│ [8] include(::String) at ./client.jl:388
│ [9] top-level scope at none:0
│ in expression starting at /home/seth/.julia/packages/CodecZlib/3RErA/deps/build.jl:98
└ @ Pkg.Operations /buildworker/worker/package_linuxarmv7l/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1068
pigz is meant as a multithreaded, drop-in replacement for zlib; could we add a depdendency on pigz_jll and switch to using it internally for CodecZlib.jl?
Thank you for fixing the is_file issue
I am still unable to build though, see below.
Any ideas?
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_) | Documentation: https://docs.julialang.org
_ _ _| |_ __ _ | Type "?help" for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.7.0-DEV.4690 (2018-03-23 22:25 UTC)
_/ |\__'_|_|_|\__'_| | Commit 78c7d87369* (11 days old master)
|__/ | x86_64-w64-mingw32
julia> Pkg.update();versioninfo()
WARNING: Base.Pkg is deprecated, run `using Pkg` instead
in module Main
[ Info: Updating METADATA...
[ Info: Updating cache of PooledArrays...
[ Info: Updating cache of BinaryProvider...
[ Info: Updating CodecZlib master...
┌ Warning: Package FileIO: skipping update (dirty)...
└ @ Pkg.Entry entry.jl:431
[ Info: Updating WinRPM master...
[ Info: Updating JLD2 master...
[ Info: Updating OnlineStatsBase master...
[ Info: Updating PyCall master...
[ Info: Updating BinaryProvider master...
[ Info: Computing changes...
[ Info: Upgrading CSV: v0.2.3 => v0.2.4
[ Info: Building CodecZlib
┌ Warning: `rsearch(s::AbstractString, c::Char)` is deprecated, use `coalesce(findlast(isequal(c), s), 0)` instead.
│ caller = extract_platform_key(::String) at Prefix.jl:223
└ @ BinaryProvider Prefix.jl:223
┌ Warning: `wait(t::Task)` is deprecated, use `fetch(t)` instead.
│ caller = macro expansion at OutputCollector.jl:62 [inlined]
└ @ Core OutputCollector.jl:62
┌ Warning: `wait(t::Task)` is deprecated, use `fetch(t)` instead.
│ caller = wait(::BinaryProvider.OutputCollector) at OutputCollector.jl:158
└ @ BinaryProvider OutputCollector.jl:158
┌ Warning: `wait(t::Task)` is deprecated, use `fetch(t)` instead.
│ caller = wait(::BinaryProvider.OutputCollector) at OutputCollector.jl:159
└ @ BinaryProvider OutputCollector.jl:159
┌ Warning: `find` is deprecated, use `findall` instead.
│ caller = parse_7z_list(::String) at PlatformEngines.jl:366
└ @ BinaryProvider PlatformEngines.jl:366
┌ Warning: `contains(haystack, needle)` is deprecated, use `occursin(needle, haystack)` instead.
│ caller = #75 at <missing>:0 [inlined]
└ @ Core <missing>:0
┌ Warning: `contains(haystack, needle)` is deprecated, use `occursin(needle, haystack)` instead.
│ caller = #75 at <missing>:0 [inlined]
└ @ Core <missing>:0
┌ Error: ------------------------------------------------------------
│ # Build failed for CodecZlib
│ exception =
│ LoadError: MethodError: no method matching keys(::Base.Generator{Array{SubString{String},1},getfield(BinaryProvider, Symbol("##75#81"))})
│ Closest candidates are:
│ keys(::Cmd) at process.jl:838
│ keys(::Tuple) at tuple.jl:42
│ keys(::Tuple, ::Tuple...) at tuple.jl:48
│ ...
│ Stacktrace:
│ [1] pairs(::Base.Generator{Array{SubString{String},1},getfield(BinaryProvider, Symbol("##75#81"))}) at .\abstractdict.jl:138
│ [2] findall at .\array.jl:1969 [inlined]
│ [3] find(::Base.Generator{Array{SubString{String},1},getfield(BinaryProvider, Symbol("##75#81"))}) at .\deprecated.jl:32
│ [4] parse_7z_list(::String) at C:\Users\bernhard.konig\.julia\v0.7\BinaryProvider\src\PlatformEngines.jl:366
│ [5] #list_tarball_files#119(::Bool, ::Function, ::String) at C:\Users\bernhard.konig\.julia\v0.7\BinaryProvider\src\Prefix.jl:446
│ [6] list_tarball_files at C:\Users\bernhard.konig\.julia\v0.7\BinaryProvider\src\Prefix.jl:433 [inlined]
│ [7] #install#109(::BinaryProvider.Prefix, ::Bool, ::Bool, ::Bool, ::Function, ::String, ::String) at C:\Users\bernhard.konig\.julia\v0.7\BinaryProvider\src\Prefix.jl:302
│ [8] (::getfield(BinaryProvider, Symbol("#kw##install")))(::NamedTuple{(:prefix, :force, :verbose),Tuple{BinaryProvider.Prefix,Bool,Bool}}, ::typeof(BinaryProvider.install), ::String, ::String) at .\<missing>:0
│ [9] top-level scope at C:\Users\bernhard.konig\.julia\v0.7\CodecZlib\deps\build.jl:92
│ [10] include(::Module, ::String) at .\boot.jl:306
│ [11] include_relative(::Module, ::String) at .\loading.jl:1067
│ [12] include at .\sysimg.jl:29 [inlined]
│ [13] include(::String) at .\loading.jl:1101
│ [14] top-level scope
│ [15] eval at .\boot.jl:309 [inlined]
│ [16] eval at .\sysimg.jl:74 [inlined]
│ [17] evalfile(::String, ::Array{String,1}) at .\loading.jl:1096 (repeats 2 times)
│ [18] #6 at .\none:13 [inlined]
│ [19] cd(::getfield(Main, Symbol("##6#8")){String}, ::String) at .\file.jl:61
│ [20] (::getfield(Main, Symbol("##5#7")))(::IOStream) at .\none:12
│ [21] #open#321(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(Main, Symbol("##5#7")), ::String, ::Vararg{String,N} where N) at .\iostream.jl:369
│ [22] open(::Function, ::String, ::String) at .\iostream.jl:367
│ [23] top-level scope
│ [24] eval at .\boot.jl:309 [inlined]
│ [25] eval(::Module, ::Expr) at .\sysimg.jl:74
│ [26] exec_options(::Base.JLOptions) at .\client.jl:304
│ [27] _start() at .\client.jl:455
│ in expression starting at C:\Users\bernhard.konig\.julia\v0.7\CodecZlib\deps\build.jl:92
└ @ Main none:16
[ Info: Building SpecialFunctions
WARNING: importing deprecated binding Base.Associative into BinDeps.
WARNING: Base.Associative is deprecated, use AbstractDict instead.
likely near C:\Users\bernhard.konig\.julia\v0.7\SpecialFunctions\deps\binaries.jl:44
in @install at C:\Users\bernhard.konig\.julia\v0.7\BinDeps\src\dependencies.jl
┌ Warning: ------------------------------------------------------------
│ # Build error summary
│
│ CodecZlib had build errors.
│
│ - packages with build errors remain installed in C:\Users\bernhard.konig\.julia\v0.7
│ - build the package(s) and all dependencies with `Pkg.build("CodecZlib")`
│ - build a single package by running its `deps/build.jl` script
└ @ Pkg.Entry entry.jl:649
Julia Version 0.7.0-DEV.4690
Commit 78c7d87369* (2018-03-23 22:25 UTC)
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-3.9.1 (ORCJIT, skylake)
Environment:
JULIA_BINDIR = C:\Julia-0.7.X\bin\
JULIA_HOME = C:\Julia-0.6.X\bin\
julia>
With LZ4 one can pass a source pointer and a destination pointer to a Julia function to avoid copying result over:
https://github.com/tamasgal/UnROOT.jl/pull/128/files#diff-525588e68b2421901be164965272940effa093de733a3fd36c4b9a4344b8c20cR154
how to do it with Zlib? We're aware that ZStream is stateful, so it's not exactly straightforward, but it may be worth providing such a function? Happy to make a PR
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.