Giter Club home page Giter Club logo

codeczlib.jl's People

Contributors

ararslan avatar bicycle1885 avatar bjarthur avatar dependabot[bot] avatar felixhorger avatar fredrikekre avatar jakobnissen avatar joaoaparicio avatar juliatagbot avatar mkitti avatar musm avatar nhz2 avatar ranocha avatar rened avatar staticfloat avatar stefankarpinski avatar viralbshah avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

codeczlib.jl's Issues

Build error on FreeBSD for v0.5.1.

I'm getting a build error on FreeBSD for v0.5.1. The error looks to be related to bicycle1885/ZlibBuilder#11 .

 Installed CodecZlib ─ v0.5.1
  Building CodecZlib → `~/.julia/packages/CodecZlib/DAjXH/deps/build.log`
┌ Error: Error building `CodecZlib`:
│ [ Info: Downloading https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.3/Zlib.v1.2.11.x86_64-unknown-freebsd11.1.tar.gz to /home/username/.julia/packages/CodecZlib/DAjXH/deps/usr/downloads/Zlib.v1.2.11.x86_64-unknown-freebsd11.1.tar.gz...
│ ERROR: LoadError: LibraryProduct(nothing, ["libz"], :libz, "Prefix(/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /home/username/.julia/packages/BinaryProvider/4F5Hq/src/Products.jl:414
│  [3] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0
│  [4] top-level scope at logging.jl:310
│  [5] top-level scope at /home/username/.julia/packages/CodecZlib/DAjXH/deps/build.jl:82
│  [6] include at ./boot.jl:317 [inlined]
│  [7] include_relative(::Module, ::String) at ./loading.jl:1041
│  [8] include(::Module, ::String) at ./sysimg.jl:29
│  [9] include(::String) at ./client.jl:388
│  [10] top-level scope at none:0
│ in expression starting at /home/username/.julia/packages/CodecZlib/DAjXH/deps/build.jl:78
[00:03:57] ######################################################################## 100.0%
└ @ Pkg.Operations /home/vagrant/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1069
 username@hal  ~/.j/p/C/D/deps  julia build.jl --verbose                                                                                                                                                       Wed Dec 12 00:11:07 2018
Executing ~/.julia/config/startup.jl
[ Info: Directory /usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr/lib does not exist!
[ Info: Destination file /usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr/downloads/Zlib.v1.2.11.x86_64-unknown-freebsd11.1.tar.gz already exists, verifying...
[ Info: Hash cache is consistent, returning true
[ Info: Installing /usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr/downloads/Zlib.v1.2.11.x86_64-unknown-freebsd11.1.tar.gz into /usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr
[ Info: Removing files installed by manifests/Zlib.v1.2.11.x86_64-unknown-freebsd11.1.list
[ Info:   share/man/man3/zlib.3 removed
[ Info:   Culling empty directory share/man/man3
[ Info:   include/zlib.h removed
[ Info:   include/zconf.h removed
[ Info:   Culling empty directory include
[ Info:   logs/Zlib.log removed
[ Info:   Culling empty directory logs
[ Info:   /usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr/lib/libz.a does not exist, but ignoring
[ Info:   /usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr/lib/pkgconfig/zlib.pc does not exist, but ignoring
[ Info:   manifests/Zlib.v1.2.11.x86_64-unknown-freebsd11.1.list removed
[ Info: Could not locate libz inside /usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr/lib
ERROR: LoadError: LibraryProduct(nothing, ["libz"], :libz, "Prefix(/usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr)") is not satisfied, cannot generate deps.jl!
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /home/username/.julia/packages/BinaryProvider/4F5Hq/src/Products.jl:414
 [3] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0
 [4] top-level scope at logging.jl:310
 [5] top-level scope at /usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/build.jl:82
 [6] include at ./boot.jl:317 [inlined]
 [7] include_relative(::Module, ::String) at ./loading.jl:1041
 [8] include(::Module, ::String) at ./sysimg.jl:29
 [9] exec_options(::Base.JLOptions) at ./client.jl:229
 [10] _start() at ./client.jl:421
in expression starting at /usr/home/username/.julia/packages/CodecZlib/DAjXH/deps/build.jl:78

Building CodecZlib.jl on windows requires admin privileges

This is bad for corporate users. I'll be opening similar issues on affected important packages.

  Building CodecZlib → `D:\Users\FNORO\.julia\packages\CodecZlib\9jDi1\deps\build.log`
┌ Error: Error building `CodecZlib`:
│ ┌ Warning: On Windows, creating file symlinks requires Administrator privileges
│ └ @ Base.Filesystem file.jl:794
│ [ Info: Downloading https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.4/Zlib.v1.2.11.x86_64-w64-mingw32.tar.gz to D:\Users\FNORO\.julia\packages\CodecZlib\9jDi1\deps\usr\downloads\Zlib.v1.2.11.x86_64-w64-mingw32.tar.gz...
│ ERROR: LoadError: Could not download https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.4/Zlib.v1.2.11.x86_64-w64-mingw32.tar.gz to D:\Users\FNORO\.julia\packages\CodecZlib\9jDi1\deps\usr\downloads\Zlib.v1.2.11.x86_64-w64-mingw32.tar.gz:
│ ErrorException("")
│ Stacktrace:
│  [1] error(::String) at .\error.jl:33
│  [2] #download#93(::Bool, ::Function, ::String, ::String) at D:\Users\FNORO\.julia\packages\BinaryProvider\4F5Hq\src\PlatformEngines.jl:498
│  [3] #download at .\none:0 [inlined]
│  [4] #download_verify#94(::Bool, ::Bool, ::Bool, ::Function, ::String, ::String, ::String) at D:\Users\FNORO\.julia\packages\BinaryProvider\4F5Hq\src\PlatformEngines.jl:567
│  [5] #download_verify at .\none:0 [inlined]
│  [6] #install#133(::Prefix, ::String, ::Bool, ::Bool, ::Bool, ::Function, ::String, ::String) at D:\Users\FNORO\.julia\packages\BinaryProvider\4F5Hq\src\Prefix.jl:314
│  [7] (::getfield(BinaryProvider, Symbol("#kw##install")))(::NamedTuple{(:prefix, :force, :verbose),Tuple{Prefix,Bool,Bool}}, ::typeof(install), ::String, ::String) at .\none:0
│  [8] top-level scope at D:\Users\FNORO\.julia\packages\CodecZlib\9jDi1\deps\build.jl:89
│  [9] include at .\boot.jl:326 [inlined]
│  [10] include_relative(::Module, ::String) at .\loading.jl:1038
│  [11] include(::Module, ::String) at .\sysimg.jl:29
│  [12] include(::String) at .\client.jl:403
│  [13] top-level scope at none:0
│ in expression starting at D:\Users\FNORO\.julia\packages\CodecZlib\9jDi1\deps\build.jl:78
│ Exce��o ao chamar "DownloadFile" com "2" argumento(s): "O servidor remoto retornou um erro: (407) Autentica��o de
│ Proxy Obrigat�ria."
│ No linha:5 caractere:1
│ + $webclient.DownloadFile("https://github.com/bicycle1885/ZlibBuilder/r ...
│ + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
│     + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
│     + FullyQualifiedErrorId : WebException
│
└ @ Pkg.Operations C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Pkg\src\Operations.jl:1075

Ref:
JuliaIO/EzXML.jl#103

Package cannot be installed on M1 Macs

I am trying to install but I get a build error. Running on M1 Mac.

julia> Pkg.build("CodecZlib")
    Building CodecZlib → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/05916673a2627dd91b4969ff8ba6941bc85a960e/build.log`
ERROR: Error building `CodecZlib`:
┌ Warning: Platform `arm64-apple-darwin21.1.0` is not an officially supported platform
└ @ BinaryProvider ~/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:450
ERROR: LoadError: KeyError: key "unknown" not found

Error while building

Hi, I'm trying to build the package Pkg.build("CodecZlib") but, I get the following error.

julia> Pkg.build("CodecZlib")
INFO: Building CodecZlib
Info: Downloading https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.0/Zlib.x86_64-apple-darwin14.tar.gz to /Users/ladino/.julia/v0.6/CodecZlib/deps/usr/downloads/Zlib.x86_64-apple-darwin14.tar.gz...
[14:50:57] curl: (48) An unknown option was passed in to libcurl
=====================================================================[ ERROR: CodecZlib ]======================================================================

LoadError: Could not download https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.0/Zlib.x86_64-apple-darwin14.tar.gz to /Users/ladino/.julia/v0.6/CodecZlib/deps/usr/downloads/Zlib.x86_64-apple-darwin14.tar.gz
while loading /Users/ladino/.julia/v0.6/CodecZlib/deps/build.jl, in expression starting on line 92

===============================================================================================================================================================

=======================================================================[ BUILD ERRORS ]========================================================================

WARNING: CodecZlib had build errors.

 - packages with build errors remain installed in /Users/ladino/.julia/v0.6
 - build the package(s) and all dependencies with `Pkg.build("CodecZlib")`
 - build a single package by running its `deps/build.jl` script

===============================================================================================================================================================

julia>

Any suggestions on how to proceed?

Thanks in advance,

installing/building on 0.7-beta2, Win64 user account

At build CodecZlib (after doing add CodecZlib#master) , I get the following error message (if I am logged in as a User)

ERROR: Can not create symbolic link : Access is denied. : C:\Users\Me\.julia\packages\CodecZlib\POfe\deps\usr\bin\libz.dll

ERROR: Can not create symbolic link : Access is denied. : C:\Users\Me\.julia\packages\CodecZlib\POfe\deps\usr\bin\libz-1.dll

Instead, when logged in as an Admin, then it works.

Unzipping a MMTF file gives strange results sometimes

Most zipped MMTF files are unzipped fine by CodecZlib but a few seem to give strange behaviour. For example http://mmtf.rcsb.org/v1.0/full/1XXJ.mmtf.gz gives different behaviour using CodecZlib to using my system gunzip.

Using gunzip to get 1XXJ.mmtf and then unpacking:

julia> using MsgPack

julia> unpack(open("1XXJ.mmtf"))
Dict{Any,Any} with 39 entries:
  "numGroups"       => 1211
  "chainNameList"   => UInt8[0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x0d, 0x00, 0x00    0x00, 0x00, 0x43, 0x00, 0x00, 0x0
  "bondAtomList"    => UInt8[0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x09, 0x30, 0x00, 0x00    0x24, 0xbb, 0x00, 0x00, 0x24, 0xc
  "bFactorList"     => UInt8[0x00, 0x00, 0x00, 0x0a, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00    0xfb, 0x01, 0x03, 0x13, 0xfe, 0xd
  "groupIdList"     => UInt8[0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x04, 0xbb, 0x00, 0x00    0x00, 0x03, 0x00, 0x00, 0x00, 0x0
  "structureId"     => "1XXJ"
  "numAtoms"        => 9501
  "unitCell"        => Any[126.245, 142.265, 81.322, 90.0, 90.0, 90.0]
  "groupList"       => Any[Dict{Any,Any}("groupName"=>"VAL","bondAtomList"=>Any[1, 0, 2, 1, 3, 2, 4, 1, 5, 4, 6, 4],"elementL…
  "numChains"       => 13
  "groupTypeList"   => UInt8[0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x04, 0xbb, 0x00, 0x00  …  0x00, 0x05, 0x00, 0x00, 0x00, 0x0…
  "numModels"       => 1
  "altLocList"      => UInt8[0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, …
  "rWork"           => 0.189
  "rFree"           => 0.248
  "chainIdList"     => UInt8[0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x0d, 0x00, 0x00  …  0x00, 0x00, 0x4c, 0x00, 0x00, 0x0…
  "bondOrderList"   => UInt8[0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x04, 0x98, 0x00, 0x00  …  0x01, 0x01, 0x01, 0x01, 0x01, 0x0…
  "resolution"      => 2.8
  "chainsPerModel"  => Any[13]
  "ncsOperatorList" => Any[]
  "xCoordList"      => UInt8[0x00, 0x00, 0x00, 0x0a, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00  …  0xf6, 0xba, 0x73, 0x1a, 0xb9, 0xd…
  "yCoordList"      => UInt8[0x00, 0x00, 0x00, 0x0a, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00  …  0xb3, 0xa4, 0x7d, 0x4e, 0xa5, 0x4…
  "numBonds"        => 9687
  "secStructList"   => UInt8[0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x04, 0xbb, 0x00, 0x00  …  0xff, 0xff, 0xff, 0xff, 0xff, 0xf…
  "occupancyList"   => UInt8[0x00, 0x00, 0x00, 0x09, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00  …  0x00, 0x06, 0x00, 0x00, 0x00, 0x6…
  "groupsPerChain"  => Any[295, 295, 295, 295, 1, 1, 1, 1, 1, 8, 6, 6, 6]
  "insCodeList"     => UInt8[0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x04, 0xbb, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, …
  "releaseDate"     => "2005-03-22"
  ⋮                 => ⋮

Using CodecZlib and then unpacking:

julia> using MsgPack, CodecZlib

julia> stream = GzipDecompressorStream(open("1XXJ.mmtf.gz"))

julia> open("1XXJ.mmtf", "w") do output
           for line in eachline(stream)
               println(output, line)
           end
       end

julia> close(stream)

julia> unpack(open("1XXJ.mmtf"))
Dict{Any,Any} with 38 entries:
  "sequenceIndexList" => UInt8[0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x04, 0xbb, 0x00, 0x00    0x00, 0x01, 0x00, 0x00, 0x00, 0
  "numGroups"         => 1211
  "chainNameList"     => UInt8[0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x0d, 0x00, 0x00    0x00, 0x00, 0x43, 0x00, 0x00, 0
  "bondAtomList"      => UInt8[0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x09, 0x30, 0x00, 0x00    0x24, 0xbb, 0x00, 0x00, 0x24, 0
  98                  => 70
  114                 => 76
  "groupIdList"       => UInt8[0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x04, 0xbb, 0x00, 0x00    0x00, 0x03, 0x00, 0x00, 0x00, 0
  "structureId"       => "1XXJ"
  "numAtoms"          => 9501
  "unitCell"          => Any[126.245, 142.265, 81.322, 90.0, 90.0, 90.0]
  "groupList"         => Any[Dict{Any,Any}("groupName"=>"VAL","bondAtomList"=>Any[1, 0, 2, 1, 3, 2, 4, 1, 5, 4, 6, 4],"elemen…
  "numChains"         => 13
  "groupTypeList"     => UInt8[0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x04, 0xbb, 0x00, 0x00  …  0x00, 0x05, 0x00, 0x00, 0x00, 0…
  "numModels"         => 1
  "altLocList"        => UInt8[0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00…
  "rFree"             => 0.248
  "chainIdList"       => UInt8[0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x0d, 0x00, 0x00  …  0x00, 0x00, 0x4c, 0x00, 0x00, 0…
  "bondOrderList"     => UInt8[0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x04, 0x98, 0x00, 0x00  …  0x01, 0x01, 0x01, 0x01, 0x01, 0…
  "resolution"        => 2.8
  "chainsPerModel"    => Any[13]
  116                 => UInt8[0x00, 0x00, 0x00, 0x0a, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00  …  0xfb, 0x01, 0x03, 0x13, 0xfe, 0…
  "xCoordList"        => UInt8[0x00, 0x00, 0x00, 0x0a, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00  …  0xf6, 0xba, 0x73, 0x1a, 0xb9, 0…
  "yCoordList"        => UInt8[0x00, 0x00, 0x00, 0x0a, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00  …  0xb3, 0xa4, 0x7d, 0x4e, 0xa5, 0…
  "numBonds"          => 9687
  "secStructList"     => UInt8[0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x04, 0xbb, 0x00, 0x00  …  0xff, 0xff, 0xff, 0xff, 0xff, 0…
  "occupancyList"     => UInt8[0x00, 0x00, 0x00, 0x09, 0x00, 0x00, 0x25, 0x1d, 0x00, 0x00  …  0x00, 0x06, 0x00, 0x00, 0x00, 0…
  "groupsPerChain"    => Any[295, 295, 295, 295, 1, 1, 1, 1, 1, 8, 6, 6, 6]
  "insCodeList"       => UInt8[0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x04, 0xbb, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00…
  ⋮                   => ⋮

Some of the dictionary keys are Int64 in this case, which is incorrect. Any idea what's going on? Other examples are 2ILI and 4X4T.

Segmentation fault when using transcode

I have been running some data processing in which I compress the output array of bytes (FlatBuffer built bytes) using the transcode API, however sometimes some of the workers die due to this crash inside zlib.

From worker 2:
From worker 2:	signal (11): Segmentation fault
From worker 2:	in expression starting at no file:0
From worker 2:	unknown function (ip: 0x7f85e663d04d)
From worker 2:	read_buf at /root/.julia/packages/CodecZlib/DAjXH/deps/usr/lib/libz.so (unknown line)
From worker 2:	fill_window at /root/.julia/packages/CodecZlib/DAjXH/deps/usr/lib/libz.so (unknown line)
From worker 2:	deflate_slow at /root/.julia/packages/CodecZlib/DAjXH/deps/usr/lib/libz.so (unknown line)
From worker 2:	deflate at /root/.julia/packages/CodecZlib/DAjXH/deps/usr/lib/libz.so (unknown line)
From worker 2:	deflate! at /root/.julia/packages/CodecZlib/DAjXH/src/libz.jl:77 [inlined]
From worker 2:	process at /root/.julia/packages/CodecZlib/DAjXH/src/compression.jl:175 [inlined]
From worker 2:	callprocess at /root/.julia/packages/TranscodingStreams/SaPZ8/src/stream.jl:603
From worker 2:	flushbuffer at /root/.julia/packages/TranscodingStreams/SaPZ8/src/stream.jl:560
From worker 2:	flushbufferall at /root/.julia/packages/TranscodingStreams/SaPZ8/src/stream.jl:567 [inlined]
From worker 2:	write at /root/.julia/packages/TranscodingStreams/SaPZ8/src/stream.jl:450 [inlined]
From worker 2:	transcode at /root/.julia/packages/TranscodingStreams/SaPZ8/src/transcode.jl:85
From worker 2:	transcode at /root/.julia/packages/TranscodingStreams/SaPZ8/src/transcode.jl:34
From worker 2:	loadPartAndStore at ./REPL[24]:6
From worker 2:	unknown function (ip: 0x7f85d42848d7)
From worker 2:	jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
From worker 2:	#28 at ./REPL[27]:7
From worker 2:	jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
From worker 2:	jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1538 [inlined]
From worker 2:	jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:563
From worker 2:	#112 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/process_messages.jl:269
From worker 2:	run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/process_messages.jl:56
From worker 2:	macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/process_messages.jl:269 [inlined]
From worker 2:	#111 at ./task.jl:262
From worker 2:	jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
From worker 2:	jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1538 [inlined]
From worker 2:	start_task at /buildworker/worker/package_linux64/build/src/task.c:268
From worker 2:	unknown function (ip: 0xffffffffffffffff)
From worker 2:	Allocations: 2455881495 (Pool: 2455857282; Big: 24213); GC: 5298
Worker 2 terminated.

I could not find any reproducible example, it just happens every so often. The code I have been using is somewhat reminiscent to this one.

using CodecZlib
using AWSS3

files = # files to process

@everywhere function loadPartAndStore(file)
	part = # download file
	outputBytes = #process part
	AWSS3.s3_put(..., CodecZlib.transcode(CodecZlib.GzipCompressor, outputBytes))
end

pmap(loadPartAndStore, files)

I am aware of the fact that this is more of zlib issue, however it might be possible, that the way I use it is somewhat non-standard. For example I know, that the code above allocates the working space of GzipCompressor every iteration, that is mainly because I am not sure how to treat the preallocation (described here) in distributed environment.

Julia v0.7.0
CodecZlib v0.5.1
TranscodingStreams v0.8.1

Upstream zlib library not well optimized

I was studying why FASTX.jl on gzipped FASTQ files was so slow in the benchmark at https://github.com/lh3/biofast. It appears the unzipping is at least 2x slower than whatever he uses for his C code.

I confirmed this by timing this code:

function bar(path)
	stream = GzipDecompressorStream(open(path))
	v = read(stream)
	close(stream)
	return length(v)
end

On a gzipped FASTQ file and comparing it with $ gunzip -dc my_file.fq.gz > test. Whereas the Julia code took around 2.1 seconds, the Bash code took 0.6 seconds. Profiling confirms that nearly 100% of the time is spent in the ccall to libz. Hence, it appears to be a problem with the binary itself.

Jaakko Ruohio on the Julia Slack suggested this might have something to do with how the Libz artifact is compiled. Perhaps it's missing some compile flags which makes it poorly optimized?

Update: JuliaPackaging/Yggdrasil#1051 sped up the Julia code by 2x, but it remains around 2x slower than both gunzip and C code that calls into zlib.h.

Error during build

I'm unable to build CodecZlib and am experiencing the following build error:

julia> Pkg.build("CodecZlib")
INFO: Building CodecZlib
Info: Downloading https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.0/Zlib.x86_64-w64-mingw32.tar.gz to C:\Users\jcarey\.julia\v0.6\CodecZlib\deps\usr\downloads
\Zlib.x86_64-w64-mingw32.tar.gz...
Exception setting "SecurityProtocol": "Cannot convert null to type "System.Net.SecurityProtocolType" due to invalid enumeration values. Specify one of the following enumeratio
n values and try again. The possible enumeration values are "Ssl3, Tls"."
At line:1 char:35
+ [System.Net.ServicePointManager]:: <<<< SecurityProtocol =
    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException
    + FullyQualifiedErrorId : PropertyAssignmentException

Exception calling "DownloadFile" with "2" argument(s): "The underlying connection was closed: An unexpected error occurred on a send."
At line:4 char:24
+ $webclient.DownloadFile <<<< ("https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.0/Zlib.x86_64-w64-mingw32.tar.gz", "C:\Users\jcarey\.julia\v0.6\CodecZlib\de
ps\usr\downloads\Zlib.x86_64-w64-mingw32.tar.gz")
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : DotNetMethodException

==============================================================================[ ERROR: CodecZlib ]==============================================================================


LoadError: Could not download https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.0/Zlib.x86_64-w64-mingw32.tar.gz to C:\Users\jcarey\.julia\v0.6\CodecZlib\deps\u
sr\downloads\Zlib.x86_64-w64-mingw32.tar.gz
while loading C:\Users\jcarey\.julia\v0.6\CodecZlib\deps\build.jl, in expression starting on line 92

================================================================================================================================================================================


================================================================================[ BUILD ERRORS ]================================================================================


WARNING: CodecZlib had build errors.

 - packages with build errors remain installed in C:\Users\jcarey\.julia\v0.6
 - build the package(s) and all dependencies with `Pkg.build("CodecZlib")`
 - build a single package by running its `deps/build.jl` script

================================================================================================================================================================================

Is there something that I need to change with my setup, or is there a problem with CodecZlib?

position (for an approximate progress bar)

I am reading and processing a large gzip-compressed file and want to show some progress information, preferably with ProgressMeter.jl, which has the following idiom:

seekend(io)
fileSize = position(io)
seekstart(io)
p = Progress(fileSize, 1)   # minimum update interval: 1 second
while !eof(io)
    line = readline(io)
    # Here's where you do all the hard, slow work
    
    update!(p, position(io))
end

In the example above, io would come from a GzipDecompressorStream. But

julia> position(io)
ERROR: MethodError: no method matching position(::TranscodingStreams.TranscodingStream{CodecZlib.GzipDecompressor,IOStream})

I understand that the position may not be clearly defined because of buffering, but since seekend and seekstart are supported, is it possible to get eg the position of the original stream?

unable to build under dev 0.7

I ran into the issue below when trying to build (I came across this as I tried using JLD2)

I do not know much about binary dependincies; it is interesting to see that the error refers to a file of BinaryProvider though.

any ideas?

julia> Pkg.checkout("CodecZlib")
WARNING: Base.Pkg is deprecated, run `using Pkg` instead
 in module Main
[ Info: Checking out CodecZlib master...
[ Info: Pulling CodecZlib latest master...
[ Info: No packages to install, update or remove

julia> Pkg.upWARNING: Base.Pkg is deprecated, run `using Pkg` instead
 in module Main
julia> Pkg.update()
WARNING: Base.Pkg is deprecated, run `using Pkg` instead
 in module Main
[ Info: Updating METADATA...
[ Info: Updating CodecZlib master...
┌ Warning: Package FileIO: skipping update (dirty)...
└ @ Pkg.Entry entry.jl:431
[ Info: Updating WinRPM master...
[ Info: Updating JLD2 master...
[ Info: Updating OnlineStatsBase master...
[ Info: Updating PyCall master...
[ Info: Computing changes...
[ Info: No packages to install, update or remove

julia> Pkg.build("CodecZlib")
WARNING: Base.Pkg is deprecated, run `using Pkg` instead
 in module Main
[ Info: Building CodecZlib
┌ Warning: `rsearch(s::AbstractString, c::Char)` is deprecated, use `coalesce(findlast(isequal(c), s), 0)` instead.
│   caller = extract_platform_key(::String) at Prefix.jl:210
└ @ BinaryProvider Prefix.jl:210
┌ Error: ------------------------------------------------------------
│ # Build failed for CodecZlib
│   exception =
│    LoadError: stat: invalid argument (EINVAL)
│    Stacktrace:
│     [1] stat(::String) at .\stat.jl:68
│     [2] isfile at .\stat.jl:303 [inlined]
│     [3] #install#106(::BinaryProvider.Prefix, ::Bool, ::Bool, ::Bool, ::Function, ::String, ::String) at C:\Users\bernhard.konig\.julia\v0.7\BinaryProvider\src\Prefix.jl:273
│     [4] (::getfield(BinaryProvider, Symbol("#kw##install")))(::NamedTuple{(:prefix, :force, :verbose),Tuple{BinaryProvider.Prefix,Bool,Bool}}, ::typeof(BinaryProvider.install), ::String, ::String) at .\<missing>:0
│     [5] top-level scope at C:\Users\bernhard.konig\.julia\v0.7\CodecZlib\deps\build.jl:92
│     [6] include(::Module, ::String) at .\boot.jl:306
│     [7] include_relative(::Module, ::String) at .\loading.jl:1067
│     [8] include at .\sysimg.jl:29 [inlined]
│     [9] include(::String) at .\loading.jl:1101
│     [10] top-level scope
│     [11] eval at .\boot.jl:309 [inlined]
│     [12] eval at .\sysimg.jl:74 [inlined]
│     [13] evalfile(::String, ::Array{String,1}) at .\loading.jl:1096 (repeats 2 times)
│     [14] #6 at .\none:13 [inlined]
│     [15] cd(::getfield(Main, Symbol("##6#8")){String}, ::String) at .\file.jl:61
│     [16] (::getfield(Main, Symbol("##5#7")))(::IOStream) at .\none:12
│     [17] #open#321(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(Main, Symbol("##5#7")), ::String, ::Vararg{String,N} where N) at .\iostream.jl:369
│     [18] open(::Function, ::String, ::String) at .\iostream.jl:367
│     [19] top-level scope
│     [20] eval at .\boot.jl:309 [inlined]
│     [21] eval(::Module, ::Expr) at .\sysimg.jl:74
│     [22] exec_options(::Base.JLOptions) at .\client.jl:304
│     [23] _start() at .\client.jl:455
│    in expression starting at C:\Users\bernhard.konig\.julia\v0.7\CodecZlib\deps\build.jl:92
└ @ Main none:16
┌ Warning: ------------------------------------------------------------
│ # Build error summary
│
│ CodecZlib had build errors.
│
│  - packages with build errors remain installed in C:\Users\bernhard.konig\.julia\v0.7
│  - build the package(s) and all dependencies with `Pkg.build("CodecZlib")`
│  - build a single package by running its `deps/build.jl` script
└ @ Pkg.Entry entry.jl:649

julia> versioninfo()
Julia Version 0.7.0-DEV.4690
Commit 78c7d87369* (2018-03-23 22:25 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, skylake)
Environment:
  JULIA_BINDIR = C:\Julia-0.7.X\bin\
  JULIA_HOME = C:\Julia-0.6.X\bin\

julia>

How to write valid empty gzip file?

How should one proceed to generate a valid empty gzip file using CodecZlib?

Here is what I tried:

import CodecZlib
gz = CodecZlib
fh = gz.GzipCompressorStream(open("test_out.gz", "w"))
close(fh)

However, the following results in an empty file (zero bytes) that gzip cannot handle properly:

$ zcat test_out.gz 

gzip: test_out.gz: unexpected end of file
$

I can generate valid gzipped file using Python:

>>> from gzip import open as gzopen
>>> with gzopen("test_out.gz", "w") as fh:
...     pass
... 

The resulting file is not really empty (a few bytes), and gzip does not complain when I try to zcat it:

$ zcat test_out.gz
$

Build error in Ubuntu 20.04

trying to load the RDatasets package I get the following error: CodecZlib.jl is not installed properly, run Pkg.build("CodecZlib") and restart Julia. However, when I run Pkg.build("CodecZlib") I get:

   Building CodecZlib → `~/.julia/packages/CodecZlib/5t9zO/deps/build.log`
┌ Error: Error building `CodecZlib`: 
│ ERROR: LoadError: LibraryProduct(nothing, ["libz"], :libz, "Prefix(/home/sam/.julia/packages/CodecZlib/5t9zO/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] write_deps_file(::String, ::Array{LibraryProduct,1}; verbose::Bool, isolate::Bool) at /home/sam/.julia/packages/BinaryProvider/GeAtj/src/Products.jl:419
│  [3] top-level scope at /home/sam/.julia/packages/CodecZlib/5t9zO/deps/build.jl:93
│  [4] include(::String) at ./client.jl:439
│  [5] top-level scope at none:5
│ in expression starting at /home/sam/.julia/packages/CodecZlib/5t9zO/deps/build.jl:78
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Operations.jl:892

This happens on Ubuntu 20.04 with Julia version:

versioninfo()
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, ivybridge)
Environment:
  JULIA_NUM_THREADS = 8

zlib error: <no message> (code: -5)

Hi,

I am parsing through a zlib compressed XML file and get the below error during the decompression. The package that I am testing can be found here: https://bitbucket.org/SSamanipour/ms_import.jl/src/master/.

Any help is highly appreciated.

Cheers,
Saer

Got exception outside of a @test
zlib error: (code: -5)
Stacktrace:
[1] transcode(codec::CodecZlib.ZlibDecompressor, data::Vector{UInt8})
@ TranscodingStreams ~/.julia/packages/TranscodingStreams/IVlnc/src/transcode.jl:121
[2] transcode(#unused#::Type{CodecZlib.ZlibDecompressor}, data::Vector{UInt8})
@ TranscodingStreams ~/.julia/packages/TranscodingStreams/IVlnc/src/transcode.jl:34
[3] read_scan_info(line::LightXML.XMLElement, mz_thresh::Vector{Int64}, Int_thresh::Int64)
@ MS_Import ~/Desktop/dev/pkgs/ms_import.jl/src/MS_Import.jl:398
[4] read_scan(msRun::LightXML.XMLElement, mz_thresh::Vector{Int64}, Int_thresh::Int64)
@ MS_Import ~/Desktop/dev/pkgs/ms_import.jl/src/MS_Import.jl:330
[5] mzxml_read(path2mzxml::String, mz_thresh::Vector{Int64}, Int_thresh::Int64)
@ MS_Import ~/Desktop/dev/pkgs/ms_import.jl/src/MS_Import.jl:295
[6] import_files(pathin::String, filenames::Vector{String}, mz_thresh::Vector{Int64}, Int_thresh::Int64)
@ MS_Import ~/Desktop/dev/pkgs/ms_import.jl/src/MS_Import.jl:116
[7] import_files(pathin::String, filenames::Vector{String}, mz_thresh::Vector{Int64})
@ MS_Import ~/Desktop/dev/pkgs/ms_import.jl/src/MS_Import.jl:100
[8] macro expansion
@ ~/Desktop/dev/pkgs/ms_import.jl/test/runtests.jl:49 [inlined]
[9] macro expansion
@ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
[10] top-level scope
@ ~/Desktop/dev/pkgs/ms_import.jl/test/runtests.jl:34
[11] include(fname::String)
@ Base.MainInclude ./client.jl:444
[12] top-level scope
@ none:6
[13] eval
@ ./boot.jl:360 [inlined]
[14] exec_options(opts::Base.JLOptions)
@ Base ./client.jl:261
[15] _start()
@ Base ./client.jl:485

llvm-devel packages required for building

Sorry to bother you. I was trying to install this package and found that I needed to install the development libraries for llvm (this was on fedora 28). Should this be mentioned somewhere in the documentation?

Error building CodecZlib (not #38)

Any chance anyone knows what this is?

julia> Pkg.build("CodecZlib")
  Building CodecZlib → `~/.julia/packages/CodecZlib/9jDi1/deps/build.log`
┌ Error: Error building `CodecZlib`: 
│ [ Info: Downloading https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.4/Zlib.v1.2.11.x86_64-apple-darwin14.tar.gz to /Users/Nick/.julia/packages/CodecZlib/9jDi1/deps/usr/downloads/Zlib.v1.2.11.x86_64-apple-darwin14.tar.gz...
│ ERROR: LoadError: Could not download https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.4/Zlib.v1.2.11.x86_64-apple-darwin14.tar.gz to /Users/Nick/.julia/packages/CodecZlib/9jDi1/deps/usr/downloads/Zlib.v1.2.11.x86_64-apple-darwin14.tar.gz:
│ ErrorException("")
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] #download#89(::Bool, ::Function, ::String, ::String) at /Users/Nick/.julia/packages/BinaryProvider/4F5Hq/src/PlatformEngines.jl:498
│  [3] #download at ./none:0 [inlined]
│  [4] #download_verify#90(::Bool, ::Bool, ::Bool, ::Function, ::String, ::String, ::String) at /Users/Nick/.julia/packages/BinaryProvider/4F5Hq/src/PlatformEngines.jl:567
│  [5] #download_verify at ./none:0 [inlined]
│  [6] #install#129(::Prefix, ::String, ::Bool, ::Bool, ::Bool, ::Function, ::String, ::String) at /Users/Nick/.julia/packages/BinaryProvider/4F5Hq/src/Prefix.jl:314
│  [7] (::getfield(BinaryProvider, Symbol("#kw##install")))(::NamedTuple{(:prefix, :force, :verbose),Tuple{Prefix,Bool,Bool}}, ::typeof(install), ::String, ::String) at ./none:0
│  [8] top-level scope at /Users/Nick/.julia/packages/CodecZlib/9jDi1/deps/build.jl:89
│  [9] include at ./boot.jl:317 [inlined]
│  [10] include_relative(::Module, ::String) at ./loading.jl:1044
│  [11] include(::Module, ::String) at ./sysimg.jl:29
│  [12] include(::String) at ./client.jl:392
│  [13] top-level scope at none:0
│ in expression starting at /Users/Nick/.julia/packages/CodecZlib/9jDi1/deps/build.jl:78
│ [17:59:44] dyld: Library not loaded: @rpath/libssl.1.0.0.dylib
│ [17:59:44]   Referenced from: /Users/Nick/anaconda/lib/libssh2.1.dylib
│ [17:59:44]   Reason: image not found
└ @ Pkg.Operations /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1097

error when running example from README

The example from the README, compressed = transcode(GzipCompressor, "test string"), gives the following error:

MethodError: transcode(::Type{GzipCompressor}, ::String) is ambiguous. Candidates:
  transcode(T, src::String) in Base at c.jl:306
  transcode(::Type{C}, args...) where C<:TranscodingStreams.Codec in TranscodingStreams at ...\.julia\packages\TranscodingStreams\5yQuA\src\transcode.jl:33
Possible fix, define
  transcode(::Type{C}, ::String) where C<:TranscodingStreams.Codec

This seems to be a new issue with the recently released version of TranscodingStreams (v0.9.12).

CRC call

Would this be a good package to implement the crc calls to zlib?

Error building `CodecZlib`

I am getting this error:

julia> Pkg.build("CodecZlib")
  Building CodecZlib → `~/.julia/packages/CodecZlib/DAjXH/deps/build.log`
┌ Error: Error building `CodecZlib`:
│ ERROR: LoadError: IOError: could not spawn `/software/packages/julia/julia-1.0.2/bin/julia -Cnative -J/software/packages/julia/julia-1.0.2/lib/julia/sys.so --compile=yes --depwarn=yes -O0 --output-ji /home/trecakov/.julia/compiled/v1.0/BinaryProvider/ek6VZ.ji --output-incremental=yes --startup-file=no --history-file=no --warn-overwrite=yes --color=no --eval "while !eof(stdin)
│     code = readuntil(stdin, '\0')
│     eval(Meta.parse(code))
│ end
│ "`: resource temporarily unavailable (EAGAIN)
│ Stacktrace:
│  [1] _jl_spawn(::String, ::Array{String,1}, ::Cmd, ::Tuple{Base.PipeEndpoint,RawFD,RawFD}) at ./process.jl:367
│  [2] (::getfield(Base, Symbol("##494#495")){Cmd})(::Tuple{Base.PipeEndpoint,RawFD,RawFD}) at ./process.jl:509
│  [3] setup_stdio(::getfield(Base, Symbol("##494#495")){Cmd}, ::Tuple{Pipe,IOStream,IOStream}) at ./process.jl:490
│  [4] #_spawn#493(::Nothing, ::Function, ::Cmd, ::Tuple{Pipe,IOStream,IOStream}) at ./process.jl:508
│  [5] (::getfield(Base, Symbol("#kw##_spawn")))(::NamedTuple{(:chain,),Tuple{Nothing}}, ::typeof(Base._spawn), ::Cmd, ::Tuple{Pipe,IOStream,IOStream}) at ./none:0
│  [6] #_spawn#490(::Nothing, ::Function, ::Base.CmdRedirect, ::Tuple{Pipe,IOStream,IOStream}) at ./process.jl:398
│  [7] _spawn(::Base.CmdRedirect, ::Tuple{Pipe,IOStream,IOStream}) at ./process.jl:398
│  [8] #open#503(::Bool, ::Bool, ::Function, ::Base.CmdRedirect, ::IOStream) at ./process.jl:599
│  [9] #open at ./none:0 [inlined]
│  [10] open(::Base.CmdRedirect, ::String, ::IOStream) at ./process.jl:567
│  [11] create_expr_cache(::String, ::String, ::Array{Pair{Base.PkgId,UInt64},1}, ::Base.UUID) at ./loading.jl:1111
│  [12] macro expansion at ./logging.jl:306 [inlined]
│  [13] compilecache(::Base.PkgId, ::String) at ./loading.jl:1190
│  [14] _require(::Base.PkgId) at ./logging.jl:309
│  [15] require(::Base.PkgId) at ./loading.jl:858
│  [16] macro expansion at ./logging.jl:309 [inlined]
│  [17] require(::Module, ::Symbol) at ./loading.jl:840
│  [18] include at ./boot.jl:317 [inlined]
│  [19] include_relative(::Module, ::String) at ./loading.jl:1044
│  [20] include(::Module, ::String) at ./sysimg.jl:29
│  [21] include(::String) at ./client.jl:392
│  [22] top-level scope at none:0
│ in expression starting at /home/trecakov/.julia/packages/CodecZlib/DAjXH/deps/build.jl:1
└ @ Pkg.Operations /software/packages/julia/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1097

Any suggestions?

Input too large?

I'm trying to decompress a 60GB gzippe'd file (this is the compressed size, not sure what the uncompressed is but I expect a high compression ratio), and I hit this error:

ERROR: InexactError: trunc(UInt32, 4294986918)
Stacktrace:
  [1] throw_inexacterror(f::Symbol, #unused#::Type{UInt32}, val::UInt64)
    @ Core ./boot.jl:614
  [2] checked_trunc_uint
    @ ./boot.jl:644 [inlined]
  [3] toUInt32
    @ ./boot.jl:733 [inlined]
  [4] UInt32
    @ ./boot.jl:768 [inlined]
  [5] convert
    @ ./number.jl:7 [inlined]
  [6] setproperty!
    @ ./Base.jl:39 [inlined]
  [7] process(codec::CodecZlib.GzipDecompressor, input::TranscodingStreams.Memory, output::TranscodingStreams.Memory, error::TranscodingStreams.Error)
    @ CodecZlib ~/.julia/packages/CodecZlib/ruMLE/src/decompression.jl:160

It would seem that the issue lies in this line

zstream.avail_in = input.size

avail_in is a UInt32 (Cuint) while size is a UInt64 (UInt, see TranscodingStreams.jl/src/memory.jl#L11)

I imagine that either Zstream.avail_in needs to change to a Culong or max(input.size, typemax(Cuint)) needs to be applied.

  • Julia 1.8.3
  • CodecZlib 0.7.0

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Cannot generate deps.jl

Build error on FreeBSD 12.0. Likely related to bicycle1885/ZlibBuilder#11.

(v1.0) pkg> add CodecZlib
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
 Installed StatsBase ─ v0.27.0
  Updating `/usr/home/username/.julia/environments/v1.0/Project.toml`
 [no changes]
  Updating `/usr/home/username/.julia/environments/v1.0/Manifest.toml`
  [2913bbd2] ↑ StatsBase v0.26.0 ⇒ v0.27.0

(v1.0) pkg> build CodecZlib
  Building CodecZlib → `~/.julia/packages/CodecZlib/DAjXH/deps/build.log`
┌ Error: Error building `CodecZlib`:
│ ERROR: LoadError: LibraryProduct(nothing, ["libz"], :libz, "Prefix(/home/username/.julia/packages/CodecZlib/DAjXH/deps/usr)") is not satisfied, cannot generate deps.jl!                                           
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /home/username/.julia/packages/BinaryProvider/4F5Hq/src/Products.jl:414                                                      
│  [3] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0                             
│  [4] top-level scope at /home/username/.julia/packages/CodecZlib/DAjXH/deps/build.jl:93
│  [5] include at ./boot.jl:317 [inlined]
│  [6] include_relative(::Module, ::String) at ./loading.jl:1044
│  [7] include(::Module, ::String) at ./sysimg.jl:29
│  [8] include(::String) at ./client.jl:392
│  [9] top-level scope at none:0
│ in expression starting at /home/username/.julia/packages/CodecZlib/DAjXH/deps/build.jl:78
└ @ Pkg.Operations /home/vagrant/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1097                                                                                             

(v1.0) pkg> 
julia> versioninfo()
Julia Version 1.0.3
Commit 099e826241 (2018-12-18 01:34 UTC)
Platform Info:
  OS: FreeBSD (x86_64-unknown-freebsd11.1)
  CPU: AMD Opteron 63xx class CPU
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, bdver1)
uname -a
FreeBSD hostname.novalocal 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC  amd64

IOStream API completeness

Would it be possible to add the other IOStream API methods, e.g. position(), seek()/seekstart()/seekend() to a generic TranscodingStream?

transcode failing if codec is initialized

This looks current in the doc, but no longer (I believe this is a pattern I used before) works for me:

   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.3.0 (2019-11-26)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Pkg; Pkg.status()
Project HH v0.1.0
    Status `~/src/bitbucket.org/garborg/HH/Project.toml`
  [4f18b42c] BusinessDays v0.9.6
  [944b1d66] CodecZlib v0.6.0
  [34da2185] Compat v3.2.0
  [864edb3b] DataStructures v0.17.9
  [cd3eb016] HTTP v0.8.8
  [682c06a0] JSON v0.21.0
  [6ac74813] MD5 v0.2.1
  [295af30f] Revise v2.5.0
  [ade2ca70] Dates 

julia> using CodecZlib: transcode, GzipCompressor

julia> bytes = codeunits("asfjkl")
6-element Base.CodeUnits{UInt8,String}:
 0x61
 0x73
 0x66
 0x6a
 0x6b
 0x6c

julia> transcode(GzipCompressor, bytes)
26-element Array{UInt8,1}:
 0x1f
 0x8b
 0x08
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0x03
 0x4b
 0x2c
 0x4e
 0xcb
 0xca
 0xce
 0x01
 0x00
 0x75
 0x9e
 0x18
 0x55
 0x06
 0x00
 0x00
 0x00

julia> transcode(GzipCompressor(), bytes)
ERROR: zlib error: <no message> (code: -2)
Stacktrace:
 [1] transcode(::GzipCompressor, ::Base.CodeUnits{UInt8,String}) at /home/sean/.julia/packages/TranscodingStreams/MsN8d/src/transcode.jl:121
 [2] top-level scope at REPL[5]:1

Find end of Zlib stream

I'm trying to read git packfiles, which contain chunks of Zlib-compressed data, so I want to do something like:

f = open(packfile)
nobjs = readheader(f)
for i = 1:nobjs
     nbytes = readchunkheader(f) # number of bytes in _decompressed_ data
     data = read(ZlibDecompressionStream(f), nbytes)
     # do something with `data`
end

However the problem is that ZlibDecompressionStream will buffer past the end of the relevant end of the data in f (so the position in the second loop will be in the wrong place). Is it possible to set the position of f back to the end of the zlib data?

Install fails on Raspberry Pi (arm)

  Building CodecZlib → `~/.julia/packages/CodecZlib/3RErA/deps/build.log`
┌ Error: Error building `CodecZlib`:
│ ERROR: LoadError: LibraryProduct(nothing, ["libz"], :libz, "Prefix(/home/seth/.julia/packages/CodecZlib/3RErA/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] #write_deps_file#134(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /home/seth/.julia/packages/BinaryProvider/ZbFxn/src/Products.jl:393
│  [3] write_deps_file(::String, ::Array{LibraryProduct,1}) at /home/seth/.julia/packages/BinaryProvider/ZbFxn/src/Products.jl:380
│  [4] top-level scope at none:0
│  [5] include at ./boot.jl:317 [inlined]
│  [6] include_relative(::Module, ::String) at ./loading.jl:1038
│  [7] include(::Module, ::String) at ./sysimg.jl:29
│  [8] include(::String) at ./client.jl:388
│  [9] top-level scope at none:0
│ in expression starting at /home/seth/.julia/packages/CodecZlib/3RErA/deps/build.jl:98
└ @ Pkg.Operations /buildworker/worker/package_linuxarmv7l/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1068

another build error on 0.7 dev

Thank you for fixing the is_file issue

I am still unable to build though, see below.
Any ideas?


   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.7.0-DEV.4690 (2018-03-23 22:25 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit 78c7d87369* (11 days old master)
|__/                   |  x86_64-w64-mingw32

julia> Pkg.update();versioninfo()
WARNING: Base.Pkg is deprecated, run `using Pkg` instead
 in module Main
[ Info: Updating METADATA...
[ Info: Updating cache of PooledArrays...
[ Info: Updating cache of BinaryProvider...
[ Info: Updating CodecZlib master...
┌ Warning: Package FileIO: skipping update (dirty)...
└ @ Pkg.Entry entry.jl:431
[ Info: Updating WinRPM master...
[ Info: Updating JLD2 master...
[ Info: Updating OnlineStatsBase master...
[ Info: Updating PyCall master...
[ Info: Updating BinaryProvider master...
[ Info: Computing changes...
[ Info: Upgrading CSV: v0.2.3 => v0.2.4
[ Info: Building CodecZlib
┌ Warning: `rsearch(s::AbstractString, c::Char)` is deprecated, use `coalesce(findlast(isequal(c), s), 0)` instead.
│   caller = extract_platform_key(::String) at Prefix.jl:223
└ @ BinaryProvider Prefix.jl:223
┌ Warning: `wait(t::Task)` is deprecated, use `fetch(t)` instead.
│   caller = macro expansion at OutputCollector.jl:62 [inlined]
└ @ Core OutputCollector.jl:62
┌ Warning: `wait(t::Task)` is deprecated, use `fetch(t)` instead.
│   caller = wait(::BinaryProvider.OutputCollector) at OutputCollector.jl:158
└ @ BinaryProvider OutputCollector.jl:158
┌ Warning: `wait(t::Task)` is deprecated, use `fetch(t)` instead.
│   caller = wait(::BinaryProvider.OutputCollector) at OutputCollector.jl:159
└ @ BinaryProvider OutputCollector.jl:159
┌ Warning: `find` is deprecated, use `findall` instead.
│   caller = parse_7z_list(::String) at PlatformEngines.jl:366
└ @ BinaryProvider PlatformEngines.jl:366
┌ Warning: `contains(haystack, needle)` is deprecated, use `occursin(needle, haystack)` instead.
│   caller = #75 at <missing>:0 [inlined]
└ @ Core <missing>:0
┌ Warning: `contains(haystack, needle)` is deprecated, use `occursin(needle, haystack)` instead.
│   caller = #75 at <missing>:0 [inlined]
└ @ Core <missing>:0
┌ Error: ------------------------------------------------------------
│ # Build failed for CodecZlib
│   exception =
│    LoadError: MethodError: no method matching keys(::Base.Generator{Array{SubString{String},1},getfield(BinaryProvider, Symbol("##75#81"))})
│    Closest candidates are:
│      keys(::Cmd) at process.jl:838
│      keys(::Tuple) at tuple.jl:42
│      keys(::Tuple, ::Tuple...) at tuple.jl:48
│      ...
│    Stacktrace:
│     [1] pairs(::Base.Generator{Array{SubString{String},1},getfield(BinaryProvider, Symbol("##75#81"))}) at .\abstractdict.jl:138
│     [2] findall at .\array.jl:1969 [inlined]
│     [3] find(::Base.Generator{Array{SubString{String},1},getfield(BinaryProvider, Symbol("##75#81"))}) at .\deprecated.jl:32
│     [4] parse_7z_list(::String) at C:\Users\bernhard.konig\.julia\v0.7\BinaryProvider\src\PlatformEngines.jl:366
│     [5] #list_tarball_files#119(::Bool, ::Function, ::String) at C:\Users\bernhard.konig\.julia\v0.7\BinaryProvider\src\Prefix.jl:446
│     [6] list_tarball_files at C:\Users\bernhard.konig\.julia\v0.7\BinaryProvider\src\Prefix.jl:433 [inlined]
│     [7] #install#109(::BinaryProvider.Prefix, ::Bool, ::Bool, ::Bool, ::Function, ::String, ::String) at C:\Users\bernhard.konig\.julia\v0.7\BinaryProvider\src\Prefix.jl:302
│     [8] (::getfield(BinaryProvider, Symbol("#kw##install")))(::NamedTuple{(:prefix, :force, :verbose),Tuple{BinaryProvider.Prefix,Bool,Bool}}, ::typeof(BinaryProvider.install), ::String, ::String) at .\<missing>:0
│     [9] top-level scope at C:\Users\bernhard.konig\.julia\v0.7\CodecZlib\deps\build.jl:92
│     [10] include(::Module, ::String) at .\boot.jl:306
│     [11] include_relative(::Module, ::String) at .\loading.jl:1067
│     [12] include at .\sysimg.jl:29 [inlined]
│     [13] include(::String) at .\loading.jl:1101
│     [14] top-level scope
│     [15] eval at .\boot.jl:309 [inlined]
│     [16] eval at .\sysimg.jl:74 [inlined]
│     [17] evalfile(::String, ::Array{String,1}) at .\loading.jl:1096 (repeats 2 times)
│     [18] #6 at .\none:13 [inlined]
│     [19] cd(::getfield(Main, Symbol("##6#8")){String}, ::String) at .\file.jl:61
│     [20] (::getfield(Main, Symbol("##5#7")))(::IOStream) at .\none:12
│     [21] #open#321(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(Main, Symbol("##5#7")), ::String, ::Vararg{String,N} where N) at .\iostream.jl:369
│     [22] open(::Function, ::String, ::String) at .\iostream.jl:367
│     [23] top-level scope
│     [24] eval at .\boot.jl:309 [inlined]
│     [25] eval(::Module, ::Expr) at .\sysimg.jl:74
│     [26] exec_options(::Base.JLOptions) at .\client.jl:304
│     [27] _start() at .\client.jl:455
│    in expression starting at C:\Users\bernhard.konig\.julia\v0.7\CodecZlib\deps\build.jl:92
└ @ Main none:16
[ Info: Building SpecialFunctions
WARNING: importing deprecated binding Base.Associative into BinDeps.
WARNING: Base.Associative is deprecated, use AbstractDict instead.
  likely near C:\Users\bernhard.konig\.julia\v0.7\SpecialFunctions\deps\binaries.jl:44
in @install at C:\Users\bernhard.konig\.julia\v0.7\BinDeps\src\dependencies.jl
┌ Warning: ------------------------------------------------------------
│ # Build error summary
│
│ CodecZlib had build errors.
│
│  - packages with build errors remain installed in C:\Users\bernhard.konig\.julia\v0.7
│  - build the package(s) and all dependencies with `Pkg.build("CodecZlib")`
│  - build a single package by running its `deps/build.jl` script
└ @ Pkg.Entry entry.jl:649
Julia Version 0.7.0-DEV.4690
Commit 78c7d87369* (2018-03-23 22:25 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, skylake)
Environment:
  JULIA_BINDIR = C:\Julia-0.7.X\bin\
  JULIA_HOME = C:\Julia-0.6.X\bin\

julia>

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.