juliaio / jld.jl Goto Github PK
View Code? Open in Web Editor NEWSaving and loading julia variables while preserving native types
License: MIT License
Saving and loading julia variables while preserving native types
License: MIT License
I encountered a weird error while writing some julia code for my project and distilled the essence of it as the code below. What it does is basically to read data from a HDF5 file in a separate Task thread and save calculated results to another JLD file.
using JLD
using HDF5
h5open("a.hdf5", "w") do file
a = reshape([1],1,1,1,1)
file["data"] = a
end
function io_task_impl()
while true
file = h5open("a.hdf5", "r")
produce((1,1,file["data"]))
close(file)
produce((1,0,nothing))
end
end
io_task = Task(io_task_impl)
while true
jldopen("aaa.jld", "w") do file
write(file, "arglist", (Float64, Vector([28*28, 300, 10]), 100))
end
d = consume(io_task)
end
The code above produces the following error after a few loops:
HDF5-DIAG: Error detected in HDF5 (1.8.14) thread 0:
#000: H5Dio.c line 271 in H5Dwrite(): can't prepare for writing data
major: Dataset
minor: Write failed
#001: H5Dio.c line 352 in H5D__pre_write(): can't write data
major: Dataset
minor: Write failed
#002: H5Dio.c line 788 in H5D__write(): can't write data
major: Dataset
minor: Write failed
#003: H5Dcontig.c line 580 in H5D__contig_write(): contiguous write failed
major: Dataset
minor: Write failed
#004: H5Dscatgath.c line 678 in H5D__scatgath_write(): datatype conversion failed
major: Dataset
minor: Can't convert datatypes
#005: H5T.c line 4816 in H5T_convert(): data type conversion failed
major: Attribute
minor: Unable to encode value
#006: H5Tconv.c line 2571 in H5T__conv_struct_opt(): unable to convert compound datatype member
major: Datatype
minor: Unable to initialize object
#007: H5T.c line 4816 in H5T_convert(): data type conversion failed
major: Attribute
minor: Unable to encode value
#008: H5Tconv.c line 2172 in H5T__conv_struct(): not a datatype
major: Datatype
minor: Inappropriate type
ERROR: Error writing dataset
in h5d_write at /Users/gloine/.julia/v0.5/HDF5/src/plain.jl:1928
[inlined code] from /Users/gloine/.julia/v0.5/HDF5/src/plain.jl:1803
in write_compound at /Users/gloine/.julia/v0.5/JLD/src/JLD.jl:699
in write at /Users/gloine/.julia/v0.5/JLD/src/JLD.jl:687
in write at /Users/gloine/.julia/v0.5/JLD/src/JLD.jl:509
in anonymous at none:3
in jldopen at /Users/gloine/.julia/v0.5/JLD/src/JLD.jl:245
[inlined code] from /Users/gloine/.julia/v0.5/JLD/src/JLD.jl:243
in anonymous at no file:0
in eval at /Applications/Julia-0.5.0-dev-b0a84f7a3b.app/Contents/Resources/julia/lib/julia/sys.dylib
It sometimes segfaults, and sometimes gives me the error above depending on the code I insert in between. The code writes the same Tuple item to the JLD file every time, so it is weird to have a random failure.
I am using the latest master branch (could be several commits behind) on Mac OS X El Capitan. I read that HDF5 is not thread safe by default. Would installing a thread safe version of HDF5 solve the problem above?
Thanks,
Gloine
I have a data set data
which has a lot of redundancy. It is represented as 10x1 Vector{Matrix{Float32}}
, where each element is approx. 150x5*10^6 Matrix{Float32}
.
I can save it on my drive via save("data.jld", "data", data)
with no problem. It takes approx. 30 GB of space and 10 min to complete the writing.
When I transform it to redundancy free representation as 130 entry Dict{Int32, Vector{Float32}}
(call it dic
) and 5*10^7x1 Vector{Vector{Int32}}
(call it arr
), where each element is 20-25x1 Vector{Int32}
I am able to save the dic
via save("dic.jld", "dic", dic)
. However when I do the same for arr
: save("arr.jld", "arr", arr)
it never finishes writing. This line has been running for 2 days and it is still not finished writing. This makes no sense to me because much larger data
gets written in 10 mins.
I'm getting a lot of warnings from Julia 0.4.0-rc3 for calls to JLD.jl.
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 21
WARNING: Base.Nothing is deprecated, use Void instead.
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in eval at /home/travis/.julia/v0.4/JLD/src/JLD.jl:7
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/jld_types.jl, in expression starting on line 123
WARNING: Base.Nothing is deprecated, use Void instead.
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/jld_types.jl, in expression starting on line 168
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/jld_types.jl, in expression starting on line 174
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/jld_types.jl, in expression starting on line 232
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/jld_types.jl, in expression starting on line 236
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/jld_types.jl, in expression starting on line 244
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/jld_types.jl, in expression starting on line 296
WARNING: Base.UnionType is deprecated, use Union instead.
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 102
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 133
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 235
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 238
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 245
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 246
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 254
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 255
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 256
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 257
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 258
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 277
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 281
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 284
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 286
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 287
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 288
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 296
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 306
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 360
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 454
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 454
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 454
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 454
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 454
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 454
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 474
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 474
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 474
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 474
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 474
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 474
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 474
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 474
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 474
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 494
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 494
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 494
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 522
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 522
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 522
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 522
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 522
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 522
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 605
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 605
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 605
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 622
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 622
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 622
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 625
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 625
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 625
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 655
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 670
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 703
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 704
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 706
WARNING: Base.String is deprecated, use AbstractString instead.
WARNING: Base.String is deprecated, use AbstractString instead.
WARNING: Base.String is deprecated, use AbstractString instead.
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 877
WARNING: Base.UnionType is deprecated, use Union instead.
WARNING: Union(args...) is deprecated, use Union{args...} instead.
in depwarn at deprecated.jl:73
in call at deprecated.jl:50
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
[inlined code] from none:2
in anonymous at no file:0
in process_options at ./client.jl:284
in _start at ./client.jl:411
while loading /home/travis/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 984
Not entirely sure what's going on, but JLD.save
doesn't like saving this particular type, which contains many Nullable
fields:
julia> using JLD, GitHub; JLD.save("test.jld", "g", GitHub.Owner("jiahao"))
ERROR: UndefRefError: access to undefined reference
in h5convert! at /Users/jiahao/.julia/v0.4/JLD/src/jld_types.jl:627
in write_compound at /Users/jiahao/.julia/v0.4/JLD/src/JLD.jl:693
in write_ref at /Users/jiahao/.julia/v0.4/JLD/src/JLD.jl:651
[inlined code] from /Users/jiahao/.julia/v0.4/JLD/src/jld_types.jl:615
in h5convert! at /Users/jiahao/.julia/v0.4/JLD/src/jld_types.jl:627
in write_compound at /Users/jiahao/.julia/v0.4/JLD/src/JLD.jl:693
in write at /Users/jiahao/.julia/v0.4/JLD/src/JLD.jl:509
in anonymous at /Users/jiahao/.julia/v0.4/JLD/src/JLD.jl:1175
in jldopen at /Users/jiahao/.julia/v0.4/JLD/src/JLD.jl:245
in save at /Users/jiahao/.julia/v0.4/JLD/src/JLD.jl:1173
in save at /Users/jiahao/.julia/v0.4/FileIO/src/loadsave.jl:51
jldTimings.zip
When loading an array with DateTimes with the load() function the loading times
increase over time. The same does not happen when loading an array of floats.
Attached are a julia script and two data files to reproduce this behavior.
Run the script with julia timeJLD.jl N
where N is the number of iterations.
myDates.jld has the array with datetimes date
myArray.jld has the array with floats yy
I ran with N = 5k to 10k.
using HDF5
using JLD
using PyPlot
function myLoop(N, timings)
for i = 1:N
timings[i] = @elapsed tt = load(fileName, "date")
#timings[i] = @elapsed tt = load(fileName, "yy")
end
end
N = parse(Int, ARGS[1])
fileName = "myDates.jld"
#fileName = "myArray.jld"
timings = Array(Float64, N)
myLoop(N, timings)
figure()
semilogy(timings)
show()
In real life I load the content from different files of course...
Cheers
Andre
i am not sure it is the problem of my ubuntu system or the package, so i'll fire straight a issue here.
it used to work on another windows system to save file to our shared file server, but when i setup a ubuntu system today, save just failed. the network shared folder is mounted properly, since i could browse the content, create and delete files. i checked out teh/jld_split branch of HDF5, problem remains.
using JLD
save("/share folder mount path/test.jld", "a",[1,2,3,4])
HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 139963186603840:
#000: ../../../src/H5F.c line 1516 in H5Fcreate(): unable to create file
major: File accessibilty
minor: Unable to open file
#001: ../../../src/H5F.c line 1305 in H5F_open(): unable to open file: time = Tue Jul 21 17:14:46 2015
, name = '/run/user/1000/gvfs/smb-share:server=rvdh05,share=d/exp/Remapping/analysis/alex/test.jld', tent_flags = 13
major: File accessibilty
minor: Unable to open file
#002: ../../../src/H5FD.c line 985 in H5FD_open(): open failed
major: Virtual File Layer
minor: Unable to initialize object
#003: ../../../src/H5FDsec2.c line 343 in H5FD_sec2_open(): unable to open file: name = '/run/user/1000/gvfs/smb-share:server=rvdh05,share=d/exp/Remapping/analysis/alex/test.jld', errno = 2, error message = 'No such file or directory', flags = 13, o_flags = 242
major: File accessibilty
minor: Unable to open file
Error creating file /run/user/1000/gvfs/smb-share:server=rvdh05,share=d/exp/Remapping/analysis/alex/test.jld
while loading In[3], in expression starting on line 2
in h5f_create at /home/alex/.julia/v0.3/HDF5/src/plain.jl:2021
in jldopen at /home/alex/.julia/v0.3/JLD/src/JLD.jl:167
in jldopen at /home/alex/.julia/v0.3/JLD/src/JLD.jl:219
in jldopen at /home/alex/.julia/v0.3/JLD/src/JLD.jl:229
in save at /home/alex/.julia/v0.3/JLD/src/JLD.jl:1049
instead, save("/home/alex/test.jld", "a",[1,2,3,4]) works well.
Hello,
I noticed in doc
t = 15
z = [1,3]
save("/tmp/myfile.jld", "t", t, "arr", z)
maybe API should be improved to avoid user to give a name ("t" or "arr")
Passing a symbol could probably be enough.
So we could do
save("myfile.jld", :t)
or
save("myfile.jld", :t, :z)
What is your opinion ?
Kind regards
Hi,
It seems there is a way to break type saving/loading,
if you do multiple writes of a suitably complex type:
The MWE I could create is:
using JLD
using HDF5
jldopen("example.jld", "w") do fh
write(fh, "weights", [(:Heavy,100), (:Light,50)])
end
jldopen("example.jld", "r+") do fh
write(fh, "otherweights", [(:VeryHeavy, 200), (:VeryLight, 25)])
end
load("example.jld")
when you try to do the load you get:
LoadError: stored type Core.Tuple{Core.Symbol,Core.Int64} does not match currently loaded type
while loading In[5], in expression starting on line 9
in jldatatype at /home/ubuntu/.julia/v0.5/JLD/src/jld_types.jl:689
in read at /home/ubuntu/.julia/v0.5/JLD/src/JLD.jl:370
in read_ref at /home/ubuntu/.julia/v0.5/JLD/src/JLD.jl:498
in read_refs at /home/ubuntu/.julia/v0.5/JLD/src/JLD.jl:471
in read_array at /home/ubuntu/.julia/v0.5/JLD/src/JLD.jl:407
in read at /home/ubuntu/.julia/v0.5/JLD/src/JLD.jl:372
in read at /home/ubuntu/.julia/v0.5/JLD/src/JLD.jl:346
in anonymous at /home/ubuntu/.julia/v0.5/JLD/src/JLD.jl:1185
in jldopen at /home/ubuntu/.julia/v0.5/JLD/src/JLD.jl:245
[inlined code] from /home/ubuntu/.julia/v0.5/JLD/src/JLD.jl:243
in load at /home/ubuntu/.julia/v0.5/JLD/src/JLD.jl:1184
in load at /home/ubuntu/.julia/v0.5/FileIO/src/loadsave.jl:42
[inlined code] from essentials.jl:114
however both load("example.jld","weights")
and load("example.jld","otherweights")
work.
It also works fine if they are both written at the same time.
A few more examples of things that work leading up to it
are on https://gist.github.com/oxinabox/95f71548345e8fc583
This is with
Julia Version 0.5.0-dev+1331
Commit 9d6dee5 (2015-11-18 04:38 UTC)
Platform Info:
System: Linux (x86_64-linux-gnu)
CPU: AMD Opteron 63xx class CPU
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Piledriver)
LAPACK: libopenblas64_
LIBM: libopenlibm
LLVM: libLLVM-3.3JLD 0.5.7
HDF5 0.5.7
https://travis-ci.org/JuliaLang/JLD.jl/builds/80684396 suggests JLD may be segfaulting on nightly. Look into this.
I accidentally called save(File(format"JLD","/tmp/foo"),A)
instead of save(File(format"JLD","/tmp/foo"),"A",A)
, and this resulted in Julia exiting with a segmentation fault.
The actual error log is too long to post, but it involved the following lines repeated consecutively (many times):
jl_apply at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/./julia.h:1325
jl_apply at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/./julia.h:1325
save at /Users/Ty/.julia/v0.4/FileIO/src/loadsave.jl:92
julia_save_21900 at (unknown line)
and ended with,
julia_save_21900 at (unknown line)
jl_apply at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/gf.c:1691
jl_apply at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/interpreter.c:55
eval at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/interpreter.c:213
jl_toplevel_eval_flex at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/toplevel.c:527
jl_toplevel_eval_in at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/builtins.c:579
eval_user_input at REPL.jl:62
jlcall_eval_user_input_21372 at (unknown line)
jl_apply at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/./julia.h:1325
anonymous at REPL.jl:92
jl_apply at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/task.c:241
Segmentation fault: 11
I was wondering if this could/should result in a MethodError
instead?
using JLD
a = rand(10)
save("a.jld", "a", a)
workspace()
using JLD
using Base.Test
a = rand(10)
b = rand(10)
c = rand(10)
@save "thing.jld"
@test contains(readall(`ls`), "thing.jld")
running julia test.jl
gives the following error at the end:
julia(68545,0x7fff7dc61300) malloc: *** error for object 0x7fdea3c18000: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
signal (6): Abort trap: 6
__pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line)
Abort trap: 6
Actually the file works over the REPL and otherwise, but once you exit Julia, that error is returned.
It would be really nice if JLD could support Saving into an IOBuffer.
FileIO suggests that the code bellow would be how it is done.
However that code just hangs
using FileIO
using JLD
ss = Stream(format"JLD",IOBuffer())
JLD.save(ss, Dict("ii"=> 55555))
my particular use case is that I would like to save my data directly into a OpenStack Swift Object storage, without writing it to disk.
because right now the work around I am looking at is
1 . in memory data
2. Write JLD to disk
3. Read JLD file into IOBuffer()
/Vector{UInt8}
4. Upload to Object storage
where as it could be:
1 . in memory data
2. Write as JLD format into `IOBuffer()
3. Upload to Object storage
This cuts out the Reading and Writing from disk.
Which is a big speed-up for multi-gigabyte files.
Failing adding support for writing to Streams,
it would be nice if rather than handing it threw a method error.
This is with:
Julia Version 0.5.0-rc0+150
Commit 389dc1c (2016-08-03 04:22 UTC)
Platform Info:
System: Linux (x86_64-linux-gnu)
CPU: AMD Opteron 63xx class CPU
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Piledriver)
LAPACK: libopenblas64_
LIBM: libopenlibm
LLVM: libLLVM-3.7.1 (ORCJIT, bdver2)
I have encountered a situation in which i have a array of anonymous functions that i want to save for later use.
but for now, the JLD complained that it cannot write a pointer to JLD file
.
I am curious that is it possible to do this, may be by save the function as an expression in some form of string and re-eval when read into current work space.
any thoughts?
All the documentations gives examples of usine @write file A
etc.
And says that it is no longer generally required to first import HDF5.
But using @write
without fist using HDF5
will throw an error as the macro is not defined.
Should be an easy fix to add @write
to the things being imported. (Next to the function write
)
Hi!
This might be due to the delayed Ubuntu builds for Julia Nightlies.
I get a start
method error for this:
using JLD
save("a.jld", "a", Dict("a" => 1))
The error is:
ERROR: MethodError: `start` has no method matching start(::Type{Tuple{ASCIIString,Int64}})
My specs:
Version 0.4.0-dev+6095 (2015-07-20 14:38 UTC)
Commit c75f4d4* (27 days old master)
x86_64-linux-gnu
This error occurs in Julia 0.4.5 (released version 2016-03-18), JLD version 0.6.0, Windows 8.1; the following statement crashes Julia (i.e., the REPL abruptly exits):
maprelabs = JLD.load("maprelabs.jld", "maprelabs")
The file maprelabs.jld was produced with a JLD.save invocation and is supposed to contain two sparse matrices. I have posted it on my webpage so that you can download it:
This snippets shows how a LineNumberedNode
is lost when loading the object back again.
Reference: https://gitter.im/JuliaLang/julia?at=570e50175cd40114649ae521
julia> using JLD
julia> type T
e::Expr
a::Array{Any,1}
end
julia> ex = :(function f() return true end)
:(function f() # none, line 1:
return true
end)
julia> t = T(ex, ex.args[2].args)
T(:(function f() # none, line 1:
return true
end),Any[:( # none, line 1:),:(return true)])
julia> save("file.jld", "t", t)
julia> t1 = load("file.jld")["t"]
T(:(function f()
return true
end),Any[:( # none, line 1:),:(return true)])
julia> t1.e.args[2].args === t1.a # should be true?
false
julia> t1.e.args[2].args == t1.a # should be true!
false
julia> VERSION
v"0.4.5"
julia> Pkg.status("JLD")
- JLD 0.5.9
Notice that deserialize
does keep the node correctly:
julia> type T
e::Expr
a::Array{Any,1}
end
julia> ex = :(function f() return true end)
:(function f() # none, line 1:
return true
end)
julia> t = T(ex, ex.args[2].args) # a refers to function's block
T(:(function f() # none, line 1:
return true
end),Any[:( # none, line 1:),:(return true)])
julia> io = open("file.dat", "w")
IOStream(<file file.dat>)
julia> serialize(io, t)
julia> close(io)
julia> io = open("file.dat")
IOStream(<file file.dat>)
julia> t1 = deserialize(io)
T(:(function f() # none, line 1:
return true
end),Any[:( # none, line 1:),:(return true)])
julia> t1.e.args[2].args === t1.a # should be true?
false
julia> t1.e.args[2].args == t1.a # should be true!
true
julia> close(io)
We are running a program to parse many ascii files. We want to identify the keywords of such ascii files and to compare the keywords of some files with the keywords of other files. We are using JLD.jl to write the intermediate results into .jld files.
We use to compare hundreds of files among each other. Sometimes the program crashes and returns the following error message:
HDF5-DIAG: Error detected in HDF5 (1.8.11) thread 139869298599808:
#000: ../../../src/H5O.c line 1059 in H5Oclose(): not a valid object
major: Invalid arguments to routine
minor: Bad value
ERROR: LoadError: Error closing object
in h5o_close at /home/cdesantana/.julia/v0.4/HDF5/src/plain.jl:1927
in close at /home/cdesantana/.julia/v0.4/JLD/src/JLD.jl:122
in jldopen at /home/cdesantana/.julia/v0.4/JLD/src/JLD.jl:242
in load at /home/cdesantana/.julia/v0.4/JLD/src/JLD.jl:1180
in load at /home/cdesantana/.julia/v0.4/FileIO/src/loadsave.jl:42
in load_pairs_repeats at /home/cdesantana/Data/where2publish/src/creategraph.jl:120
in text_graph at /home/cdesantana/Data/where2publish/src/creategraph.jl:145
in anonymous at /home/cdesantana/Data/where2publish/src/index.jl:56
in open at iostream.jl:114
in compare_manyabstracts! at /home/cdesantana/Data/where2publish/src/index.jl:45
in main at /home/cdesantana/Data/where2publish/src/index.jl:78
in include at ./boot.jl:261
in include_from_node1 at ./loading.jl:304
in process_options at ./client.jl:308
in _start at ./client.jl:411
The line of our code where the program crashes is the following one:
pairs = load(file_name, "pairs")
Something curious is that after the program crashes we are able to run it again starting from the file for which it crashed. So I suppose this error may be related to an overload of the JLD.jl functions we use or something like that.
Any idea?
Not sure how we should handle this.
using JLD
JLD.readsafely
returns a ERROR: UndefVarError: readsafely not defined
.
This happens for me on Julia 0.4.6
with the latest JLD
release as well as master
.
function save{F}(q::Formatted{F}, data...; options...)
unknown(q) && throw(UnknownFormat(q))
libraries = applicable_savers(q)
failures = Any[]
for library in libraries
try
Library = checked_import(library)
return Library.save(q, data...; options...)
catch e
handle_current_error(e, library, library == last(libraries))
push!(failures, (e, q))
end
end
handle_error(failures)
end
When you execute the below code, an endless loop comes because Library
is JLD
.
using JLD
using FileIO
a = 1
save(query("test.jld"), a)
julia> versioninfo()
Julia Version 0.5.0-dev+1516
Commit a7b41f5 (2015-11-30 00:08 UTC)
Platform Info:
System: Linux (x86_64-linux-gnu)
CPU: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
LAPACK: libopenblas64_
LIBM: libopenlibm
LLVM: libLLVM-3.3
Hi,
Following an update a few days ago I am getting this error:
ERROR: unexpected non-leaf type Distributions.Normal{T<:Real}
in h5type at /Users/alancrawford/.julia/v0.4/JLD/src/jld_types.jl:391
in jldatatype at /Users/alancrawford/.julia/v0.4/JLD/src/jld_types.jl:698
in jldatatype at /Users/alancrawford/.julia/v0.4/JLD/src/jld_types.jl:689
in read at /Users/alancrawford/.julia/v0.4/JLD/src/JLD.jl:381
in read_ref at /Users/alancrawford/.julia/v0.4/JLD/src/JLD.jl:509
[inlined code] from /Users/alancrawford/.julia/v0.4/JLD/src/jld_types.jl:430
in jlconvert at /Users/alancrawford/.julia/v0.4/JLD/src/jld_types.jl:438
in read_scalar at /Users/alancrawford/.julia/v0.4/JLD/src/JLD.jl:409
in read at /Users/alancrawford/.julia/v0.4/JLD/src/JLD.jl:381
in read_ref at /Users/alancrawford/.julia/v0.4/JLD/src/JLD.jl:509
[inlined code] from /Users/alancrawford/.julia/v0.4/JLD/src/jld_types.jl:430
in jlconvert at /Users/alancrawford/.julia/v0.4/JLD/src/jld_types.jl:438
in read_scalar at /Users/alancrawford/.julia/v0.4/JLD/src/JLD.jl:409
in read at /Users/alancrawford/.julia/v0.4/JLD/src/JLD.jl:381
in read_ref at /Users/alancrawford/.julia/v0.4/JLD/src/JLD.jl:509
[inlined code] from /Users/alancrawford/.julia/v0.4/JLD/src/jld_types.jl:430
in jlconvert at /Users/alancrawford/.julia/v0.4/JLD/src/jld_types.jl:438
in read_scalar at /Users/alancrawford/.julia/v0.4/JLD/src/JLD.jl:409
in read at /Users/alancrawford/.julia/v0.4/JLD/src/JLD.jl:381
in read_ref at /Users/alancrawford/.julia/v0.4/JLD/src/JLD.jl:509
[inlined code] from /Users/alancrawford/.julia/v0.4/JLD/src/jld_types.jl:430
in jlconvert at /Users/alancrawford/.julia/v0.4/JLD/src/jld_types.jl:438
in read_scalar at /Users/alancrawford/.julia/v0.4/JLD/src/JLD.jl:409
in read at /Users/alancrawford/.julia/v0.4/JLD/src/JLD.jl:381
in read at /Users/alancrawford/.julia/v0.4/JLD/src/JLD.jl:357
in read at /Users/alancrawford/.julia/v0.4/HDF5/src/datafile.jl:31
in anonymous at none:8
in jldopen at /Users/alancrawford/.julia/v0.4/JLD/src/JLD.jl:256
[inlined code] from none:7
in anonymous at no file:0
This piece of code is unchanged and was working prior to the update.
Any help resolving the issue appreciated.
Thanks
Alan
this might be related to issues/158
I'm currently working on simulation program and using JLD for logging.
but while analysing log data, I found strange behavior with JLD. That same function "load" and "read" does not work as same function.
here is my example code
using JLD
type Item
uid::UInt32
id::Int32
count::Int32
let counter = UInt32(0)
Item(id, count) = new(counter += 1, id, count)
end
end
# create dummy Data
items = Item[]
for i in 1:10, j in 1:5
push!(items, Item(i, j))
end
jldopen("test.jld", "w") do file
write(file, "log1", Dict(:date => DateTime(1,1,1), :item=>items))
end
for i in 1:10
items[i].count += rand(1:100)
end
jldopen("test.jld", "r+") do file
write(file, "log2", Dict(:date => DateTime(1,1,2), :item=>items))
end
This works fine
println(load("test.jld", "log1"))`
println(load("test.jld", "log2"))
But this does not work!
pops up "stored type Base.Dates... does not match currently loaded type" error which is not even my custom type.
jldopen("test.jld", "r") do file
println(read(file, "log1"))
println(read(file, "log2"))
end
Getting an error after heavy writing.
I am running this on a Google Compute instance.
jldopen(filepath, "w") do file
group = JLD.g_create(file, "keyMap")
for (key, data) in keyMap
# Convert to sparse map for space efficiency.
group[key] = sparse(data)
end
end
where,
keyMap::Dict{AbstractString, Array{Float64}}
*** Error in `julia': corrupted double-linked list: 0x00000000081b76e0 ***
======= Backtrace: =========
/usr/bin/../lib64/libc.so.6(+0x7b184)[0x7fd97ec5d184]
/usr/bin/../lib64/libc.so.6(+0x7e8f6)[0x7fd97ec608f6]
/usr/bin/../lib64/libc.so.6(__libc_malloc+0x4c)[0x7fd97ec6187c]
/lib64/libhdf5.so.8(+0x100e4c)[0x7fd76502de4c]
/lib64/libhdf5.so.8(H5FL_reg_malloc+0x19f)[0x7fd76502e370]
/lib64/libhdf5.so.8(H5FL_reg_calloc+0xb1)[0x7fd76502e4b0]
/lib64/libhdf5.so.8(H5O_create+0x2b8)[0x7fd7650be0f7]
/lib64/libhdf5.so.8(+0xaccdb)[0x7fd764fd9cdb]
/lib64/libhdf5.so.8(H5D__create+0xcf2)[0x7fd764fdafda]
/lib64/libhdf5.so.8(+0xb73c3)[0x7fd764fe43c3]
/lib64/libhdf5.so.8(H5O_obj_create+0x123)[0x7fd7650c36e9]
/lib64/libhdf5.so.8(+0x180026)[0x7fd7650ad026]
/lib64/libhdf5.so.8(+0x139e3f)[0x7fd765066e3f]
/lib64/libhdf5.so.8(H5G_traverse+0x2e7)[0x7fd765067e83]
/lib64/libhdf5.so.8(+0x180b7d)[0x7fd7650adb7d]
/lib64/libhdf5.so.8(H5L_link_object+0x93)[0x7fd7650ace4b]
/lib64/libhdf5.so.8(H5D__create_named+0xb3)[0x7fd764fd8b74]
/lib64/libhdf5.so.8(H5Dcreate2+0x4f4)[0x7fd764fb7292]
[0x7fd9822473b6]
======= Memory map: ========
00400000-00405000 r-xp 00000000 08:01 9673907 /usr/bin/julia
00604000-00605000 r--p 00004000 08:01 9673907 /usr/bin/julia
00605000-00606000 rw-p 00005000 08:01 9673907 /usr/bin/julia
01860000-0c830000 rw-p 00000000 00:00 0 [heap]
11460000-50400000 rw-p 00000000 00:00 0
7fd760000000-7fd760021000 rw-p 00000000 00:00 0
7fd760021000-7fd764000000 ---p 00000000 00:00 0
7fd764b03000-7fd764b28000 r-xp 00000000 08:01 23026289 /usr/lib64/libopenlibm.so.1.0
7fd764b28000-7fd764d27000 ---p 00025000 08:01 23026289 /usr/lib64/libopenlibm.so.1.0
7fd764d27000-7fd764d28000 r--p 00024000 08:01 23026289 /usr/lib64/libopenlibm.so.1.0
7fd764d28000-7fd764d29000 rw-p 00025000 08:01 23026289 /usr/lib64/libopenlibm.so.1.0
Julia Version 0.4.3 (2016-01-12 21:37 UTC)
x86_64-redhat-linux
Been mostly running Julia on a Mac. Switched to a more powerful (Windows) machine and while setting everything up and loading files with JLD received this error. Will attempt to debug myself, but submitting here just in case:
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ffaf5d00b7d -- RtlFreeHeap at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
RtlFreeHeap at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
RtlFreeHeap at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
free at C:\Windows\system32\msvcrt.dll (unknown line)
h5t_get_member_name at C:\Users\SERG.julia\v0.4\HDF5\src\plain.jl:2167
jldatatype at C:\Users\SERG.julia\v0.4\JLD\src\jld_types.jl:685
read at C:\Users\SERG.julia\v0.4\JLD\src\JLD.jl:381
jl_apply_generic at C:\Users\SERG\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
read at C:\Users\SERG.julia\v0.4\JLD\src\JLD.jl:357
anonymous at no file:0
jl_eval_with_compiler_p at C:\Users\SERG\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_parse_eval_all at C:\Users\SERG\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_load_file_string at C:\Users\SERG\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
include_string at C:\Users\SERG.julia\v0.4\CodeTools\src\eval.jl:28
jlcall_include_string_1697 at (unknown line)
jl_apply_generic at C:\Users\SERG\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_interpret_toplevel_expr at C:\Users\SERG\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_interpret_toplevel_thunk_with at C:\Users\SERG\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_eval_with_compiler_p at C:\Users\SERG\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_f_tuple at C:\Users\SERG\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
include_string at C:\Users\SERG.julia\v0.4\CodeTools\src\eval.jl:32
jl_apply_generic at C:\Users\SERG\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
anonymous at C:\Users\SERG.julia\v0.4\Atom\src\eval.jl:39
withpath at C:\Users\SERG.julia\v0.4\Requires\src\require.jl:37
jl_apply_generic at C:\Users\SERG\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
withpath at C:\Users\SERG.julia\v0.4\Atom\src\eval.jl:53
jl_apply_generic at C:\Users\SERG\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
anonymous at C:\Users\SERG.julia\v0.4\Atom\src\eval.jl:61
jl_unprotect_stack at C:\Users\SERG\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
From @timholy on December 23, 2014 13:39
I needed to create a custom format for a type I'm working on, so I thought while learning how to do this I'd create a demo for the benefit of others. I suspect the main issue worth discussing is whether this is the best approach, or would it be better/simpler to create custom read
and write
methods? CC @simonster.
Once this settles, I'll also add this material to the documentation.
Copied from original issue: JuliaIO/HDF5.jl/pull/191
I want to use this module but I dont have the option to have/install HDF5. It seems that by default HDF5 is installed with this module and depends on it. A past commit states that HDF5 is not needed.
Is it just a matter of using the HDF5 module but not necessary?
Is there a clean solution to avoid downloading and building HDF5 dependancies?
Consider
jld = jldopen("test.jld","w")
jld["a"]=1
has(jld,"a") # ERROR: MethodError: `has` has no method ...
has(jld.plain,"a")
Is there a reason this doesn't work?
The documentation says:
Int128/Uint128: presumably similar to Complex128 (encode as pair of Uint64). The holdup: is the sign bit portable?
HDF5 allows defining arbitrary integer datatypes by specifying their number of bits. This would seem to be the most natural approach.
Otherwise, a similarly "natural" approach would be to manually decompose them into their lower and upper 64 bits, and storing these as Uint64
and Int64
, respectively. (Uint128
is then stored as two UInt64
.) This is portable; there is no computing architecture today that deviates from the signed two's complement representation.
This issue has been reported initially at JuliaLang/julia#17554. Discussion can continue here.
So I found this useful repository that allows you to compute an Simon Funk SVD in Julia. As such, I created a massive one using 170 million user rankings and it worked beautifully taking about three hours to converge. To save times in future runs, I thought it would be a great idea export the "model" object yielded from the train function to the JLD format. So I did. An hour and a half ago... So far JLD has allocated an approximately 800MB file to save it in but has only saved 750MB and it appears to be slowing down... and taking longer and longer to fully fill the file. Also, my resource monitor (Windows 10) is reporting 300MB/s of SSD access from Julia despite the fact that if that were the case the file would have been written in 3 seconds. I am little confused about why the performance is so bad. I mean I suffered slow performance when reading the CSVs into from my dataset, but WOW. That was 5GBs of data and it took 10 minutes. As such, it take significantly under an hour to save 800MB. Any suggestions on what could be causing the increduously bad performance? I just installed Julia today, running the latest version on Windows 10 64bit.
using DataStructures, JLD
d = DefaultDict(Int, Int, () -> 0)
d[1] = 3
@save "test.jld" d
@load "test.jld"
gives me
`DefaultDict{Int64,Int64,Function}` has no method matching DefaultDict{Int64,Int64,Function}()
while loading In[1], in expression starting on line 5
which is expected in the retrospect, but hopefully we can do better.
Hi there,
just ran into this:
(gdb) r
Starting program: /home/s/julia/usr/bin/julia-debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Traceback (most recent call last):
File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19-gdb.py", line 63, in <module>
from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named 'libstdcxx'
[New Thread 0x7ffdefe94700 (LWP 2388)]
[New Thread 0x7ffdef693700 (LWP 2389)]
[New Thread 0x7ffdece92700 (LWP 2390)]
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_) | Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type "?help" for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.4.0-pre+7178 (2015-09-04 16:16 UTC)
_/ |\__'_|_|_|\__'_| | Commit cd87abb (0 days old master)
|__/ | x86_64-linux-gnu
julia> include(Pkg.dir("JLD", "test", "runtests.jl"))
jldtests.jl
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff6da8808 in jl_method_table_assoc_exact (mt=0x7ffdf2bf9730, args=0x7fffffff5d10, n=2) at gf.c:254
254 assert(jl_is_datatype(ty));
(gdb) bt
#0 0x00007ffff6da8808 in jl_method_table_assoc_exact (mt=0x7ffdf2bf9730, args=0x7fffffff5d10, n=2) at gf.c:254
#1 0x00007ffff6dadbb2 in jl_apply_generic (F=0x7ffdf2c2bb50, args=0x7fffffff5d10, nargs=2) at gf.c:1655
#2 0x00007ffdf1c455b3 in ?? ()
#3 0x0000000000000004 in ?? ()
#4 0x00007fffffff5ed0 in ?? ()
#5 0x0000000002a7b5d0 in ?? ()
#6 0x00007ffdf43dea40 in ?? ()
#7 0x72f87b53a0194900 in ?? ()
#8 0x00007ffdf4bede78 in ?? ()
#9 0x00007ffff7cffd88 in jl_true () from /home/s/julia/usr/bin/../lib/libjulia-debug.so
#10 0x00007fffffffbc40 in ?? ()
#11 0x00007ffff6dadb32 in jl_compile_hint (f=0x7ffdf4bede78, types=0x72f87b53a0194900) at gf.c:1617
#12 0x00007fffffffbc30 in ?? ()
#13 0x00007fffffff5d70 in ?? ()
#14 0x00007ffdf1c454f1 in ?? ()
#15 0x00007ffdf7754dd0 in ?? ()
#16 0x72f87b53a0194900 in ?? ()
#17 0x00007fffffff5d80 in ?? ()
#18 0x00007ffdf1c454a7 in ?? ()
#19 0x00007fffffff5dd0 in ?? ()
#20 0x00007ffff6db4aee in jl_apply (f=0x48e8458b48e0558b, args=0x48c031f845894800, nargs=10277) at julia.h:1269
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb)
reproducible under travis and:
julia> versioninfo()
Julia Version 0.4.0-pre+7178
Commit cd87abb (2015-09-04 16:16 UTC)
DEBUG build
Platform Info:
System: Linux (x86_64-linux-gnu)
CPU: Intel(R) Core(TM) i5-4200U CPU @ 1.60GHz
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT NO_AFFINITY NEHALEM)
LAPACK: libopenblas
LIBM: libopenlibm
LLVM: libLLVM-3.3
__precompile__(false)
is not helping.
eltype
is not implemented for JldDataset
, which means eltype(::JldDataset)
always returns Any
. This should be fixed, following a similar definition in HDF5.jl.
ref @timholy comment. I didn't see an issue here for it, so I figured I should open one.
(Thanks for pointing this out, Tim - I hadn't yet realized @nanosoldier had broken).
Unable to run JLD test. Running Julia 0.4.6 64bit on Windows 10. When I tried test("JLD"), it failed with the following message. JLD version 0.6.0.
jldtests.jl
=================================[ ERROR: JLD ]=================================
failed process: Process('C:\Users\username\AppData\Local\Julia-0.4.6\bin\julia' --check-bounds=yes --code-coverage=none --color=no 'C:\Users\username\.julia\v0.4\JLD\test\runtests.jl'
, ProcessExited(3221226356)) [3221226356]
Hi, I have the problem that a type identity check via isa
fails after I reload my custom type. Here's code to reproduce the issue:
# Foo.jl
module Foo
type Bar
end
end
using JLD
reload("Foo")
## store
x = Foo.Bar()
@save "data.jld" x
## load 1
reload("Foo")
@load "data.jld"
@show isa(x,Foo.Bar) # = true
## load 2
reload("Foo")
@load "data.jld"
@show isa(x,Foo.Bar) # = false
Shouldn't both isa
calls return true?
Best,
Uwe
Julia version 0.4.3
JLD version 0.5.7
On current Julia master, the unit tests hang for me on OSX, with zero processor load:
Julia Version 0.4.0-dev+6479
Commit 8dcdf83* (2015-08-03 05:28 UTC)
Platform Info:
System: Darwin (x86_64-apple-darwin13.4.0)
CPU: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
The unit tests for HDF5 pass. On Julia commit a2ed42c from about a month ago the JLD tests pass on OSX. On Linux the tests pass just as they should.
When hitting Ctrl-C Julia simply returns to the command line without any stack trace / info.
julia> @save "tachyon1.jlprof" li lidict
WARNING: Method definition h5convert!(Ptr, JLD.JldFile, Bool, JLD.JldWriteSession) in module JLD at /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:590 overwritten at /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:590.
WARNING: Method definition h5convert!(Ptr, JLD.JldFile, Bool, JLD.JldWriteSession) in module JLD at /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:590 overwritten at /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:590.
WARNING: Method definition h5convert!(Ptr, JLD.JldFile, Bool, JLD.JldWriteSession) in module JLD at /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:590 overwritten at /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:590.
WARNING: Method definition h5convert!(Ptr, JLD.JldFile, Bool, JLD.JldWriteSession) in module JLD at /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:590 overwritten at /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:590.
WARNING: Method definition h5convert!(Ptr, JLD.JldFile, Bool, JLD.JldWriteSession) in module JLD at /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:590 overwritten at /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:590.
ERROR: TypeError: getfield: expected Symbol, got Int64
in h5convert!(::Ptr{UInt8}, ::JLD.JldFile, ::Module, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:627
in #write_compound#15(::Array{Any,1}, ::Any, ::JLD.JldGroup, ::ASCIIString, ::Module, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:693
in write_ref(::JLD.JldFile, ::Module, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:651
in h5convert_array(::JLD.JldFile, ::Array{Any,1}, ::JLD.JldDatatype, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:595
in #_write#11(::Array{Any,1}, ::Any, ::JLD.JldGroup, ::ASCIIString, ::Array{Any,1}, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:559
in write_ref(::JLD.JldFile, ::Array{Any,1}, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:651
[inlined code] from /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:615
in h5convert!(::Ptr{UInt8}, ::JLD.JldFile, ::LambdaInfo, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:627
in #write_compound#15(::Array{Any,1}, ::Any, ::JLD.JldGroup, ::ASCIIString, ::LambdaInfo, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:693
in write_ref(::JLD.JldFile, ::LambdaInfo, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:651
[inlined code] from /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:615
in h5convert!(::Ptr{UInt8}, ::JLD.JldFile, ::Nullable{LambdaInfo}, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:627
in #write_compound#15(::Array{Any,1}, ::Any, ::JLD.JldGroup, ::ASCIIString, ::Nullable{LambdaInfo}, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:693
in write_ref(::JLD.JldFile, ::Nullable{LambdaInfo}, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:651
[inlined code] from /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:615
in h5convert!(::Ptr{UInt8}, ::JLD.JldFile, ::StackFrame, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:627
in #write_compound#15(::Array{Any,1}, ::Any, ::JLD.JldGroup, ::ASCIIString, ::StackFrame, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:693
[inlined code] from ./boot.jl:303
in #_write#14(::Array{Any,1}, ::Any, ::JLD.JldGroup, ::ASCIIString, ::StackFrame, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:687
in write_ref(::JLD.JldFile, ::StackFrame, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:651
in h5convert_array(::JLD.JldFile, ::Array{StackFrame,1}, ::JLD.JldDatatype, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:595
in #_write#11(::Array{Any,1}, ::Any, ::JLD.JldGroup, ::ASCIIString, ::Array{StackFrame,1}, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:559
in write_ref(::JLD.JldFile, ::Array{StackFrame,1}, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:651
[inlined code] from /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:615
in h5convert!(::Ptr{UInt8}, ::JLD.JldFile, ::JLD.AssociativeWrapper{UInt64,StackFrame,Dict{UInt64,StackFrame}}, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/jld_types.jl:627
in #write_compound#15(::Array{Any,1}, ::Any, ::JLD.JldFile, ::ASCIIString, ::JLD.AssociativeWrapper{UInt64,StackFrame,Dict{UInt64,StackFrame}}, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:693
in #_write#14(::Array{Any,1}, ::Any, ::JLD.JldFile, ::ASCIIString, ::JLD.AssociativeWrapper{UInt64,StackFrame,Dict{UInt64,StackFrame}}, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:682
[inlined code] from ./boot.jl:303
in #write#8(::Array{Any,1}, ::Any, ::JLD.JldFile, ::ASCIIString, ::Dict{UInt64,StackFrame}, ::JLD.JldWriteSession) at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:682
in anonymous at /home/kfischer/.julia/v0.5/JLD/src/JLD.jl:1127
in eval(::Module, ::Any) at ./boot.jl:237
I'm having problems with the load macro on Julia v0.5 (JLD version 0.6.3). Here is a short example.
Here I use the save macro to save x
and y
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_) | Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type "?help" for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.5.0-pre+5628 (2016-07-22 16:48 UTC)
_/ |\__'_|_|_|\__'_| | Commit 1ef4eb1 (0 days old master)
|__/ | x86_64-apple-darwin15.6.0
julia> x = [1, 2]
2-element Array{Int64,1}:
1
2
julia> y = [3, 4]
2-element Array{Int64,1}:
3
4
julia> using JLD
julia> @save "test.jld" x y
If I then use the load macro in Julia v0.5, I only recover x
.
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_) | Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type "?help" for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.5.0-pre+5628 (2016-07-22 16:48 UTC)
_/ |\__'_|_|_|\__'_| | Commit 1ef4eb1 (0 days old master)
|__/ | x86_64-apple-darwin15.6.0
julia> using JLD
julia> @load "test.jld"
2-element Array{Symbol,1}:
:x
:y
julia> y
ERROR: UndefVarError: y not defined
in eval(::Module, ::Any) at ./boot.jl:234
in macro expansion at ./REPL.jl:92 [inlined]
in (::Base.REPL.##3#4{Base.REPL.REPLBackend})() at ./event.jl:46
julia> x
2-element Array{Int64,1}:
1
2
However, everything works as expected on Julia v0.4
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_) | Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type "?help" for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.4.7-pre+1 (2016-06-19 17:17 UTC)
_/ |\__'_|_|_|\__'_| | Commit 57d0834 (33 days old release-0.4)
|__/ | x86_64-apple-darwin15.5.0
julia> using JLD
julia> @load "test.jld"
2-element Array{Symbol,1}:
:x
:y
julia> y
2-element Array{Int64,1}:
3
4
julia> x
2-element Array{Int64,1}:
1
2
julia> immutable Foo
x::Int
y::Int
end
julia> using JLD
julia> JLD.save("test.jld", "f", Foo(1,1))
julia> JLD.load("test.jld", "f")
ERROR: type DataType has no field pointerfree
in _gen_jlconvert_immutable(::JLD.JldTypeInfo, ::Any) at /Users/jarrettrevels/.julia/v0.5/JLD/src/jld_types.jl:454
in gen_jlconvert(::JLD.JldTypeInfo, ::Any) at /Users/jarrettrevels/.julia/v0.5/JLD/src/jld_types.jl:551
in jldatatype(::JLD.JldFile, ::HDF5.HDF5Datatype) at /Users/jarrettrevels/.julia/v0.5/JLD/src/jld_types.jl:694
in read(::JLD.JldDataset) at /Users/jarrettrevels/.julia/v0.5/JLD/src/JLD.jl:381
in read(::JLD.JldFile, ::String) at /Users/jarrettrevels/.julia/v0.5/JLD/src/JLD.jl:357
in #jldopen#7(::Array{Any,1}, ::Function, ::JLD.##29#30{String}, ::String, ::Vararg{String,N}) at /Users/jarrettrevels/.julia/v0.5/JLD/src/JLD.jl:256
in load(::FileIO.File{FileIO.DataFormat{:JLD}}, ::String) at /Users/jarrettrevels/.julia/v0.5/JLD/src/JLD.jl:1217
in #load#13(::Array{Any,1}, ::Function, ::String, ::String, ::Vararg{String,N}) at /Users/jarrettrevels/.julia/v0.5/FileIO/src/loadsave.jl:45
in load(::String, ::String) at /Users/jarrettrevels/.julia/v0.5/FileIO/src/loadsave.jl:45
in eval(::Module, ::Any) at ./boot.jl:234
in macro expansion at ./REPL.jl:92 [inlined]
in (::Base.REPL.##1#2{Base.REPL.REPLBackend})() at ./event.jl:46
julia> versioninfo()
Julia Version 0.5.0-dev+5163
Commit 45ae7d8 (2016-07-05 17:15 UTC)
Platform Info:
System: Darwin (x86_64-apple-darwin13.4.0)
CPU: Intel(R) Core(TM) i5-4288U CPU @ 2.60GHz
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
LAPACK: libopenblas64_
LIBM: libopenlibm
LLVM: libLLVM-3.7.1 (ORCJIT, haswell)
ref various @nanosoldier benchmark job failures here, here, and likely here as well (I'm expecting it to fail)
julia> JLD.save("test.jld", "x", Nullable{UTF8String}())
ERROR: UndefRefError: access to undefined reference
in write_compound at /Users/jarrettrevels/.julia/v0.4/JLD/src/JLD.jl:683
in write at /Users/jarrettrevels/.julia/v0.4/JLD/src/JLD.jl:504
in anonymous at /Users/jarrettrevels/.julia/v0.4/JLD/src/JLD.jl:1165
in jldopen at /Users/jarrettrevels/.julia/v0.4/JLD/src/JLD.jl:240
in save at /Users/jarrettrevels/.julia/v0.4/JLD/src/JLD.jl:1163
in save at /Users/jarrettrevels/.julia/v0.4/FileIO/src/loadsave.jl:51
This is because the value
of the Nullable is #undef
:
julia> dump(Nullable{UTF8String}())
Nullable{UTF8String}
isnull: Bool true
value: #undef
Note, then, that types which initialize with values are unaffected, e.g. numeric primitives:
julia> dump(Nullable{Float64}())
Nullable{Float64}
isnull: Bool true
value: Float64 2.151511976e-314
julia> JLD.save("test.jld", "x", Nullable{Float64}()) # works fine
When I run using JLD
I get the following warnings:
INFO: Recompiling stale cache file C:\Julia\PkgDir\lib\v0.4\JLD.ji for module JLD. WARNING: Module HDF5 uuid did not match cache file This is likely because module HDF5 does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from C:\Julia\PkgDir\lib\v0.4\JLD.ji INFO: Precompiling module JLD... INFO: Recompiling stale cache file C:\Julia\PkgDir\lib\v0.4\JLD.ji for module JLD. WARNING: Module HDF5 uuid did not match cache file This is likely because module HDF5 does not support precompilation but is imported by a module that does. __precompile__(true) but require failed to create a precompiled cache file
If I then run say:
A=ones(40)
save("c:\\OneDrive\\Model\\myfile.jld", "A", A)
I get the error: UndefVarError: save not defined in include_string at loading.jl:288 in eval at C:\Julia\PkgDir\v0.4\Atom\src\Atom.jl:3 [inlined code] from C:\Julia\PkgDir\v0.4\Atom\src\eval.jl:39 in anonymous at C:\Julia\PkgDir\v0.4\Atom\src\eval.jl:108 in withpath at C:\Julia\PkgDir\v0.4\Requires\src\require.jl:37 in withpath at C:\Julia\PkgDir\v0.4\Atom\src\eval.jl:53 [inlined code] from C:\Julia\PkgDir\v0.4\Atom\src\eval.jl:107 in anonymous at task.jl:58
I'm using Atom 1.8.0 and Julia 0.4.7
I think this is the result of #30, but I now get errors like:
ERROR: LoadError: LoadError: FileIO.File{FileIO.DataFormat{:UNKNOWN}}("/tmp/juliao8DRo1") couldn't be recognized by FileIO.
when I try to read/write to a filename without the .jld extension.
Is this the new status quo?
When I try to read in a dataset I get the following:
ERROR: stored type Core.Tuple{Core.Float64,Core.Bool} does not match currently loaded type
I can't do it right this second, but soon I will construct a minimal example and post the code.
If I'm not mistaken, right now one can only specify compression via compress::Bool - but it would be helpful to specify which kind of compression to use, especially since not everyone has the blocs filter compiled & installed in hdf5/lib/plugin
I have a variable outfile
that reads a string "m1.jld" from a text file and I do fid = jldopen(outfile, "w");
The following error message pops up
ERROR: LoadError: SystemError: opening file m1.jld: No such file or directory.
But on the other hand when I do fid = jldopen("m1.jld", "w");
everything works fine. So I printed out the outfile and that is fine too. But in the folder (via terminal), a file has been created by the program under the name "m1.jld?". I even tried doing a chomp
on the outfile to get rid of any breakline characters. Any help with this issue is appreciated.
f = ()->(x->x)
save("test.jld", "f", f)
returns
HDF5-DIAG: Error detected in HDF5 (1.8.11) thread 0:
#000: ../../../src/H5Tcompound.c line 374 in H5Tinsert(): unable to insert member
major: Datatype
minor: Unable to insert object
#001: ../../../src/H5Tcompound.c line 452 in H5T__insert(): member name is not unique
major: Datatype
minor: Unable to insert object
ERROR: Error adding field _ to compound datatype
in h5t_insert at /home/numerik/bzfsikor/.julia/v0.4/HDF5/src/plain.jl:2026
in h5type at /home/numerik/bzfsikor/.julia/v0.4/JLD/src/jld_types.jl:400
in write_compound at /home/numerik/bzfsikor/.julia/v0.4/JLD/src/JLD.jl:679
in write_ref at /home/numerik/bzfsikor/.julia/v0.4/JLD/src/JLD.jl:641
in h5convert_array at /home/numerik/bzfsikor/.julia/v0.4/JLD/src/JLD.jl:585
in _write at /home/numerik/bzfsikor/.julia/v0.4/JLD/src/JLD.jl:549
in write_ref at /home/numerik/bzfsikor/.julia/v0.4/JLD/src/JLD.jl:641
[...]
Dearests,
I want to store an array of type x::Array{Tuple{Int64,Int64,Float64},1}
. Hereafter is the result
julia> x
10000-element Array{Tuple{Int64,Int64,Float64},1}
now I @save "x.jld" x
and the size of the file is
-rw-r--r-- 1 pagnani staff 4096424 Oct 15 17:02 x.jld
if I now define xmat = [x[i][j] for i=1:length(x), j=1:3]
and obtain Array{Any,2}
and do a
writedlm("xmat.txt", xmat)
260652 Oct 15 17:07 xmat.txt
In other words the jld file is almost 16 times larger. Note that 10000 * 3 * 8 /1024 = 234 Kb
should be the theoretical lower bound which is almost achieved from dlmwrite!!
It might be a known issue with tuples but I was not able to find neither here nor on the mailing list. In case it is already known ... sorry for the noise.
Thanks a lot for your work
Andrea
Hello, world! This is the new standalone JLD package. This is something I should have done a long time ago. The thing that finally kicked my lazy bum into gear is that it seems to be a design that works better with package precompilation (but again, it's something that should be done anyway).
I'd appreciate it if folks could test this. If you do, be sure to check out the teh/jld_split
branch of HDF5.jl
, or you're likely to encounter trouble.
Once we deal with any issues, I'll register this as an official package.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.