ENH: Improve interface for working with groups in jld files about hdf5.jl HOT 8 CLOSED

One issue is that @save currently creates a new file, rather than adding to an
existing file. That means it's kind of a one-shot deal. Might the idea be to
add an @Append command?

With regards to group-handling, would @load "file_name/group_name" a b c
be reasonable?

I'm kinda tied up with other coding projects right now, and may not get to
this for quite a while I'm afraid---I'm about to have to kick back into grant-
writing mode, and am simply hoping to finish my current stuff rather than start
new things. If you get impatient and want to tackle this yourself, the
relevant code is in the two macros in src/jld.jl.

Best,
--Tim

On Monday, December 23, 2013 03:47:31 PM Spencer Lyon wrote:

Just a quick idea...

We can currently organize things into groups within a .jld file, but only
using the lower-level HDF-like interface.

I think it would be great to extend the @load and @save macros to allow
for .jld files to be organized into groups. I would think that the
interface could look something like this:

@load "file_name" "group_name"  # loads all items in that group
@load "file_name" "group_name" a b c  # loads a b and c from the group

@save "file_name" "group_name"  # save all items from current module in 
group @save "file_name" "group_name"  a b c  # loads a b and c from the
group ```

Sorry, but the comments below are a bit stream of consciousness

I just thought maybe it would make more sense to have `@load_group` and
`@save_group` macros. However, multiple dispatch could make the above ideas
work because we could dispatch on two strings followed by optional symbols.

Another thought that maybe when we do this automatic creation of groups we
would have `/group_name/_refs` and `/group_name/_types` also.

---

The reason I want this feature is that I really like how easy it is  to
`@load` and `@save` many variables into/out of modules with a quick one
liner. I have projects where I have different values of underlying
parameters that are used to generate different datasets. I am currently
getting this easy functionality by creating different jld files for each
parameterization, but I would prefer to do this in a single file with
groups. Maybe I'm crazy... feedback on the idea as well as implementation
would be great.

---
Reply to this email directly or view it on GitHub:
https://github.com/timholy/HDF5.jl/issues/67

from hdf5.jl.

The idea for an @append macro and the syntax @load "file_name/group_name" a b c are both good suggestions. Right after making the initial post I realized that, unless I am still wrong, Julia doesn't currently do multiple dispatch on macros. Therefore, the suggested syntax of having a single string specifying the location (file with optional group) followed by optional variable names makes the most sense.

@timholy Thanks for the update on your situation. I will most likely take a stab at this. It doesn't seem like it will be very difficult. I'm a python guy, so the usage of macros is still a little foreign to me, but I have been doing a lot of experimentation and am picking it up.

If I have questions, I'll post them here.

from hdf5.jl.

It doesn't do MD on macro arguments, but you can call out to a function that does so. But since this is all doing run-time generation of code anyway, you can just test the type of arguments at runtime.

from hdf5.jl.

This is somewhat of a null point because I like the `"file_name/group_name" syntax better (it matches the dict like access and the internal hdf structure), but for my own understanding...

But since this is all doing run-time generation of code anyway, you can just test the type of arguments at runtime.

By this do you mean just that inside macro load(filename, vars...) we could check isa(vars[1], String) and then assume this is the group name. I'm sure this would work, but would that be considered good "Julia form"?

from hdf5.jl.

Yes, that works. The more typical Julian form might be

macro test(args...)
    test(args...)
end

function test(filename::String, restofargs...)
    # stuff goes here
end

function test(somethingelse, restofargs...)
    # Non-string version
end

But in practice I'm pretty sure that using MD here doesn't matter, because macro-expansion happens at parsing time, not at compile-time, and that means that your usual intuitions about good form probably don't apply.

As fair warning, this issue of when things run makes the @save and @load macros slightly complicated even to think about. #37 may give you a hint as to the issues (it's not many lines of code, but at least for me each line was the source of more than typical head-scratching). That said, I suspect you won't have to do major redesign of how these work, so you may be able to avoid falling into traps.

from hdf5.jl.

Ok thanks for that.

I actually tried to extend the @save and @load the other day and ran in to all sorts of scoping issues. I'll reference #37 for some pointers.

Outside the context of this package, is there a Julia convention or approach for interpolating macro arguments?

from hdf5.jl.

If you mean string-interpolation in macro arguments, I suspect it comes up very rarely. @load and @save really are unusual, because they are "invasive"---they define or access variables whose symbols are not specified by the caller. Now couple it with the desire to be able to specify the filename as a variable or interpolated string, and this forces one to break the usual recommendation of avoiding calls to eval in macros. Nasty stuff.

from hdf5.jl.

I'd like to suggest that this has, for all practical purposes, been closed by @mbauman in #101. Feel free to complain if you disagree.

from hdf5.jl.