Restrict transform to columns? about tabletransforms.jl HOT 4 CLOSED

Can you elaborate on why the large cache is an issue in this scenario? Is it because you expected more performance? Or because the memory footprint is too high for the application?

Assuming that it is indeed an issue, would it make sense to "materialize" the pipeline at a specific step? For example, you can always drop the cache by evaluating the pipeline with pipeline(table) and then take the resulting table and insert it into a second pipeline. Would that work?

I am asking these questions because we could consider adding a column selection feature to all Colwise transforms, but this increased code complexity must be justified with a popular use case.

from tabletransforms.jl.

It is not really an issue for my application since the amount of data is not large.

In my use case I save the cache of the pipeline to later use it on new data samples. Saving/Using the cache of Select is a problem in this case since revert would re-add the column from the old data, which might not even be the same size as the new data.

Thus one needs to carefully split up the pipeline and only save/use the correct cache which is not so nice.

Problem with revert and Select:

julia> p = Select(:a) → MinMax()
Sequential(TableTransforms.Transform[Select{Tuple{Symbol}}((:a,)), Scale{Int64}(0, 1)])

julia> t = (a=rand(10), c=rand(10));

julia> _, cache = apply(p, t);

julia> z = reapply(p, (a=rand(3), c=rand(3)), cache);

julia> length(revert(p, z, cache).c)
10

from tabletransforms.jl.

I think the issue we are facing here is more profound. It has to do with the fact that Select's revertibility is tied to a specific input table. When you reapply the pipeline to a new table the cache isn't modified and so you cannot "unselect" the columns of the new table that were never stored. I wonder what could be done differently?

What is your proposal for this new Restrict transform and how it would address the issue above?

from tabletransforms.jl.

@antholzer I will close this issue, but feel free to reopen it if you have a proposal that we could brainstorm further.

from tabletransforms.jl.