Needed to use gpu function to move model to gpu.

Might be easier if flux issue #803 fix was incorporated first. However, gpu method (mapleaves really) seems to assume all (array type?) leaves are parameter arrays?

Have not seen any good way to shrink the size. Best I can think of is to remove from edges, picking the row/col with smallest magnitude.

Because NoParams is not an AbstractMutableComp...

It tries to perform multiplication between cpu and gpu arrays

Need to restrict functor fields for it

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Seems to happen for large activations (typically 2D conv with 500+ feature maps). The symptoms are low GPU utilization which seems to be due to excessive GC:ing.

The elementwise multiplication seems to be the culprit as replacing everything inside the gradient hook with just act .* grad gives more or less identical performance as the full implementation.

Don't think there is much I can do about the operation itself, but given that the metric is a bit of a statistical thing it is probably ok to enable computing it

1. every X batches
2. on a subset of the examples in a batch

Need to do a few more experiments to see which one seems more useful and/or if both shall be allowed.

As it is now, one must manually add NaiveNASlib before adding NaiveNASflux:
Pkg.add("https://github.com/DrChainsaw/NaiveNASlib.jl")

Thread:
https://discourse.julialang.org/t/manage-dependencies-to-unofficial-packages/26129

For example,

nout(Conv((2,), 3=>4)) = 1

Rel 2 of NaiveNASlib will force a rel 2 due to reexport. Just as with NaiveNASlib, I plan to be quite liberal with breakage. Let me know if you are a user and would prefer less breakage.

Other things to fix:

• Rename mutable to fluxvertex (wtf was I thinking?!).
• Don't reexport Flux.
• Greatly reduce the amount of exported functions and types.
• Merge #53.
• Make all layerwrappers real functors instead of relying on mutation (if possible).
• Rename neuron_value to neuronutility (same for types affecting its behaviour)

Useful when inserting layers.

Haven't thought about how to do this for recurrent layers. Maybe trivial, maybe impossible...

Always returns a plain array

Stateful optimizers (like ADAM) store state per parameter array in a dict. Since parameter arrays are replaced with new instances when mutating the dict just keeps getting new entries.

This also obviously prevents the optimizers from working as they should.

Current implementation is from the days when DepthwiseConv was the only grouped convolution in Flux. DepthwiseConv was hardcoded to have groups == nin.

I think/hope that constraints can be handled by just applying the current constrains to subsets of the in/out variables. Not sure this will enable the full flexibilty of changing the nin/groups ratio but I'm also not sure if it is meaningful to allow more flexibilty than what the current constraints allow.

What could be the biggest headache is to let the part which creates the new parameter arrays to understand the number of groups and/or the groups/nin ratio from just the selected in/out indicies. Perhaps one needs to be able to pass arbitrary other data in the result from the solver which would require updates to NaiveNASlib.

Will save this one for a rainy day...