Overloading the sampling functions of probability distributions about autonn HOT 2 CLOSED

Hi, first of all instead of overloading Layer by modifying its source code, you could also use the convenience function Layer.fromFunction. It will return a Layer generator from your function (i.e. equivalent to overloading by modifying the source). There's a detailed guide in the docs (help Layer.fromFunction).

As for saving the output, there are a few work-arounds:

1. Recompute the forward pass in your backward function to get it (while it seems wasteful, if this is not the bottleneck of your network then it might be ok, and easier than any alternative).

2. Save it using the persistent keyword (check official Matlab docs). If you can guarantee only one instance of this layer, this will work without any complication. If there's many, you'll want separate state variables for each one; this can be implemented with the same persistent variable but with some additional bookkeeping to ensure it accesses a different state for each layer (I don't recommend it though).

3. Many layers with state can actually be implemented in a state-less way if you decompose them into smaller layers. For example, DropOut would need state, but in autonn it's broken up in 2: one is the mask generator (samples random masks), the other is the dropout layer that just applies the mask. So what would normally have to be stored as state, is now just a simple variable passed from one layer to another. It might take some creativity and I realize it might not apply when you're stuck with someone else's monolithic function, but if possible this is the cleanest choice in my opinion.

from autonn.

Wow, I didn't expect to get an answer this quick.

What a neat solution. I tried Option 1, and it seems to work well so far!

Thanks!

from autonn.