better update! interface about knet.jl HOT 8 CLOSED

The docs indicate constructors like Momentum(), in actual code you require Momentum(w). Should we go back to Momentum()?

from knet.jl.

I am not sure if the functional interface: (w,p)=update!(w,g,p) is worth supporting. The original motivation was scalar parameters. But you use axpy! so you don't support this anyway?

from knet.jl.

There should be a update!(w,g;lr=0.1) which defaults to Sgd. This will come in handy if w,g are iterators, and we just want to apply a simple Sgd update to all without having to construct separate optimization objects.

from knet.jl.

Since all current update! methods require a well typed prms argument, the untyped update!(w,g,p) can be used to implement the iterator method. In fact this will support iterators within iterators (embedded arrays). The only thing it won't support is w::Dict, which requires its own method. In each case we require w,g,p to have parallel structure.

from knet.jl.

Do we have the Nesterov version of Momentum or the regular? Maybe we can make that a flag...

from knet.jl.

I am working on this in the newupdate branch.

from knet.jl.

We use the interface (w,p)=update!(w,g,p) to allow further developments. We don't want everyone change their own code when we change our inner code.

We don't have any nesterov...

from knet.jl.

The new calls have been integrated. The problem with the (w,p)=update!(w,g,p) syntax is (1) either we are modifying in-place, in which case it is not necessary, (2) or we are constructing new w,p in which case we need to match their types to the originals, i.e. dictionary, tuple, array, struct etc. I don't think it is worth the trouble.

from knet.jl.