Maximum Step Size about optim.jl HOT 6 CLOSED

As a further question are their any line searches that attempt to find the same minimum that corresponds to taking infinitesimally small steps along the negative gradient? This is normally how a basin of attraction is defined. I imagine that a backtracking line search that has a maximum step size that is smaller than the width of a typical basin might work okay.

I typically use optimization algorithms without a sophisticated line search. Either directly using the LBFGS search direction and length or conjugate gradients with a single newton's method step along the descent direction. Both of which with a maximum step size.

I'd like to see if a fancy line search can improve performance for the problems I am interested in, but as it stands I can't get any of the available line searches in Optim to work. Each line search seems to have options that should be able to be changed, but I don't see any way to do that. The line search seems to be undocumented.

from optim.jl.

I'd certainly like to include max step sizes as an option. This will take some time to do, though, as the line search code is fairly complex.

I'm not sure about what infinitesimally small steps along the gradient means in floating point. You should use a constant-step gradient descent algorithm with step sizes equal to the the floating point epsilon, but this seems like a bad idea for both convergence and numerical stability. I suspect I'm missing something.

The line search is definitely undocumented. Some of the options are also not available because we haven't done enough to generalize our line search framework to discuss all of the different options in an easy way.

If you'd like to experiment with alternative line searches, you can copy the code for backtracking line search and then modify it to your taste. As long as you maintain the existing interface, you can pass in your new line search function as an argument to optimize.

from optim.jl.

linesearch_hz already has the ability to specify a maximum step size (it's needed for fminbox).

from optim.jl.

I think I see a way forward at least for my own personal testing.

I'll clarify what I meant about "infinitesimally small steps along the negative gradient". I have the view that the goal of a local optimization routine should be to try find the same minimum that would be found with gradient descent in the limit of the step size going to zero. This of course is only a theoretical definition, not a real approach to optimization.

Here is a recent paper that maps out which minima are reached from different starting coordinates for different optimization algorithms in atomistic systems. This sort of study is interesting because some algorithms try to associate points in space with their associated minimum. However, which minimum will be reached from a given initial point is dependent on many different parameters.

from optim.jl.

Since hz_linesearch is the default line search algorithm, and it includes alphamax as an option, can this be closed?

from optim.jl.

As mentioned, the functionality is already here, so I am closing this.

from optim.jl.