Possible performance regression / potential optimisation about prima HOT 2 OPEN

Hello @desal , thank you so much for your interest in adopting PRIMA in dlib, and for raising your concern.

Apologies for my long-delayed response.

A short answer to your question: PRIMA pays more attention to reducing the number of function evaluations and improving the code clarity & simplicity, even if this is at the cost of more flop taken inside the solvers.

Why is this reasonable? PRIMA is designed as a package for derivative-free optimization problems, where the cost is dominated by function evaluations, not the flop inside the solvers. In these problems, each function evaluation may take several minutes, hours, or days, and hence it makes little sense to care about how to save the flop and reduce the internal computational time by several milliseconds per iteration.

Of course, this does not mean that PRIMA does not care about the internal cost of solvers. This just means that there is a priority:

Reducing number of function evaluations >> improving code simplicity & clarity >> saving internal cost of solvers.

In your test, as you measured, each function evaluation takes an extremely short time. Therefore, the computing time is dominated by the internal cost of the solvers. However, this is normally not the case in practice --- or I should say, PRIMA is not designed for such a situation.

If your interest is to solve problems where the function evaluation is cheap, then Powell's solver may not be the correct choice. If you decide to adopt Powell's solvers while concerned about the internal cost of the solvers, then there are two possibilities.

1. Stay with the original F77 implementation (or its C translation) of the solvers, tolerating the fact that they are impossible to maintain and contain bugs.
2. Consider modifying PRIMA to utilize BLAS for the underlying matrix-verctor procedures (matprod, inprod, etc; you only need to consider the procedures in fortran/common/linalg.f90). However, I am not sure whether such a version will be much more efficient (in terms of internal costs of the solvers) than the current one.

To elaborate things better, I have posted a discussion. I hope it is not too long to read (the length partially explains the delay). I will be very glad to see your comments there.

from prima.

Given the above comments, the results of your test are still a bit strange to me.

1. What kind of flags did you provide to the compiler when compiling PRIMA?
2. Did you by any chance set PRIMA_DEBUGGING to 1 in fortran/common/ppf.h? Such a setting will be expensive and should not be used in production.

from prima.