patwie / cppnumericalsolvers Goto Github PK

Hi!

I used the Fortran-code of LBFGSB by Nocedal et al frequently in the past and tried out your code.

My question is if it is possible to implement the version of Nocedal with strong Wolfe-Powell line search conditions. I found your implementation of LBFGS not very robust for most of my optimization problems in large dimensions....

Hi,

I'm not really sure how to do a feature request, so I'm sorry if opening an issue is not the proper way to do so.

Here it goes: is there a plan to add something like itermax, xabs/xrel and fabs/frel tolerance options in the future?

Would be great to print some info after each iteration (norm, objective, etc) like R can do. Or maybe provide an iteration callback so user can assess Solver state.

A few commits ago, there was already added an info struct, which allows to query some variables.

I think we should also add some more infos about the result of the minimization. Up until now, you don't even now whether the algorithm converged or not.

So I suggest to add at least the information whether the algorithm was successfull. If it wasn't, some information should also be provided. E.g. NO_CONVERGENCE, LEFT_DOMAIN, etc.

I also experienced a case, where the solver had a smaller minimum at one point and later got worse. Maybe it would also be better to store the overall minimum + location and return this one as the minimum and not the last point.
This would also make sense in some applications, where you need to get back a valid position/value in any cases.

These are just a few small suggestions that came to my mind when using the library. Feel free to correct me if you don't agree!

Hi,

At line 33 of CMakeLists.txt, I think that

link_libraries(::Eigen3)

should be replaced by

include_directories(SYSTEM ${EIGEN3_INCLUDE_DIR})

Since Eigen3 is header only, I do not get the linking through cmake's link_libraries, which gives returns an error on my computer (OSX).

Best,
Thibault

Add an example of using https://github.com/dmlc/mshadow in an objective function or even re-writing some parts to use CUDA in the solvers.

With the current version of code from master branch:

$ clang++-3.6 -std=c++11 -I ../../include/cppoptlib/ -I /usr/include/eigen3/ nonnegls.cpp

$ ./a.out
true b  =   0.05349  0.539828 -0.199543  0.783059       loss:0
start b = 0.433371 0.295083 0.615449 0.838053   loss:3.37031
a.out: /usr/include/eigen3/Eigen/src/Core/CwiseBinaryOp.h:109: Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<double, double>, const Eigen::CwiseBinaryOp<Eigen::internal::scalar_max_op<double, double>, const Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<double, double>, const Eigen::Matrix<double, -1, 1, 0, -1, 1>, const Eigen::Matrix<double, -1, 1, 0, -1, 1> >, const Eigen::Matrix<double, -1, 1, 0, -1, 1> >, const Eigen::Matrix<double, -1, 1, 0, -1, 1> >::CwiseBinaryOp(const Lhs &, const Rhs &, const BinaryOp &) [BinaryOp = Eigen::internal::scalar_min_op<double, double>, Lhs = const Eigen::CwiseBinaryOp<Eigen::internal::scalar_max_op<double, double>, const Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<double, double>, const Eigen::Matrix<double, -1, 1, 0, -1, 1>, const Eigen::Matrix<double, -1, 1, 0, -1, 1> >, const Eigen::Matrix<double, -1, 1, 0, -1, 1> >, Rhs = const Eigen::Matrix<double, -1, 1, 0, -1, 1>]: Assertion `aLhs.rows() == aRhs.rows() && aLhs.cols() == aRhs.cols()' failed.
Aborted

The error is caused by BoundedProblem::upperBound() returning a 10-element array, while DIM=4.

Hi!

Eigen/src/Core/CwiseNullaryOp.h failed at YOU_CALLED_A_FIXED_SIZE_METHOD_ON_A_DYNAMIC_SIZE_MATRIX_OR_VECTOR assertion. It is a constant matrix (Eigen::DenseBase::ConstantReturnType) which is expected somewhere (getters of bounds, I presume) and its size must be known beforehand.

My problem is defined like this:

class OptWrtTProblem : public cppoptlib::BoundedProblem<Double> {
    public:
        using Superclass    = cppoptlib::BoundedProblem<Double>;
        using DVector       = TVector;
        using EigenMD       = Map<Matrix>;
        using EigenMDconst  = Map<const Matrix>;
        using DVectorM      = Map<DVector>;
        using DVectorMconst = Map<const DVector>;


    private:
        const Matrix& mD;
        const Matrix& mTfixed;
        const Matrix& mA;
        double lambda;

        const size_t m;
        const size_t n;
        const size_t r;

        /* some time savers */
        const Matrix& onesmr;
        Matrix mAAt;
        Matrix mDAt;
        Double regTfixed;
        Matrix gradRegTfixed;
        Double dotGradTfixed;

    public:
        OptWrtTProblem(const Matrix& mDIn,
                       const Matrix& mTfixedIn,
                       const Matrix& mAIn,
                       double lambdaIn,
                       const Matrix& onesmrIn,
                       const DVector& lb,
                       const DVector& ub
        )
        : Superclass(lb, ub),
          mD{mDIn},
          mTfixed{mTfixedIn},
          mA{mAIn},
          lambda{lambdaIn},
          onesmr{onesmrIn},
          m(mD.rows()),
          n(mD.cols()),
          r(mA.rows())
        {
            mAAt = mA * mA.transpose();
            mDAt = mD * mA.transpose();

            regTfixed = mTfixed.sum() - mTfixed.squaredNorm();
            gradRegTfixed = onesmr - 2 * mTfixed;
            dotGradTfixed = gradRegTfixed.cwiseProduct(mTfixed).sum();
        }

        Double value(const DVector& mTVec) {
            EigenMDconst mT(mTVec.data(), m, r);
            Double loss = (mD - mT * mA).squaredNorm();
            return 0.5 * (loss + lambda * (regTfixed
                        + gradRegTfixed.cwiseProduct(mT).sum() - dotGradTfixed));
        }

        void gradient(const DVector& mTVec, DVector& gradVec) {
            EigenMDconst mT(mTVec.data(), m, r);
            EigenMD grad(gradVec.data(), m, r);
            grad = (mT * mAAt - mDAt) + 0.5 * lambda * gradRegTfixed;
        }
};

And this is how I create an instance:

        OptWrtTProblem problem(mD, mTfixed, mA, lambda, onesmr,
                OptWrtTProblem::DVector::Zero(DIM),
                OptWrtTProblem::DVector::Ones(DIM));
        cppoptlib::LbfgsbSolver<OptWrtTProblem> solver;

Am I doing something wrong? My c++ is quite poor...

Thank you very much for your time!

There are now 2 test-cases which do not work

[  FAILED  ] 2 tests, listed below:
[  FAILED  ] CMAesTest.RosenbrockFarValue
[  FAILED  ] CMAesTest/0.RosenbrockFarFull, where TypeParam = float

Hi,

I am encountering problems when using BoundedProblem without defining the gradient analytically.

Looking at the code, it seems that the function finiteGradient does not adjust for the case where xx[d] += coeff2[accuracy][s]*eps; might result in a value outside the bounds. Am I missing something here?

I am not a C++ expert. I've tried to understand the examples and tests. I see there is a lot of code for using BoundedProblem (lbfgsbsolver.h), but I am having a hard time picking out a comprehensible example I can follow.

In bfgssolver.h, the gradient is calculated twice within the minimization loop, apparently vain. The first call to objFunc.gradient() (line 27) seems to be for getting the initial gradients and thus could be outside the loop, right?

Hi !

Thank you for your work on this library. I'm working on a reimplementation of one of my method from matlab to C++ and your library helped me a lot simplified the readability of my code.

I work on VS10 (not supporting C++11 features) and I rewrote parts of your library to make it work.
Do you want me to contribute to your library (create a C++03 branch for instance)?

Cheers,

Using Ctypes, it should be possible to write the objective function in Python and call the c++ solver

There's a problem in lbfgsbsolver.h which can cause a error.

From line 249:

if (test > 1e-7 * newY.squaredNorm()) {
    if (k < this->settings_.m) {
          yHistory.conservativeResize(DIM, k + 1);
          sHistory.conservativeResize(DIM, k + 1);
    } else {
          yHistory.leftCols(this->settings_.m - 1) = yHistory.rightCols(this->settings_.m - 1).eval();
          sHistory.leftCols(this->settings_.m - 1) = sHistory.rightCols(this->settings_.m - 1).eval();
    }

If the condition is false, i.e. test <= 1e-7 * newY.squaredNorm(), then k is incremented on line 276.
If the condition is true the next time, k has been incremented but the yHistory & sHistory matrix was not resized, that's why I got an error on the line

yHistory.leftCols(this->settings_.m - 1) = yHistory.rightCols(this->settings_.m - 1).eval();

since the matrices don't have the given number of columns.

A possible, simple fix would just be:

 if (yHistory.cols() < this->settings_.m) {
          yHistory.conservativeResize(DIM, yHistory.cols() + 1);
          sHistory.conservativeResize(DIM, sHistory.cols() + 1);
 } 

The BFGS solver fails to solve the Rosenbrock problem for x0 = (-1.2, 1): it returns (NaN, NaN). This starting point is used in Rosenbrock's paper ( http://comjnl.oxfordjournals.org/content/3/3/175.full.pdf+html ) and makes the optimization more difficult than x0=(1,2) which is proposed in the README.

I suspect this is caused by the used line search (seems to be the Armijo rule). As far as I know the Armijo rule is not suitable for BFGS because the approximation of the inversed Hessian does not need to stay positive definite.

Test case:

auto rosenbrock = [](const Vector &x) -> double {
const double t1 = (1-x[0]);
const double t2 = (x[1]-x[0]_x[0]);
return t1_t1 + 100_t2_t2;
};

// create derivative of function
auto Drosenbrock = [] (const Vector x, Vector &grad) -> void {
    grad[0]  = -2*(1-x[0])+200*(x[1]-x[0]*x[0])*(-2*x[0]);
    grad[1]  = 200*(x[1]-x[0]*x[0]);
};

// initial guess
Vector x0(2);x0 << -1.2,1;
// get derivative
Vector dx0(2);
Drosenbrock(x0,dx0);
// verify gradient by finite differences
checkGradient(rosenbrock,x0,dx0);

BfgsSolver g;
g.solve(x0,rosenbrock,Drosenbrock);

// prints -nan -nan
std::cout << std::endl<<std::endl<< x0.transpose();

I tried to install the CppNumericalSolvers in our linux service. However, I am confused about the step of adjusting paths and compiler (supports g++, clang++). If we do not have the Matlab, whether can I install the library? Grateful if you can give a detailed description of the CppNumericalSolvers.config file.

Hello,

The code compiled and verified correctly in the terminal but will not compile properly in Matlab.

I use gcc 4.7 and g++ 4.7 on Ubuntu 16.04.

Matlab R2015a

The errors encountered are below. I don't know much (if anything) about C++, from the small amount I read it might be a name space referencing error when referencing cppoptlib?

If you have any insight as to what the problem might be I would be quite grateful.

Thanks,

Kevin

Error using mex
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:30:7: error: ‘Vector’ does not name a type
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:35:17: error: ‘Vector’ does not name a type
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:35:17: error: ISO C++ forbids declaration of ‘parameter’ with no type
[-fpermissive]
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:35:23: error: expected ‘,’ or ‘...’ before ‘<’ token
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:44:23: error: ‘Vector’ does not name a type
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:44:23: error: ISO C++ forbids declaration of ‘parameter’ with no type
[-fpermissive]
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:44:29: error: expected ‘,’ or ‘...’ before ‘<’ token
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:65:22: error: ‘Vector’ does not name a type
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:65:22: error: ISO C++ forbids declaration of ‘parameter’ with no type
[-fpermissive]
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:65:28: error: expected ‘,’ or ‘...’ before ‘<’ token
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx: In member function ‘T MATLABobjective<T>::value(int)’:
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:37:47: error: ‘x’ was not declared in this scope
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx: In member function ‘void MATLABobjective<T>::gradient(int)’:
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:47:49: error: ‘x’ was not declared in this scope
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:58:7: error: ‘grad’ was not declared in this scope
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:61:28: error: ‘x’ was not declared in this scope
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:61:31: error: ‘grad’ was not declared in this scope
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx: In member function ‘void MATLABobjective<T>::hessian(int)’:
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:69:49: error: ‘x’ was not declared in this scope
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:83:27: error: ‘x’ was not declared in this scope
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx: In function ‘void mexFunction(int, mxArray**, int, const mxArray**)’:
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:93:27: error: cannot declare variable ‘f’ to be of abstract type
‘MATLABobjective<double>’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:33:7: note:   because the following virtual functions are pure within
‘MATLABobjective<double>’:
In file included from /home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:8:0:
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/problem.h:45:18: note: 	cppoptlib::Problem<Scalar_,
Dim_>::Scalar cppoptlib::Problem<Scalar_, Dim_>::value(const TVector&) [with Scalar_ = double; int Dim_ = -1;
cppoptlib::Problem<Scalar_, Dim_>::Scalar = double; cppoptlib::Problem<Scalar_, Dim_>::TVector = Eigen::Matrix<double, -1, 1>]
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:114:3: error: ‘Vector’ was not declared in this scope
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:114:10: error: expected primary-expression before ‘double’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:114:10: error: expected ‘;’ before ‘double’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:201:16: error: expected primary-expression before ‘double’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:201:16: error: expected ‘;’ before ‘double’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:202:11: error: ‘class MATLABobjective<double>’ has no member named
‘setLowerBound’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:202:25: error: ‘t’ was not declared in this scope
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:218:16: error: expected primary-expression before ‘double’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:218:16: error: expected ‘;’ before ‘double’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:219:11: error: ‘class MATLABobjective<double>’ has no member named
‘setUpperBound’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:219:25: error: ‘t’ was not declared in this scope
In file included from /home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/gradientdescentsolver.h:6:0,
                 from /home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:10:
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h: In instantiation of ‘class
cppoptlib::ISolver<double, 1>’:
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/lbfgssolver.h:13:7:   required from ‘class
cppoptlib::LbfgsSolver<double>’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:232:25:   required from here
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h:15:51: error: ‘double’ is not a class, struct, or
union type
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h:16:52: error: ‘double’ is not a class, struct, or
union type
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h:17:53: error: ‘double’ is not a class, struct, or
union type
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h:18:54: error: ‘double’ is not a class, struct, or
union type
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:233:12: error: ‘class cppoptlib::LbfgsSolver<double>’ has no member named
‘minimize’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:233:24: error: ‘x’ was not declared in this scope
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:238:12: error: ‘class cppoptlib::LbfgsbSolver<double>’ has no member
named ‘minimize’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:238:24: error: ‘x’ was not declared in this scope
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:243:12: error: ‘class cppoptlib::BfgsSolver<double>’ has no member named
‘minimize’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:243:24: error: ‘x’ was not declared in this scope
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:248:12: error: ‘class cppoptlib::GradientDescentSolver<double>’ has no
member named ‘minimize’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:248:24: error: ‘x’ was not declared in this scope
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:253:12: error: ‘class cppoptlib::ConjugatedGradientDescentSolver<double>’
has no member named ‘minimize’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:253:24: error: ‘x’ was not declared in this scope
In file included from /home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/gradientdescentsolver.h:6:0,
                 from /home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:10:
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h: In instantiation of ‘class
cppoptlib::ISolver<double, 2>’:
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/newtondescentsolver.h:13:7:   required from ‘class
cppoptlib::NewtonDescentSolver<double>’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:257:33:   required from here
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h:15:51: error: ‘double’ is not a class, struct, or
union type
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h:16:52: error: ‘double’ is not a class, struct, or
union type
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h:17:53: error: ‘double’ is not a class, struct, or
union type
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h:18:54: error: ‘double’ is not a class, struct, or
union type
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:258:12: error: ‘class cppoptlib::NewtonDescentSolver<double>’ has no
member named ‘minimize’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:258:24: error: ‘x’ was not declared in this scope
In file included from /home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/gradientdescentsolver.h:6:0,
                 from /home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:10:
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h: In instantiation of ‘class
cppoptlib::ISolver<double, 0>’:
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/neldermeadsolver.h:12:7:   required from ‘class
cppoptlib::NelderMeadSolver<double>’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:262:30:   required from here
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h:15:51: error: ‘double’ is not a class, struct, or
union type
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h:16:52: error: ‘double’ is not a class, struct, or
union type
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h:17:53: error: ‘double’ is not a class, struct, or
union type
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h:18:54: error: ‘double’ is not a class, struct, or
union type
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:263:12: error: ‘class cppoptlib::NelderMeadSolver<double>’ has no member
named ‘minimize’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:263:24: error: ‘x’ was not declared in this scope
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:268:12: error: ‘class cppoptlib::CMAesSolver<double>’ has no member named
‘minimize’
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:268:24: error: ‘x’ was not declared in this scope
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:277:38: error: ‘x’ was not declared in this scope
In file included from /home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/gradientdescentsolver.h:6:0,
                 from /home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:10:
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h: In instantiation of
‘cppoptlib::ISolver<ProblemType, Ord>::ISolver() [with ProblemType = double; int Ord = 1]’:
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/lbfgssolver.h:13:7:   required from here
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h:28:9: error: using invalid field
‘cppoptlib::ISolver<ProblemType, Ord>::m_stop’
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h:29:9: error: using invalid field
‘cppoptlib::ISolver<ProblemType, Ord>::m_current’
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h: In instantiation of
‘cppoptlib::ISolver<ProblemType, Ord>::ISolver() [with ProblemType = double; int Ord = 2]’:
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/newtondescentsolver.h:13:7:   required from here
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h:28:9: error: using invalid field
‘cppoptlib::ISolver<ProblemType, Ord>::m_stop’
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h:29:9: error: using invalid field
‘cppoptlib::ISolver<ProblemType, Ord>::m_current’
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h: In instantiation of
‘cppoptlib::ISolver<ProblemType, Ord>::ISolver() [with ProblemType = double; int Ord = 0]’:
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/neldermeadsolver.h:12:7:   required from here
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h:28:9: error: using invalid field
‘cppoptlib::ISolver<ProblemType, Ord>::m_stop’
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/isolver.h:29:9: error: using invalid field
‘cppoptlib::ISolver<ProblemType, Ord>::m_current’
In file included from /home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:16:0:
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/cmaessolver.h: In instantiation of
‘cppoptlib::CMAesSolver<TProblem>::CMAesSolver() [with TProblem = double]’:
/home/kevin/CppNumericalSolvers/matlab/cppoptlib.cxx:267:25:   required from here
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/cmaessolver.h:60:9: error: using invalid field
‘cppoptlib::CMAesSolver<TProblem>::m_stepSize’
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/cmaessolver.h:62:9: error: ‘m_stop’ is not a member of
‘cppoptlib::CMAesSolver<double>::Super {aka cppoptlib::ISolver<double, 1>}’
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/cmaessolver.h:63:9: error: ‘m_stop’ is not a member of
‘cppoptlib::CMAesSolver<double>::Super {aka cppoptlib::ISolver<double, 1>}’
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/cmaessolver.h:64:9: error: ‘m_stop’ is not a member of
‘cppoptlib::CMAesSolver<double>::Super {aka cppoptlib::ISolver<double, 1>}’
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/cmaessolver.h:65:9: error: ‘m_stop’ is not a member of
‘cppoptlib::CMAesSolver<double>::Super {aka cppoptlib::ISolver<double, 1>}’
/home/kevin/CppNumericalSolvers/matlab/../include/cppoptlib/solver/cmaessolver.h:66:9: error: ‘m_stop’ is not a member of
‘cppoptlib::CMAesSolver<double>::Super {aka cppoptlib::ISolver<double, 1>}’

Just write a working travis.yml

Still the same problem as in #32. The same test is still failing:

template <class T>
struct LinearFunc : public BoundedProblem<T, 2> {
    using typename Problem<T, 2>::TVector;

    LinearFunc() {
        this->setBoxConstraint(TVector::Zero(), TVector::Ones());
    }

    T value(const TVector& x) {
        return -x[0];
    }
};
TYPED_TEST(LbfgsbTest, LinearFunc)    { SOLVE_PROBLEM(cppoptlib::LbfgsbSolver,LinearFunc, 0.5, 0.5, -1.0) }

The minimum should be -1 for x_1=1. However the solver leaves the domain and yields as a result

The difference between -1.0 and f(x) is 45812984490, which exceeds 1e-4, where
-1.0 evaluates to -1,
f(x) evaluates to -45812984491, and
1e-4 evaluates to 0.0001.

Hi,

I am wondering what is the expected results from the executable benchmark. Naively, as I usually am, I would have expected if the code was correctly installed to have all tests passed. Unfortunately, on two machines - a Mac os x with a GNU g++ 5.2 and a centos 6 linux box with a GNU g++ 4.9.2 - I've got quite a lot of failed tests. I have read the README file, which states "... not all solvers are equally good at all objective functions ..." On the Mac os x, "48 tests from 8 test cases ran", 30 passed and 18 failed; on the linux box 31 passed and 17 failed.... Should I be worried??? Is this a normal (acceptable) result?

Sincerely
Max

https://github.com/PatWie/CppNumericalSolvers/blob/master/src/LbfgsbSolver.cpp#L61
should be hasbound_upper = true; not hasbound_lower = true;

Use LbfgsbSolver to test on Rosenbrock function:

void testLBFGSB()
{
    pwie::Vector x(2), lb(2), ub(2);
    x(0) = 0;
    x(1) = 0;
    lb(0) = -2; lb(1) = -2;
    ub(0) = 2; ub(1) = 2;
    pwie::LbfgsbSolver solver;
    solver.setLowerBound(lb);
    solver.setUpperBound(ub);
    solver.solve(x, rosenbrock, Drosenbrock);
    std::cout << "Results: " << x(0) << ' ' << x(1) << std::endl;
}

Since the bounds are set to [-2, 2], which cover the global minimum, the solution should still be similar to (1,1). However, the results returned are always the initial values.

Matrix H = Matrix::Identity(DIM, DIM);

Seems to defeat the purpose of limited memory BFGS and isn't it useless anyways?

As seen in #68, the newer version of Eigen has an extra catch for using empty matrices. As an explanation of what's going on, the following snippet works under 3.2 but not 3.3

MatrixXd m,v;
std::cout << m.lu().solve(v)<<std::endl;

Theoretically, we should be able to change the line here to be something like

if(v.size()!=0) v = N.lu().solve(v); 

but two tests still fail:
[ FAILED ] LbfgsbTest.RosenbrockMixFull
[ FAILED ] LbfgsbTest/0.RosenbrockFarFull, where TypeParam = float

Something else seems to have broken, at least for the parameters in the verify test.

Hi,

first of all, thank you for your library, it's really usefull in C++ context.

When compiling, I got some warnings and I dont' know if they are important or not :

The main warning comes from that line :

https://github.com/PatWie/CppNumericalSolvers/blob/master/include/cppoptlib/linesearch/morethuente.h#L99

I'm wondering if operators are used correctly ? Isn't || instead of | ? Is it the "OR" logical operator or the "OR" binary operator ?

The example in the current README does not compile, due to:

• The solver (BfgsSolver) should take as a template argument the problem class, not double.
• The Vector helper has been removed from meta.

This is frustrating to new users of the library.

Fixed example code:

#include <iostream>
#include "cppoptlib/meta.h"
#include "cppoptlib/problem.h"
#include "cppoptlib/solver/bfgssolver.h"

using namespace cppoptlib;
using Eigen::VectorXd;

class Rosenbrock : public Problem<double> {
  public:
    using typename cppoptlib::Problem<double>::Scalar;
    using typename cppoptlib::Problem<double>::TVector;

    double value(const TVector &x) {
        const double t1 = (1 - x[0]);
        const double t2 = (x[1] - x[0] * x[0]);
        return   t1 * t1 + 100 * t2 * t2;
    }
    void gradient(const TVector &x, TVector &grad) {
        grad[0]  = -2 * (1 - x[0]) + 200 * (x[1] - x[0] * x[0]) * (-2 * x[0]);
        grad[1]  = 200 * (x[1] - x[0] * x[0]);
    }
};

int main(int argc, char const *argv[]) {
    Rosenbrock f;
    BfgsSolver<Rosenbrock> solver;
    VectorXd x(2); x << -1, 2;
    solver.minimize(f, x);
    std::cout << "argmin      " << x.transpose() << std::endl;
    std::cout << "f in argmin " << f(x) << std::endl;
    return 0;
}

So that when we extend classes like ISolver and Problem, we can delete objects from a pointer to their base class.

This line cause a compilation error on OSX using clang :

https://github.com/PatWie/CppNumericalSolvers/blob/master/include/cppoptlib/meta.h#L43

Just a missing #include solve this.

First, thanks for this wonderful lib. Really helpful.
I clone , than run bfgs examples well. Than I change solver to neldermeadsolver, I got lots of compile erros, like:
Error C2146 syntax error: missing ')' before identifier 'initialSimplexCreated' test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 58
Error C2065 'not': undeclared identifier test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 58
Error C2059 syntax error: ')' test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 58
Error C2059 syntax error: ';' test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 59
Error C2065 'Scalar': undeclared identifier test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 63
Error C2923 'std::vector': 'Scalar' is not a valid template type argument for parameter '_Ty' test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 63
Error C3203 'allocator': unspecialized class template can't be used as a template argument for template parameter '_Alloc', expected a real type test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 63
Error C2133 'cppoptlib::f': unknown size test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 63
Error C2512 'std::vector': no appropriate default constructor available test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 63
Error C3927 '->': trailing return type is not allowed after a non-function declarator test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 63
Error C3484 syntax error: expected '->' before the return type test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 63
Error C3613 missing return type after '->' ('int' assumed) test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 63
Error C4430 missing type specifier - int assumed. Note: C++ does not support default-int test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 63
Error C2371 'cppoptlib::f': redefinition; different basic types test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 63
Error C2146 syntax error: missing ';' before identifier 'resize' test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 63
Error C3927 '->': trailing return type is not allowed after a non-function declarator test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 64
Error C3484 syntax error: expected '->' before the return type test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 64
Error C3613 missing return type after '->' ('int' assumed) test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 64
Error C4430 missing type specifier - int assumed. Note: C++ does not support default-int test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 64
Error C2371 'cppoptlib::index': redefinition; different basic types test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 64
Error C2146 syntax error: missing ';' before identifier 'resize' test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 64
Error C2059 syntax error: 'for' test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 65
Error C2143 syntax error: missing ')' before ';' test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 65
Error C2143 syntax error: missing ';' before '<' test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 65
Error C4430 missing type specifier - int assumed. Note: C++ does not support default-int test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 65
Error C2059 syntax error: '++' test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 65
Error C2059 syntax error: ')' test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 65
Error C2143 syntax error: missing ';' before '{' test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 65
Error C2447 '{': missing function header (old-style formal list?) test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 65
Error C2088 '[': illegal for class test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 70
Error C4430 missing type specifier - int assumed. Note: C++ does not support default-int test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 70
Error C2440 'initializing': cannot convert from 'cppoptlib::<lambda_7dea42c86a444d2b133737f325cb2fb5>' to 'int' test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 70
Error C2355 'this': can only be referenced inside non-static member functions or non-static data member initializers test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 73
Error C2227 left of '->m_stop' must point to class/struct/union/generic type test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 73
Error C2228 left of '.iterations' must have class/struct/union test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 73
Error C2065 'DIM': undeclared identifier test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 73
Error C2789 'cppoptlib::maxIter': an object of const-qualified type must be initialized test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 73
Error C2059 syntax error: 'while' test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 74
Error C2143 syntax error: missing ';' before '{' test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 77
Error C2447 '{': missing function header (old-style formal list?) test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 77
Error C3927 '->': trailing return type is not allowed after a non-function declarator test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 192
Error C3484 syntax error: expected '->' before the return type test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 192
Error C3613 missing return type after '->' ('int' assumed) test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 192
Error C4430 missing type specifier - int assumed. Note: C++ does not support default-int test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 192
Error C2146 syntax error: missing ';' before identifier 'detailed_callback' test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 192
Error C4430 missing type specifier - int assumed. Note: C++ does not support default-int test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 193
Error C2065 'x0': undeclared identifier test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 193
Error C2228 left of '.col' must have class/struct/union test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 193
Error C2088 '[': illegal for class test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 193
Error C2059 syntax error: '}' test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 194
Error C2143 syntax error: missing ';' before '}' test_vs2015 d:\dev_lib\new_folder\include\cppoptlib\solver\neldermeadsolver.h 194

I work with vs2015 and eigen3.3.3
Wired thing is I run neldermeadsolver well with a very old version of CppNumericalSolvers code.

Best regards

Hello,
Are there any plans to add a Levenberg-Marquardt type solver? I see the gradient descent and Newton descent solvers, but no LM solver.
Thanks,
Toby

Hi,

I tried to install the library with

mkdir build
cmake ..
cd build
sudo make install

That does not seem to work correctly. The output is

$ sudo make install
...
Install the project...
-- Install configuration: ""
-- Installing: /usr/local/bin/verify
-- Removed runtime path from "/usr/local/bin/verify"
-- Installing: /usr/local/bin/benchmark
-- Removed runtime path from "/usr/local/bin/benchmark"

I would expect that the headers will be copied to /usr/local/include.

I saw in the CMakeLists.txt that there is a custom target which points to a file cmake/install.cmake that does not exist:

add_custom_target(install COMMAND ${CMAKE_COMMAND} -P ${PROJECT_SOURCE_DIR}/cmake/install.cmake)

I wonder why there is no CMake error though.

The function I need to optimize is like a narrow hull with a well-defined global minimum. It is a 4-D surface, but its 2-D projections all look like a hull. In a previous experiment, everything worked well; the descent began with three parameters fixed at initial values (or almost fixed) and when it reached the minimum in the plane of initial descent, it made a few steps on either side of the plane and quickly turned toward the correct global minimum.

With slighly different data (can't say how different -- I forgot what data I used first, or what starting point), it is happy with the optimum it finds on the first approach and fails to look in other directions. I am pretty sure there is a healthy gradient remaining at that point. I know that because I have plotted the surface over a dense grid in that area and it seems to have a nice shape. What am I doing wrong?

Nelder-Mead is the only solver out of this bunch that converges on something. All others either fall into bizarre orbits or blow up after a few cycles.

Code and data at https://dl.dropboxusercontent.com/u/1725690/distort.tgz

I've made some cmake improvements, the tests and examples build now, too.
But I'm not able to create a decent PR because of the way Eigen is found:

The current stable Eigen has no config module, the tip has it but only the old-fashioned way of setting EIGEN_* variables. In my fork I modified Eigen to export a library target ::Eigen3 but it's not an official name.

I modified CppNumericalSolves's CMakeLists to support both ::Eigen3 and your FindEigen3.cmake, but it's not elegant. So I'm not going to make a PR but feel free to copy lines from https://github.com/adasworks/CppNumericalSolvers/blob/master/CMakeLists.txt if you find anything you need.

It looks like the L-BFGS-B solver doesn't work properly. I added a simple test case in my fork:

The problem is this check:

auto noConvergence =
    [&](Vector<Dtype> & x, Vector<Dtype> & g)->bool {
      return (((x - g).cwiseMax(lboundTemplate).cwiseMin(uboundTemplate) - x).template lpNorm<Eigen::Infinity>() >= 1e-4);
    };

I don't really understand what it does, but it looks like it's not correct.

To check if the current value is still inside the box, one could just use something like this: (here with tolerance)

auto outOfDomain =
    [&](Vector<Dtype> & x)->bool {
      return (uboundTemplate-x).minCoeff() > -1e-4 && (x-lboundTemplate).maxCoeff() > -1e-4;
    };

I guess the step/step-size in the algorithm is chosen s.t. the new position is still inside the box, so there must be an error in the algorithm implementation.

Hi,
Thnaks for this work.
It seems one has to evaluation a 'x' one at a time.

The only function in Problem we have access is :
virtual Scalar value(const TVector &x) = 0;

Could it be possible to allow the evaluation of multiple input combinations ?
Something like:
virtual TVector value(const Matrix &x) = 0;

The reason is that one can performs parallel evaluations of the function. In CMA-ES it seems relevant to use it no ?

I just change type of solver/objFunction/vectors from double to float in linear regression example and recieve several compile errors in files like problem.h/morethuente.h and etc.
So my question: is CppNumericalSolvers ready for using with float?

Most of these errors connected with constants and fabs calls in std::max/min. If it acceptably i can add (T) casts to constants and replace fabs by std::abs and make pull request (if i will have enough time).

PS Sorry for my poor English

Hello,
when trying to build the project on MAC OS X the following warning is generated:

CMake Warning (dev):
   Policy CMP0042 is not set: MACOSX_RPATH is enabled by default.  Run "cmake
   --help-policy CMP0042" for policy details.  Use the cmake_policy command to
   set the policy and suppress this warning.

   MACOSX_RPATH is not specified for the following targets:

    gtest

 This warning is for project developers.  Use -Wno-dev to suppress it.

To be honest, I don't understand @rpath at the moment, but thought you ought to know about it.

Thanks,
Toby

I find it often very confusing and inconvenient that the dimension is not a template argument of the problem. Wouldn't it make sense to add it there?

Then you don't always have to find out the dimension dynamically.

Adding this line to BUILD seems appropriate for building a (recently added?) example of a BoundedProblem:

build_example("nonnegls")

Would you consider adding support for google's bazel build system? It would make it easier to include this project as a dependency in other bazel projects

In the matlab examples, there is no L-BFGS-B binding. Can you add an example on that?

set(CMAKE_BUILD_TYPE Release)

would make everything much faster!

Mostly copy-paste from the old version

Motivated by the fork of @tesch1 I tried some time ago to implement autodiff, too in
https://github.com/PatWie/CppNumericalSolvers/blob/autodiff/src/examples/debug.cpp

However, I observed that there is currently no neat way to switch from finite difference to autodiff within a one-liner and no overhead. While this is really just

AutoDiff<RosenbrockObjective, T> f_autodiff_version;

instead of

Rosenbrock<T> f_default;

It would affect all data inside the problem. Given quadratic form x'Ax, even A currently has to be represented using the dual type.

Hello,
Thanks for you solver, really helpful!
I meet a strange error when I use visual studio 2013 64 win10 to complie the project.
The error message is like
Error 24 error C2666: 'Eigen::MatrixBase::operator ' : 4 overloads have similar conversions
And double click it lead you to neldermeadsolver.h - class NelderMeadSolver - member function minimize-line 92 const Vector x_r = ((1. + rho)) * x_bar - (rho) x0.col(index[DIM]);

I think Eigen is confused with the template form.However BFGS solver seems to function well which just I plan to use.

Hello and thank you for the library,

I am having a build issue on Ubuntu. g++ 4.8 and 5.4. Your help is appreciated.

In file included from /home/likewise-open/G3TI/scott.vincelette/Projects/TetleyScanner/PBL_1_0/CppNumericalSolvers/src/examples/nonnegls.cpp:4:0:
/home/likewise-open/G3TI/scott.vincelette/Projects/TetleyScanner/PBL_1_0/CppNumericalSolvers/src/examples/../../include/cppoptlib/solver/lbfgsbsolver.h: In member function ‘void cppoptlib::LbfgsbSolver::minimize(TProblem&, typename cppoptlib::LbfgsbSolver::Superclass::TVector&)’:
/home/likewise-open/G3TI/scott.vincelette/Projects/TetleyScanner/PBL_1_0/CppNumericalSolvers/src/examples/../../include/cppoptlib/solver/lbfgsbsolver.h:229:29: error: ‘TVector’ is not a class, namespace, or enumeration
TVector CauchyPoint = TVector::Zero(DIM);
^
make[2]: *** [src/examples/CMakeFiles/nonnegls.dir/nonnegls.cpp.o] Error 1
make[1]: *** [src/examples/CMakeFiles/nonnegls.dir/all] Error 2
make: *** [all] Error 2

Best Regards,

Scott V