argmin-rs / argmin Goto Github PK

View Code? Open in Web Editor NEW

896.0 14.0 73.0 44.38 MB

Numerical optimization in pure Rust

Home Page: http://argmin-rs.org

License: Apache License 2.0

Rust 99.97% Shell 0.03%

optimization-algorithms optimization rust numerical-optimization optimization-methods optimizer

argmin's Introduction

Mathematical optimization in pure Rust

argmin is a numerical optimization library written entirely in Rust.

argmins goal is to offer a wide range of optimization algorithms with a consistent interface. It is type-agnostic by design, meaning that any type and/or math backend, such as nalgebra or ndarray can be used -- even your own.

Observers allow one to track the progress of iterations, either by using one of the provided ones for logging to screen or disk or by implementing your own.

An optional checkpointing mechanism helps to mitigate the negative effects of crashes in unstable computing environments.

Due to Rusts powerful generics and traits, most features can be exchanged by your own tailored implementations.

argmin is designed to simplify the implementation of optimization algorithms and as such can also be used as a toolbox for the development of new algorithms. One can focus on the algorithm itself, while the handling of termination, parameter vectors, populations, gradients, Jacobians and Hessians is taken care of by the library.

Algorithms

Line searches
- Backtracking line search
- More-Thuente line search
- Hager-Zhang line search
Trust region method
- Cauchy point method
- Dogleg method
- Steihaug method
Steepest descent
Conjugate gradient method
Nonlinear conjugate gradient method
Newton methods
- Newton’s method
- Newton-CG
Quasi-Newton methods
- BFGS
- L-BFGS
- DFP
- SR1
- SR1-TrustRegion
Gauss-Newton method
Gauss-Newton method with linesearch
Golden-section search
Landweber iteration
Brent’s method
Nelder-Mead method
Simulated Annealing
Particle Swarm Optimization

External solvers compatible with argmin

External solvers which implement the Solver trait are compatible with argmins Executor, and as such can leverage features like checkpointing and observers.

License

Licensed under either of

Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
MIT License (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

argmin's People

Contributors

Stargazers

Watchers

argmin's Issues

Add WASM test pipline to CI

argmin should be tested in a WASM environment.

Make errors public

Background

Hi there!! Thank you for the great work!!

I am starting to use the crate and run into the following problem:
Implementing ArgminOp, the method apply, I would like to correctly handle errors. Then, I need to return an argmin::core::Error, but it is private. I even tried with the suitable variant of argmin::core::ArgminError, but it is also private.

Request

Make errors public and easy to construct. And add the corresponding indications in the documentation when implementing the ArgminOp trait.

Example

I would like for something like this to work.

use argmin::prelude::*;
use serde::{Deserialize, Serialize};

#[derive(Clone, Default, Serialize, Deserialize)]
pub struct MyProblem; 

impl ArgminOp for MyProblem {
    type Param = ();
    type Output = ();
    type Hessian = ();
    type Jacobian = ();
    type Float = ();

    /// Apply the cost function to a parameter `p`
    fn apply(&self, _p: &Self::Param) -> Result<Self::Output, Error> {
        argmin::core::ArgminError::InvalidParameter{text: String::from("MyProblem does not take this parameters")}
        // or better
        //  argmin::error::ArgminError::InvalidParameter::new("Error message")
    }
}

Use builder pattern to simplify the API

Hey, thanks for creating and maintaining this crate :) I only glanced over the examples, but wondered why you don't use the builder pattern. It could simplify setting up a new solver a bit, otherwise the crate is looking really useful. Greetings from Aachen.

Release new version

Now that we have nalgebra support thanks to @Maher4Ever (#68), I think it is time to release a new version.

Examples fail to build

I'm trying to build the master version of this crate and run test and am seeing some failures in examples so far,

$ cargo --version
cargo 1.35.0 (6f3e9c367 2019-04-04)
$ cargo test
cargo test                                                                                                 [tags/v0.2.4] 
   Compiling argmin v0.2.4 (/home/rth/src/argmin)
error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin::prelude::ArgminMul<f64, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not satisfied
  --> examples/trustregion_nd.rs:66:15
   |
66 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin::prelude::ArgminMul<f64, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`
   |
   = note: required because of the requirements on the impl of `argmin::prelude::Solver<Rosenbrock>` for `argmin::solver::trustregion::TrustRegion<argmin::solver::trustregion::Dogleg>`
   = note: required by `argmin::prelude::Executor::<O, S>::new`
[...]

error: aborting due to 21 previous errors

Some errors occurred: E0277, E0599.
For more information about an error, try `rustc --explain E0277`.
error: aborting due to 9 previous errors

Some errors occurred: E0277, E0599.
For more information about an error, try `rustc --explain E0277`.
error: Could not compile `argmin`.
warning: build failed, waiting for other jobs to finish...
error: Could not compile `argmin`.
warning: build failed, waiting for other jobs to finish...
error: build failed

see full output below,

cargo test                                                                                                 [tags/v0.2.4] 
   Compiling argmin v0.2.4 (/home/rth/src/argmin)
error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin::prelude::ArgminMul<f64, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not satisfied
  --> examples/trustregion_nd.rs:66:15
   |
66 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin::prelude::ArgminMul<f64, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`
   |
   = note: required because of the requirements on the impl of `argmin::prelude::Solver<Rosenbrock>` for `argmin::solver::trustregion::TrustRegion<argmin::solver::trustregion::Dogleg>`
   = note: required by `argmin::prelude::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin::prelude::ArgminNorm<f64>` is not satisfied
  --> examples/trustregion_nd.rs:66:15
   |
66 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin::prelude::ArgminNorm<f64>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`
   |
   = note: required because of the requirements on the impl of `argmin::prelude::Solver<Rosenbrock>` for `argmin::solver::trustregion::TrustRegion<argmin::solver::trustregion::Dogleg>`
   = note: required by `argmin::prelude::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin::prelude::ArgminDot<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, f64>` is not satisfied
  --> examples/trustregion_nd.rs:66:15
   |
66 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin::prelude::ArgminDot<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, f64>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`
   |
   = note: required because of the requirements on the impl of `argmin::prelude::Solver<Rosenbrock>` for `argmin::solver::trustregion::TrustRegion<argmin::solver::trustregion::Dogleg>`
   = note: required by `argmin::prelude::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin::prelude::ArgminAdd<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not satisfied
  --> examples/trustregion_nd.rs:66:15
   |
66 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin::prelude::ArgminAdd<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`
   |
   = note: required because of the requirements on the impl of `argmin::prelude::Solver<Rosenbrock>` for `argmin::solver::trustregion::TrustRegion<argmin::solver::trustregion::Dogleg>`
   = note: required by `argmin::prelude::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin::prelude::ArgminSub<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not satisfied
  --> examples/trustregion_nd.rs:66:15
   |
66 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin::prelude::ArgminSub<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`
   |
   = note: required because of the requirements on the impl of `argmin::prelude::Solver<Rosenbrock>` for `argmin::solver::trustregion::TrustRegion<argmin::solver::trustregion::Dogleg>`
   = note: required by `argmin::prelude::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin::prelude::ArgminZeroLike` is not satisfied
  --> examples/trustregion_nd.rs:66:15
   |
66 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin::prelude::ArgminZeroLike` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`
   |
   = note: required because of the requirements on the impl of `argmin::prelude::Solver<Rosenbrock>` for `argmin::solver::trustregion::TrustRegion<argmin::solver::trustregion::Dogleg>`
   = note: required by `argmin::prelude::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>: argmin::prelude::ArgminDot<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not satisfied
  --> examples/trustregion_nd.rs:66:15
   |
66 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin::prelude::ArgminDot<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>`
   |
   = note: required because of the requirements on the impl of `argmin::prelude::Solver<Rosenbrock>` for `argmin::solver::trustregion::TrustRegion<argmin::solver::trustregion::Dogleg>`
   = note: required by `argmin::prelude::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>: argmin::prelude::ArgminInv<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>>` is not satisfied
  --> examples/trustregion_nd.rs:66:15
   |
66 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin::prelude::ArgminInv<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>`
   |
   = note: required because of the requirements on the impl of `argmin::prelude::Solver<argmin::prelude::OpWrapper<Rosenbrock>>` for `argmin::solver::trustregion::Dogleg`
   = note: required because of the requirements on the impl of `argmin::prelude::Solver<Rosenbrock>` for `argmin::solver::trustregion::TrustRegion<argmin::solver::trustregion::Dogleg>`
   = note: required by `argmin::prelude::Executor::<O, S>::new`

error[E0599]: no method named `forward_diff` found for type `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>` in the current scope
  --> examples/sr1_trustregion.rs:36:17
   |
36 |         Ok((*p).forward_diff(&|x| rosenbrock(&x.to_vec(), self.a, self.b)))
   |                 ^^^^^^^^^^^^

error[E0599]: no method named `forward_hessian` found for type `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>` in the current scope
  --> examples/sr1_trustregion.rs:45:17
   |
45 |         Ok((*p).forward_hessian(&|x| self.gradient(&x).unwrap()))
   |                 ^^^^^^^^^^^^^^^

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin_core::math::ArgminSub<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not satisfied
  --> examples/sr1_trustregion.rs:69:15
   |
69 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin_core::math::ArgminSub<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`
   |
   = note: required because of the requirements on the impl of `argmin_core::Solver<Rosenbrock>` for `argmin::solver::quasinewton::SR1TrustRegion<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, argmin::solver::trustregion::Steihaug<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>>`
   = note: required by `argmin_core::executor::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin_core::math::ArgminAdd<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not satisfied
  --> examples/sr1_trustregion.rs:69:15
   |
69 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin_core::math::ArgminAdd<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`
   |
   = note: required because of the requirements on the impl of `argmin_core::Solver<Rosenbrock>` for `argmin::solver::quasinewton::SR1TrustRegion<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, argmin::solver::trustregion::Steihaug<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>>`
   = note: required by `argmin_core::executor::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin_core::math::ArgminDot<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, f64>` is not satisfied
  --> examples/sr1_trustregion.rs:69:15
   |
69 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin_core::math::ArgminDot<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, f64>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`
   |
   = note: required because of the requirements on the impl of `argmin_core::Solver<Rosenbrock>` for `argmin::solver::quasinewton::SR1TrustRegion<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, argmin::solver::trustregion::Steihaug<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>>`
   = note: required by `argmin_core::executor::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin_core::math::ArgminDot<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>>` is not satisfied
  --> examples/sr1_trustregion.rs:69:15
   |
69 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin_core::math::ArgminDot<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`
   |
   = note: required because of the requirements on the impl of `argmin_core::Solver<Rosenbrock>` for `argmin::solver::quasinewton::SR1TrustRegion<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, argmin::solver::trustregion::Steihaug<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>>`
   = note: required by `argmin_core::executor::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin_core::math::ArgminNorm<f64>` is not satisfied
  --> examples/sr1_trustregion.rs:69:15
   |
69 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin_core::math::ArgminNorm<f64>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`
   |
   = note: required because of the requirements on the impl of `argmin_core::Solver<Rosenbrock>` for `argmin::solver::quasinewton::SR1TrustRegion<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, argmin::solver::trustregion::Steihaug<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>>`
   = note: required by `argmin_core::executor::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin_core::math::ArgminZeroLike` is not satisfied
  --> examples/sr1_trustregion.rs:69:15
   |
69 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin_core::math::ArgminZeroLike` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`
   |
   = note: required because of the requirements on the impl of `argmin_core::Solver<Rosenbrock>` for `argmin::solver::quasinewton::SR1TrustRegion<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, argmin::solver::trustregion::Steihaug<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>>`
   = note: required by `argmin_core::executor::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin_core::math::ArgminMul<f64, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not satisfied
  --> examples/sr1_trustregion.rs:69:15
   |
69 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin_core::math::ArgminMul<f64, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`
   |
   = note: required because of the requirements on the impl of `argmin_core::Solver<Rosenbrock>` for `argmin::solver::quasinewton::SR1TrustRegion<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, argmin::solver::trustregion::Steihaug<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>>`
   = note: required by `argmin_core::executor::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>: argmin_core::math::ArgminSub<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>>` is not satisfied
  --> examples/sr1_trustregion.rs:69:15
   |
69 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin_core::math::ArgminSub<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>`
   |
   = note: required because of the requirements on the impl of `argmin_core::Solver<Rosenbrock>` for `argmin::solver::quasinewton::SR1TrustRegion<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, argmin::solver::trustregion::Steihaug<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>>`
   = note: required by `argmin_core::executor::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>: argmin_core::math::ArgminDot<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not satisfied
  --> examples/sr1_trustregion.rs:69:15
   |
69 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin_core::math::ArgminDot<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>`
   |
   = note: required because of the requirements on the impl of `argmin_core::Solver<Rosenbrock>` for `argmin::solver::quasinewton::SR1TrustRegion<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, argmin::solver::trustregion::Steihaug<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>>`
   = note: required by `argmin_core::executor::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>: argmin_core::math::ArgminDot<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>>` is not satisfied
  --> examples/sr1_trustregion.rs:69:15
   |
69 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin_core::math::ArgminDot<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>`
   |
   = note: required because of the requirements on the impl of `argmin_core::Solver<Rosenbrock>` for `argmin::solver::quasinewton::SR1TrustRegion<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, argmin::solver::trustregion::Steihaug<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>>`
   = note: required by `argmin_core::executor::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>: argmin_core::math::ArgminAdd<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>>` is not satisfied
  --> examples/sr1_trustregion.rs:69:15
   |
69 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin_core::math::ArgminAdd<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>`
   |
   = note: required because of the requirements on the impl of `argmin_core::Solver<Rosenbrock>` for `argmin::solver::quasinewton::SR1TrustRegion<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, argmin::solver::trustregion::Steihaug<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>>`
   = note: required by `argmin_core::executor::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>: argmin_core::math::ArgminMul<f64, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>>` is not satisfied
  --> examples/sr1_trustregion.rs:69:15
   |
69 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `argmin_core::math::ArgminMul<f64, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>`
   |
   = note: required because of the requirements on the impl of `argmin_core::Solver<Rosenbrock>` for `argmin::solver::quasinewton::SR1TrustRegion<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, argmin::solver::trustregion::Steihaug<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>>`
   = note: required by `argmin_core::executor::Executor::<O, S>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin_core::math::ArgminMul<f64, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not satisfied
  --> examples/sr1_trustregion.rs:61:38
   |
61 |     let subproblem = Steihaug::new().max_iters(20);
   |                                      ^^^^^^^^^ the trait `argmin_core::math::ArgminMul<f64, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin_core::math::ArgminDot<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, f64>` is not satisfied
  --> examples/sr1_trustregion.rs:61:38
   |
61 |     let subproblem = Steihaug::new().max_iters(20);
   |                                      ^^^^^^^^^ the trait `argmin_core::math::ArgminDot<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, f64>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin_core::math::ArgminAdd<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not satisfied
  --> examples/sr1_trustregion.rs:61:38
   |
61 |     let subproblem = Steihaug::new().max_iters(20);
   |                                      ^^^^^^^^^ the trait `argmin_core::math::ArgminAdd<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin_core::math::ArgminMul<f64, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not satisfied
  --> examples/sr1_trustregion.rs:61:22
   |
61 |     let subproblem = Steihaug::new().max_iters(20);
   |                      ^^^^^^^^^^^^^ the trait `argmin_core::math::ArgminMul<f64, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`
   |
   = note: required by `argmin::solver::trustregion::Steihaug::<P>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin_core::math::ArgminDot<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, f64>` is not satisfied
  --> examples/sr1_trustregion.rs:61:22
   |
61 |     let subproblem = Steihaug::new().max_iters(20);
   |                      ^^^^^^^^^^^^^ the trait `argmin_core::math::ArgminDot<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, f64>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`
   |
   = note: required by `argmin::solver::trustregion::Steihaug::<P>::new`

error[E0277]: the trait bound `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>: argmin_core::math::ArgminAdd<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not satisfied
  --> examples/sr1_trustregion.rs:61:22
   |
61 |     let subproblem = Steihaug::new().max_iters(20);
   |                      ^^^^^^^^^^^^^ the trait `argmin_core::math::ArgminAdd<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>, ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>` is not implemented for `ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>`
   |
   = note: required by `argmin::solver::trustregion::Steihaug::<P>::new`

error[E0599]: no method named `add_observer` found for type `argmin_core::executor::Executor<Rosenbrock, argmin::solver::quasinewton::SR1TrustRegion<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, argmin::solver::trustregion::Steihaug<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>>>` in the current scope
  --> examples/sr1_trustregion.rs:70:10
   |
70 |         .add_observer(ArgminSlogLogger::term(), ObserverMode::Always)
   |          ^^^^^^^^^^^^
   |
   = note: the method `add_observer` exists but the following trait bounds were not satisfied:
           `argmin::solver::quasinewton::SR1TrustRegion<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>>, argmin::solver::trustregion::Steihaug<ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 1]>>>> : argmin_core::Solver<Rosenbrock>`

error[E0599]: no method named `add_observer` found for type `argmin::prelude::Executor<Rosenbrock, argmin::solver::trustregion::TrustRegion<argmin::solver::trustregion::Dogleg>>` in the current scope
  --> examples/trustregion_nd.rs:67:10
   |
67 |         .add_observer(ArgminSlogLogger::term(), ObserverMode::Always)
   |          ^^^^^^^^^^^^
   |
   = note: the method `add_observer` exists but the following trait bounds were not satisfied:
           `argmin::solver::trustregion::TrustRegion<argmin::solver::trustregion::Dogleg> : argmin::prelude::Solver<Rosenbrock>`

warning: unused import: `argmin_core::finitediff`
  --> examples/sr1_trustregion.rs:15:5
   |
15 | use argmin_core::finitediff::*;
   |     ^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: #[warn(unused_imports)] on by default

error: aborting due to 21 previous errors

Some errors occurred: E0277, E0599.
For more information about an error, try `rustc --explain E0277`.
error: aborting due to 9 previous errors

Some errors occurred: E0277, E0599.
For more information about an error, try `rustc --explain E0277`.
error: Could not compile `argmin`.
warning: build failed, waiting for other jobs to finish...
error: Could not compile `argmin`.
warning: build failed, waiting for other jobs to finish...
error: build failed

ParticleSwarm usage

Hi! I am very excited to try out the ParticleSwarm optimizer. It seems however that it only provides 2 values to the objective function (presumably x/y coordinates). Would it be possible to use it for higher dimensional optimization?

Hide Serialization/Deserialization behind feature gate

Currently Serialization and Deserialization is implemented for all types which get serialized when creating a checkpoint. It would be convenient if this was optional by hiding it behind a feature gate.

Performance Enhancements

First of all, I'd like to thank everyone involved in making this crate. I've been using it in multiple places in a project of mine for the past few months, and I love the ergonomics and find the crate a joy to use - Kudos!

Anyway, I was doing some performance tuning on my code recently and discovered that in my calls to Executor::run, only about 44% of processor time was spent inside my ArgminOp::apply. When looking at where the rest of the time was being spent, I noticed three main areas:

alloc::fmt::format, ArgminKV::merge, and drop ArgminKV. This would be expected if I was using an observer, but this overhead happens even with no registered observers.
Instant::elapsed and Instant::now. These are called even if I don't care about the elapsed time (as is often the case once I've finished tuning the optimisation params)`.
ArgminCheckpoint::store_cond. This allocates a string, concats it with another string, and deallocs the result on exit - every single time through the Executor::run loop, even if Checkpoints are off, or this loop is not a checkpointed loop.

Here's a section of my flamechart from cargo-flamechart to illustrate this.
Green boxes highlight my code inside the apply function, magenta is for point (1) above, red for point (2), and blue for point (3)

I've got a branch in my fork where I've addressed points (1) and (3) by some refactoring, which I don't think will cause any issues (but you all know better than me, I may be wrong!). I've also addressed (2) by introducing a config flag to make the timer optional, similarly to how enabling/disabling Ctrl-C works. You can see my changes here: sdd@a67f56a

With the modifications above in place, I get a 44% speedup, and I can see that the executor now spends 93% of its time running my ArgminOp::apply, rather than 44% of the time. My cargo flamechart now looks more like this:

I realize that my workload may be atypical to many others, as it might be quite light. I'm really just doing some matrix and trig operations. But, even with intensive apply functions, some performace gains are there to be had for others.

I'm happy to raise a PR and whip it into the kind of shape you'd expect it to be in if you think this is a worthwhile change.

Thanks!

Remove dependency on `failure` and implement custom error types

It seems as if failure may not be the best choice. Moving to custom error types is however probably quite a big change.

CMA-ES

Hello,
CMA-ES is a powerful evolutionary algorithm for black-box optimization problems. If there is interest in adding it to this optimization suite, I would be happy to give it a try. My plan is to start by following the purecma.py implementation, a public domain pedagogical implementation by the author of the algorithm, and add more advanced features if necessary.

Remove the WIP from project title

I think it would be good to remove the WIP tag from the project title.

I understand that things can (always) be improved, and some API refactoring can still be planned but I think that a 2 yo project with some amount of users is no longer "Work in progress". Just because it implies that this is somehow less stable than any other actively developed project.

Implement Brent minimization method

Hello,

Unless I am mistaken, Brent method is the name for two algorithms: one for root-finding and one for minimization. They are respectively described under the names zero and localmin in chapters 4 and 5 of the book Algorithms for Minimization without Derivatives by Brent (1973). Although they share the same ideas, the first is not directly applicable to the second problem unless we know the derivative of the function to optimize.

It seems that the function implemented in argmin is Brent's root-finding method (currently the only one described on Wikipedia). Hence I think that we are still missing Brent's minimization method. I would like to implement the minimization method, but my question is what to do of the root-finding method already present? Is root-finding one of the purposes of argmin? In which case, maybe would it be appropriate to separate root-finding and optimization algorithms, since those are two different kinds of problems?

cc @xemwebe

ParticleSwarm::new has incorrect parameter documentation

argmin/src/solver/particleswarm/mod.rs

Lines 46 to 51 in 1cf4523

 /// Parameters: 

 /// 

 /// * `cost_function`: cost function 

 /// * `init_temp`: initial temperature 

 pub fn new( 

 search_region: (P, P),

Checkpointing should be configurable by user

Currently checkpoints can only be written to disk. It would be better if users could write their own checkpointing mechanisms. In order to do so, a checkpointing trait should be defined. This could be implemented similar to observers.

Allow users to create stopping criteria

Currently, stopping criteria are defined during the implementation of the solver. However, sometimes it might be useful if users could add their own stopping criteria. This would require a complete redesign of how termination works. One of the key aspects of termination is that it needs access to the fields of the solver which makes it difficult to implement in a generic way. However, similar to how observers work, the needed variables could be stored in a key-value store and passed to the user-defined termination function. It needs to be evaluated how efficient such an approach would be.

Ndarray/Ndarray-rand versions mismatch

As I try to upgrade linfa to ndarray 0.14 (cf. rust-ml/linfa#110), I noticed that argmin 0.4.2 has ndarray 0.14 and ndarray-rand 0.14 dependencies.

Unfortunately ndarray-rand 0.14 depends on ndarray 0.15, it should be ndarray-rand 0.13 to depend onndarray 0.14.

It could explain the compilation problems I get (?) Strangely even if I specify argmin 0.4.1 it pulls argmin 0.4.2 (not sure what is going on here!).

I would suggest to release argmin 0.4.3 reverting ndarray-rand to 0.13.

Rethink return type of executors

Currently, running an Executor returns an ArgminResult. While this holds the result of the optimization, it may give the impression of being related to error handling.

In addition, it is unnecessarily difficult to extract the actual result of the optimization as one needs to access the state field of type IterState. None of this is properly documented.

ArgminResult seems to be unnecessary, as one can just return IterState instead. But maybe there is an even better option.

Avoid copying operator in `OpWrapper`

When creating an instance of OpWrapper, the passed in operator is cloned. This can be inefficient (particularly if a solver is called inside a solver, which means at least one copy of the operator per iteration of the "outer" solver) and requires the operators to implement Clone. Removing the clone may however necessitate a change of the overall design.

Error handling and dealing with `NaN`s in `MoreThuenteLineSearch`

This is related to #38.

Prior to #38, NaNs (wherever they may come from) would cause MoreThuenteLineSearch to panic. Since #38, NaNs will be ignored. The appropriate way to handle such a case would be to check for NaNs and throw an error, so that this case can be handled by the user.

Error running solver in a webassembly environment

Calling the function Executor::run from web assembly panics with the following error:

test result: FAILED. 1 passed; 1 failed; 0 ignored
console.log div contained:
    panicked at 'time not implemented on this platform', library/std/src/sys/wasm/../unsupported/time.rs:13:9

This is probably because of the use of std::time, which is not supported in a browser.

To reproduce the error follow these steps:

clone this repository: https://github.com/giacomo-dantonio/argmin-wasm-test
build the package using wasm-pack build
run the tests using wasm-pack test

You need to install wasm-pack, if you don't have it already.

Rename `master` branch to `main`

Rename master branch to main

Constrained Optimization (especially lower+upper bounds)

Hello,
Coming from Python (scipy.optimize.minimize) and MATLAB (fmincon), I can't find a way to specify bounds (and other constraints), which to me are very essential parts of an optimization problem.

Python / Scipy:

res = minimize(opti_fun, x0, bounds=mybounds, constraints=myconstraints)

Matlab:

res = fmincon(opti_fun, x0, A, b, Aeq, beq, lowerbound, upperbound)

Looking through the documentation, I could not find that this is implemented. (With the exception of some manual bounds in the simulatedannealing.rs example).
Is this something that is being worked on? Or is it planned? Or is it too complicated / time consuming at the moment? Or did I just miss something here?
I would love to have constrained optimization (at least with bounds) in Rust!

Bug in HagerZhangLineSearch

There seems to be a bug in HagerZhangLineSearch (for instance, running the steepestdescent or nonlinear_cg examples with HagerZhangLineSearch will not converge towards the solution).

cargo build --examples fails

After I clone the repo, I can run cargo build and it's find, but to cargo build examples, there are lots of errors:

error[E0277]: the trait bound `ArrayBase<OwnedRepr<f64>, Dim<[usize; 1]>>: ArgminSub<ArrayBase<OwnedRepr<f64>, Dim<[usize; 1]>>, ArrayBase<OwnedRepr<f64>, Dim<[usize; 1]>>>` is not satisfied
  --> examples/bfgs.rs:53:15
   |
53 |     let res = Executor::new(cost, solver, init_param)
   |               ^^^^^^^^^^^^^ the trait `ArgminSub<ArrayBase<OwnedRepr<f64>, Dim<[usize; 1]>>, ArrayBase<OwnedRepr<f64>, Dim<[usize; 1]>>>` is not implemented for `ArrayBase<OwnedRepr<f64>, Dim<[usize; 1]>>`
   |
   = note: required because of the requirements on the impl of `Solver<Rosenbrock>` for `BFGS<MoreThuenteLineSearch<ArrayBase<OwnedRepr<f64>, Dim<[usize; 1]>>, f64>, ArrayBase<OwnedRepr<f64>, Dim<[usize; 2]>>, f64>`
   = note: required by `argmin::core::Executor::<O, S>::new`

Bump ndarray version to 0.15

The ndarray < 0.15 requirement currently blocks rust-ml/linfa#124. This is currently blocked on https://github.com/rust-ndarray/ndarray-linalg which still has to be published their latest version to crates.io

Warnings when using `nalgebral` feature because of deprecated type alias `MatrixMN`

When compiling argmin with the nalgebral feature, one currently gets many warnings of the kind:

warning: use of deprecated type alias `nalgebra::MatrixMN`: use SMatrix for a statically-sized matrix using integer dimensions, or OMatrix for an owned matrix using types as dimensions.
  --> src/core/math/zero_nalgebra.rs:17:34
   |
17 | impl<N, R, C> ArgminZeroLike for MatrixMN<N, R, C>
   |                                  ^^^^^^^^^^^^^^^^^

As someone who hasn't used nalgebra yet, I'm unsure which way to go. Is SMatrix or OMatrix a more appropriate replacement for MatrixMN? Or should the math traits be implemented for both? Any help by nalgebra users is highly appreciated :)

Adjust the stopping criterion

Currently the stopping criterion tolerance is often hard coded. For instance in LBFGS,

argmin/src/solver/quasinewton/lbfgs.rs

Lines 166 to 172 in 46f43cd

 fn terminate(&mut self, state: &IterState<O>) -> TerminationReason { 

 if state.get_grad().unwrap().norm() < std::f64::EPSILON.sqrt() { 

 return TerminationReason::TargetPrecisionReached; 

 } 

 if (state.get_prev_cost() - state.get_cost()).abs() < std::f64::EPSILON { 

 return TerminationReason::NoChangeInCost; 

 }

the stopping criterion for the gradient is std::f64::EPSILON.sqrt() ~ 1e-8 while in practical machine learning application 1e-4 could be sufficient, as performance matter more than perfect convergence .

So the question is what API would allow for these stopping conditions to be set by the user. For instance, it could be some combination of,
a) add tol to the struct initialization in new. That's probably not ideal, as for instance LBFGS would require 2 tolerances, and always providing them manually is not great.
b) add tol to the struct but not as an argument in new method
c) implement a Default trait with current values.
d) some form of builder pattern or a method e.g. with_tol.

For now I would likely tend to propose b) with d) using a with_tol or just tol method (which also would be backward compatible), and I'm looking for some feedback. I'm not sure what would best generalize to different solvers. Also ideally the solution should also be consistent with Executor.max_iters approach.

NelderMead requires Default but doesn't need it

I want to use the nalgebra support for argmin in master, but I am having some difficulties. Specifically, I want to use NelderMead optimization, but I have found that it requires the param to be Default. I am using nalgebra::DMatrix. DMatrix does not implement Default. I looked at the implementation of NelderMead, and it seems that it does not utilize the Default bound at all. Is it possible to remove this bound so that I can use DMatrix with it. Thank you!

Need a nudge in the right direction!

I am pretty experienced with Rust, but I have no idea how to use this, although I think if someone could just tell me how to set it up I would be golden. This seems like the exact thing I need for the job, but I can't get past the nomenclature.

I'm working on a tool that produces a GIF with @kornelski's most excellent Gifski tool. The problem is finding the right parameters in order to get as close to 15 MB without going over.

Variables:

GIF encoder (these parameters can change)
- Quality level
- FPS
- Output dimensions
Video clip (these parameters can't change)
- Length
- Dimensions
Output (what I'm trying to optimize for):
- File size

I may also use butteraugli metrics as extra data points to feed in. Was surprisingly easy to compile given their recommendation to give Bazel another try (I'll pass).

Anyways, where should I start with argmin on a problem like this? I have a bunch of variables I want to tweak in order to maximize a desired outcome. Right now I've been doing alchemy by changing weights and biases for things and tweaking formulas on my own and I would definitely prefer a more general solution. I'll keep re-reading the docs and hope I have an epiphany just in case I don't hear back ;)

Also also...thanks for the hard work to all involved! Seems like a great project, just wish I knew how to wrap my head around it, haha!

Option to set append or truncate for ArgminSlogLogger

Current behavior of ArgminSlogLogger::file is to truncate the log file on each run.

Sometimes I want to see the logs of a previous run, particularly if using checkpoints,

Proposal: Checkpointing only for `Executor`s internal state and solver

Checkpointing currently serializes all of the Executor, which requires the ArgminOp to be serializable (even if no checkpointing is used). However, one actually only needs to serialize the state and the solver field, the other things can be supplied when starting from a loaded checkpoint.

So instead of

let res = Executor::from_checkpoint(".checkpoints/landweber_exec.arg")
        .unwrap_or(Executor::new(operator, solver, init_param))
        .max_iters(iters)
        .checkpoint_dir(".checkpoints")
        .checkpoint_name("landweber_exec")
        .checkpoint_mode(CheckpointMode::Every(20))
        .add_observer(ArgminSlogLogger::term(), ObserverMode::Always)
        .run()?;

One could write this like this:

let res = Executor::from_checkpoint(".checkpoints/landweber_exec.arg", &operator)
        .unwrap_or(Executor::new(operator, solver, init_param))
        .max_iters(iters)
        .checkpoint_dir(".checkpoints")
        .checkpoint_name("landweber_exec")
        .checkpoint_mode(CheckpointMode::Every(20))
        .add_observer(ArgminSlogLogger::term(), ObserverMode::Always)
        .run()?;

This would, to a large extent, solve #7 .

Caveat:

Parameters still need to be serializable (No way around that I guess)
Operators are not allowed to have internal state which changes over the course of the optimization because it will not be checkpointed (not sure if this really is an issue)

Since solvers may have internal state, they need to be part of the checkpointing.

I'm open to suggestions.

Parallel evaluations of cost function

Some optimization algorithms require several independent cost function evaluations per iteration. Sometimes (actually most of the times in my applications), evaluating the cost function can be costly, and its evaluations dominate the optimization time. When the cost function itself cannot be parallelized, an obvious improvement is to evaluate it at these points in parallel. It could also be desirable to evaluate the cost function and its gradient or hessian in parallel when applicable.

rayon is the perfect library to allow this kind of things. Would it be okay to add it as a dependency? As a feature? Should we create an option to enable parallel evaluation on demand? We could probably even decide automatically whether it is worth to enable it or not, depending on the duration of the first few evaluations.

Preconditioner

One of the method I coded in Rust is the "Conjugate gradient solver for positive-definite matrices." I see that you already coded this method. I didn't compare them yet because there's a small difference: I'm using a diagonal conditioner

for idx_iter in 1..max_iterations {
    // We're using the diagonal conditioner.
    let z = &r / &diag;
    ... standard algo, I think ...

As you know, I'm not a mathematician at all, so I can't explain you what this is and why we need it in argmin. Maybe almost nobody uses that and we shouldn't code it. I don't know. This looks like a new trait and a new template parameter, something you may be tired of :)

Is there a minimum-effort way to compare our algo? Like, using what you already coded to update the array or some magic that you know?

Improve clippy linting in CI

Use the following as suggested in #136 :

cargo clippy --all-targets --all-features -- -D warnings

XorShiftRng has poor statistical quality

rand_xorshift::XorShiftRng uses Xorshift128 (also called xor128) which is from Marsaglia's original paper. It has significant weaknesses ([0][1]). It was replaced as rand's default noncryptographic PRNG in 2018.

Stochastic optimization requires high quality random numbers and Xorshift128 does not provide that.

argmin could use a sensible default like rand's current xoshiro256++ or even allow users to select one via traits (I'm willing to write a PR for that).

Road to 1.0.0

This is a long-term issue in order to keep these things in mind.

Naming (crate aligns with Rust naming conventions)
- Casing conforms to RFC 430 (C-CASE)
- Ad-hoc conversions follow as_, to_, into_ conventions (C-CONV)
- Getter names follow Rust convention (C-GETTER)
- Methods on collections that produce iterators follow iter, iter_mut, into_iter (C-ITER)
- Iterator type names match the methods that produce them (C-ITER-TY)
- Feature names are free of placeholder words (C-FEATURE)
- Names use a consistent word order (C-WORD-ORDER)
Interoperability (crate interacts nicely with other library functionality)
- Types eagerly implement common traits (C-COMMON-TRAITS)
  - Copy, Clone, Eq, PartialEq, Ord, PartialOrd, Hash, Debug,
    Display, Default
- Conversions use the standard traits From, AsRef, AsMut (C-CONV-TRAITS)
- Collections implement FromIterator and Extend (C-COLLECT)
- Data structures implement Serde's Serialize, Deserialize (C-SERDE)
- Types are Send and Sync where possible (C-SEND-SYNC)
- Error types are meaningful and well-behaved (C-GOOD-ERR)
- Binary number types provide Hex, Octal, Binary formatting (C-NUM-FMT)
- Generic reader/writer functions take R: Read and W: Write by value (C-RW-VALUE)
Macros (crate presents well-behaved macros)
- Input syntax is evocative of the output (C-EVOCATIVE)
- Macros compose well with attributes (C-MACRO-ATTR)
- Item macros work anywhere that items are allowed (C-ANYWHERE)
- Item macros support visibility specifiers (C-MACRO-VIS)
- Type fragments are flexible (C-MACRO-TY)
Documentation (crate is abundantly documented)
- Crate level docs are thorough and include examples (C-CRATE-DOC)
- All items have a rustdoc example (C-EXAMPLE)
- Examples use ?, not try!, not unwrap (C-QUESTION-MARK)
- Function docs include error, panic, and safety considerations (C-FAILURE)
- Prose contains hyperlinks to relevant things (C-LINK)
- Cargo.toml includes all common metadata (C-METADATA)
  - authors, description, license, homepage, documentation, repository,
    readme, keywords, categories
- Crate sets html_root_url attribute "https://docs.rs/CRATE/X.Y.Z" (C-HTML-ROOT)
- Release notes document all significant changes (C-RELNOTES)
- Rustdoc does not show unhelpful implementation details (C-HIDDEN)
Predictability (crate enables legible code that acts how it looks)
- Smart pointers do not add inherent methods (C-SMART-PTR)
- Conversions live on the most specific type involved (C-CONV-SPECIFIC)
- Functions with a clear receiver are methods (C-METHOD)
- Functions do not take out-parameters (C-NO-OUT)
- Operator overloads are unsurprising (C-OVERLOAD)
- Only smart pointers implement Deref and DerefMut (C-DEREF)
- Constructors are static, inherent methods (C-CTOR)
Flexibility (crate supports diverse real-world use cases)
- Functions expose intermediate results to avoid duplicate work (C-INTERMEDIATE)
- Caller decides where to copy and place data (C-CALLER-CONTROL)
- Functions minimize assumptions about parameters by using generics (C-GENERIC)
- Traits are object-safe if they may be useful as a trait object (C-OBJECT)
Type safety (crate leverages the type system effectively)
- Newtypes provide static distinctions (C-NEWTYPE)
- Arguments convey meaning through types, not bool or Option (C-CUSTOM-TYPE)
- Types for a set of flags are bitflags, not enums (C-BITFLAG)
- Builders enable construction of complex values (C-BUILDER)
Dependability (crate is unlikely to do the wrong thing)
- Functions validate their arguments (C-VALIDATE)
- Destructors never fail (C-DTOR-FAIL)
- Destructors that may block have alternatives (C-DTOR-BLOCK)
Debuggability (crate is conducive to easy debugging)
- All public types implement Debug (C-DEBUG)
- Debug representation is never empty (C-DEBUG-NONEMPTY)
Future proofing (crate is free to improve without breaking users' code)
- Sealed traits protect against downstream implementations (C-SEALED)
- Structs have private fields (C-STRUCT-PRIVATE)
- Newtypes encapsulate implementation details (C-NEWTYPE-HIDE)
- Data structures do not duplicate derived trait bounds (C-STRUCT-BOUNDS)
Necessities (to whom they matter, they really matter)
- Public dependencies of a stable crate are stable (C-STABLE)
- Crate and its dependencies have a permissive license (C-PERMISSIVE)

Gradient return type problem

Hi,

I am a beginner to use this project, I have a rather trivial question, why fn gradient(&self, p: &Self::Param) -> Result<Self::Param, Error> this function returns the type Param.

Although in the Rosenbrock example, Param is a vector type, so it doesn't matter. But for example, I have a scenario I know very well how many variables and how many parameters, so I use array instead of vector. For example I I use a one f64 number array [f64;1] as parameter, and the function is x_1+x_2+p[0], (this is a stupid function, but only for example), so I know that the gradient should be [f64;2], so, this actually won't work.

Indicate required dependencies/features for each example in Cargo.toml

If possible, set the required dependencies and features on a "per example basis" in Cargo.toml to better indicate to the user what is necessary in order to run the example. This is based on the suggestion here.

Example links in documentation of latest release point to current master implementation

This example on the master branch seems a little outdated. Only by reading through other issues in this repo I was able to find this new example.

I'd like to suggest that at least the new example should get referenced in the newest documentation. https://docs.rs/argmin/0.4.7/argmin/solver/simulatedannealing/struct.SimulatedAnnealing.html Currently, it links to the old, outdated one.

Unable to compile Nelder-Mead example

Note: I had to switch to building with the nightly toolchain because of issues with ndalgebra v0.26.2 due to "feature resolver isrequired"

I attempted to compile and run the Nelder-Mead example using the following in my Cargo.toml:

[dependencies]
argmin = { version = "0.4.4", features = ["ctrlc", "ndarrayl", "nalgebral"] }
argmin_testfunctions = "*"
ndarray = { version = "0.15", features = ["serde"] }

Adding features = ["serde"] was added to eliminate Serialize and Deserialize errors. Now I am getting the following errors:

error[E0277]: the trait bound `ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>: ArgminAdd<ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>, ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>>` is not satisfied    
  --> src\main.rs:36:30
   |
36 |           .with_initial_params(vec![
   |  ______________________________^
37 | |             // array![-2.0, 3.0],
38 | |             // array![-2.0, -1.0],
39 | |             // array![2.0, -1.0],
...  |
42 | |             array![2.0, -1.0],
43 | |         ])
   | |_________^ the trait `ArgminAdd<ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>, ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>>` is not implemented for `ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>`
   |
   = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0277]: the trait bound `ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>: ArgminSub<ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>, ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>>` is not satisfied
  --> src\main.rs:36:30
   |
36 |           .with_initial_params(vec![
   |  ______________________________^
37 | |             // array![-2.0, 3.0],
38 | |             // array![-2.0, -1.0],
39 | |             // array![2.0, -1.0],
...  |
42 | |             array![2.0, -1.0],
43 | |         ])
   | |_________^ the trait `ArgminSub<ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>, ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>>` is not implemented for `ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>`
   |
   = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0277]: the trait bound `ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>: ArgminMul<_, ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>>` is not satisfied
  --> src\main.rs:36:30
   |
36 |           .with_initial_params(vec![
   |  ______________________________^
37 | |             // array![-2.0, 3.0],
38 | |             // array![-2.0, -1.0],
39 | |             // array![2.0, -1.0],
...  |
42 | |             array![2.0, -1.0],
43 | |         ])
   | |_________^ the trait `ArgminMul<_, ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>>` is not implemented for `ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>`
   |
   = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0277]: the trait bound `ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>: ArgminAdd<ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>, ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>>` is not satisfied
  --> src\main.rs:35:18
   |
35 |     let solver = NelderMead::new()
   |                  ^^^^^^^^^^^^^^^ the trait `ArgminAdd<ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>, ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>>` is not implemented for `ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>`   
   |
   = note: required by `NelderMead::<P, F>::new`

error[E0277]: the trait bound `ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>: ArgminSub<ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>, ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>>` is not satisfied
  --> src\main.rs:35:18
   |
35 |     let solver = NelderMead::new()
   |                  ^^^^^^^^^^^^^^^ the trait `ArgminSub<ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>, ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>>` is not implemented for `ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>`   
   |
   = note: required by `NelderMead::<P, F>::new`

error[E0277]: the trait bound `ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>: ArgminMul<_, ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>>` is not satisfied
  --> src\main.rs:35:18
   |
35 |     let solver = NelderMead::new()
   |                  ^^^^^^^^^^^^^^^ the trait `ArgminMul<_, ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>>` is not implemented for `ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>`
   |
   = note: required by `NelderMead::<P, F>::new`

error[E0599]: the method `sd_tolerance` exists for struct `NelderMead<ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>, _>`, but its trait bounds were not satisfied
    --> src\main.rs:44:10
     |
44   |           .sd_tolerance(0.0001);
     |            ^^^^^^^^^^^^ private field, not a method
     | 
    ::: C:\Users\Nobody\.cargo\registry\src\github.com-1ecc6299db9ec823\ndarray-0.15.1\src\lib.rs:1276:1
     |
1276 | / pub struct ArrayBase<S, D>
1277 | | where
1278 | |     S: RawData,
1279 | | {
...    |
1289 | |     strides: D,
1290 | | }
     | | -
     | | |
     | | doesn't satisfy `_: ArgminAdd<ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>, ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>>`
     | |_doesn't satisfy `_: ArgminMul<_, ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>>`
     |   doesn't satisfy `_: ArgminSub<ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>, ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>>`
     |
     = note: the following trait bounds were not satisfied:
             `ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>: ArgminAdd<ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>, ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>>`
             `ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>: ArgminSub<ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>, ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>>`
             `ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>: ArgminMul<_, ArrayBase<OwnedRepr<{float}>, Dim<[usize; 1]>>>`

Let me know if there is any further information I can provide.

Compilation error in latest nightly

There seems to be a compilation error caused by the type resolution in the latest nightly compiler. This broke our CI system rust-ml/linfa#118 yesterday 😄 For cargo 1.52.0-nightly (1e8703890 2021-03-26) everything compiles fine, but if I update my toolchain to cargo 1.53.0-nightly (65d57e6f3 2021-04-04) I'm getting the following error:

   Compiling argmin v0.4.1
error[E0282]: type annotations needed
   --> src/solver/particleswarm/mod.rs:129:9
    |
129 | impl<O, P, F> Solver<O> for ParticleSwarm<P, F>
    |         ^ cannot infer type for type parameter `P`

error[E0282]: type annotations needed
  --> src/solver/quasinewton/bfgs.rs:62:12
   |
62 | impl<O, L, H, F> Solver<O> for BFGS<L, H, F>
   |            ^ cannot infer type for type parameter `H`

error[E0282]: type annotations needed
  --> src/solver/quasinewton/dfp.rs:52:12
   |
52 | impl<O, L, H, F> Solver<O> for DFP<L, H, F>
   |            ^ cannot infer type for type parameter `H`

error[E0282]: type annotations needed
  --> src/solver/quasinewton/sr1.rs:78:12
   |
78 | impl<O, L, H, F> Solver<O> for SR1<L, H, F>
   |            ^ cannot infer type for type parameter `H`

error[E0282]: type annotations needed
   --> src/solver/quasinewton/sr1_trustregion.rs:100:9
    |
100 | impl<O, B, R, F> Solver<O> for SR1TrustRegion<B, R, F>
    |         ^ cannot infer type for type parameter `B`

Run tests in CI

Thanks for this package!

It looks like currently Travis CI only build this crate without running tests. It would be nice if tests were also run.

Add unit tests for convergencce

It would be useful to add tests that would run the optimization algorithm and check that the expected convergence is reached.

For instance, maybe translating scipy/optimize/test/test_optimize.py could be a start.

I will look into it.

Edit: actually it looks like there is a number of classical functions implemented in https://github.com/argmin-rs/argmin-testfunctions already..

Automatic Differentiation

Automatic Differentiation (AD) generates the derivative of arbitrary order n for an input function and is heavily used in Machine Learning. Perhaps we can implement one behind a feature flag. Some previous attempts are:

I imagine a rust macro which takes a normal pure function and automatically implements the ArgminOp trait. An overview and starting point can be found here: https://arxiv.org/abs/1502.05767

Golden section search

I coded a golden section search in our enterprise project and I wonder if you want it in argmin. It's a simple algorithm with a cost function, nothing complicated.

fn golden_section_search(
    cost_function: &F,
    min_bound: f32,
    max_bound: f32,
    init_estimate: f32
) -> f32 { ... }

I just need to take the time to port it to argmin design and add a test. Are you interested in this contribution?

Popular solvers missing?

in search of a solver in rust i stumbled over this very promising library. However, I missed implementations of popular solver I used quite a lot, e.g. Brent's method (for one dimensional problems) and the methods provided by the ancient minpack package (Levenberg-Marquardt).

I was wondering what's the reason for that. Is it just a matter of priorities or are the any particular reasons to exclude these algoritms?

Condition violated: "MoreThuenteLineSearch: Search direction must be a descent direction." while using L-BFGS

I'm sorry the example is quite big, but isolating a smaller section of the codebase for this is nontrivial.

Steps to reproduce

git clone https://github.com/yukarichess/yukari -b logistic-regression
cd yukari
xz -d quiescent_positions_with_results.xz
cargo run --release --example tune

What I expected to see

e.g. if you uncomment the ArmijoCondition/BacktrackingLineSearch in yukari/src/tune.rs:

Loading FENs...
Found 1336010 positions
0.41249916
Jul 28 00:41:34.650 INFO L-BFGS, max_iters: 100
0.41249916
0.41249916
0.41249916
0.41249916
0.41249916
0.41249916
0.41249916
0.41249916
0.41249916
0.41249916
0.41249916
0.41249916
0.41249916
0.41249916
0.41249916
0.41249916
Jul 28 00:41:47.101 INFO iter: 0, cost: 0.4124991579283057, best_cost: 0.4124991579283057, cost_func_count: 1, grad_func_count: 2, jacobian_func_count: 0, hessian_func_count: 0, modify_func_count: 0, gamma: 1.0, time: 12.4510831
Jul 28 00:41:48.829 INFO iter: 1, cost: 0.4124991579283057, best_cost: 0.4124991579283057, cost_func_count: 1, grad_func_count: 3, jacobian_func_count: 0, hessian_func_count: 0, modify_func_count: 0, gamma: -16734916.191364305, time: 1.7277031
ArgminResult:
    param (best): [snipped]
    cost (best):   0.4124991579283057
    iters (best):  1
    iters (total): 2
    termination: No change in cost function value
    time:        Some(16.8272069s)

What I actually see

Loading FENs...
Found 1336010 positions
0.41249916
Jul 28 00:23:52.412 INFO L-BFGS, max_iters: 100
0.41249916
0.41249916
0.41249916
0.41249915
0.41249914
0.41249908
0.41249884
0.41249788
0.41249421
0.41248170
0.41246661
0.41296339
0.41265837
0.41254610
0.41250183
0.41248326
0.41247498
0.41247106
0.41246910
0.41246806
0.41246748
0.41246714
0.41246694
0.41246682
0.41246674
0.41246669
0.41246666
0.41246665
0.41246663
0.41246663
0.41246662
0.41246662
0.41246662
0.41246661
0.41246661
0.41246661
0.41246661
0.41246661
0.41246661
0.41246661
0.41246661
0.41246661
0.41246661
0.41246661
0.41246661
0.41246661
0.41246661
0.41246661
0.41246661
0.41246661
0.41246661
0.41246661
0.41246661
0.41246661
0.41246661
Jul 28 00:26:26.145 INFO iter: 0, cost: 0.41246661320768635, best_cost: 0.41246661320768635, cost_func_count: 1, grad_func_count: 2, jacobian_func_count: 0, hessian_func_count: 0, modify_func_count: 0, gamma: 1.0, time: 153.7329519
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Condition violated: "MoreThuenteLineSearch: Search direction must be a descent direction."', yukari/examples/tune.rs:39:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Aborted

I think this means that L-BFGS is going in the wrong direction? I admit to being very new to logistic regression, but this definitely seems like a bug to me.

Benchmark L-BFGS

Thanks for this very nice package!

On the general topic of benchmarks, I would be quite interested to know how the L-BFGS implementation in argmin would compare to,

There are Python wrappers for these, in PyLBFGS and scipy.optimize. So using pyargmin it should be relatively easy to compare them (albeit with the Python overhead).

L-BGFS is very commonly used in machine learning (also see the "Rust needs BFGS" blog by the author of the bfgs crate). I'm mostly interested in seeing of it's possible to get a comparable or faster implementation in Rust than the one in scipy.optimize, that would be more readable (and support both f64 and f32).

Opening this issue mostly to report progress on it. If you have any suggestions (e.g. on the benchmark cases to use) I would be interested as well.

Discussion about the structure of the project

When I started this project, I wanted it to be as modular as possible, which is why I've decided to split the project into a "core" part which holds all the "infrastructure" and a part where all the solvers are implemented. The idea was that others, who wanted to implement solvers based on the principles introduced here, would not need to import the entire project. However, this has lead to an awkward development-experience: In order to develop locally, all repositories have to be pulled and argmin's Cargo.toml needs to be adapted to point to the local crates.

Does anyone have a suggestion how this could be solved better? Workspaces may be an option but I'm not sure if they sufficiently improve the situation. I'm tending towards just moving the core part into argmin...
I would still keep testfunctions, finitediff, modcholesky and pyargmin separated, as they seem sufficiently self-contained.

argmin 0.4.7 violates semantic versioning

I just ran a cargo install install on https://github.com/rust-cv/cv/tree/main/vslam-sandbox. It seems that the new version of argmin released within the last hour broke the build, despite being labeled as semantically compatible.

It seems to be related to nalgebra support. Below is a resulting error (which did not occur in the previous v0.4.6, which I was building with earlier in the day):

error[E0277]: the trait bound `Matrix<f64, Const<6_usize>, Const<1_usize>, ArrayStorage<f64, 6_usize, 1_usize>>: ArgminAdd<Matrix<f64, Const<6_usize>, Const<1_usize>, ArrayStorage<f64, 6_usize, 1_usize>>, Matrix<f64, Const<6_usize>, Const<1_usize>, ArrayStorage<f64, 6_usize, 1_usize>>>` is not satisfied
  --> cv-optimize/src/single_view_optimizer.rs:22:43
   |
22 |     NelderMead::new().with_initial_params(variants)
   |                                           ^^^^^^^^ the trait `ArgminAdd<Matrix<f64, Const<6_usize>, Const<1_usize>, ArrayStorage<f64, 6_usize, 1_usize>>, Matrix<f64, Const<6_usize>, Const<1_usize>, ArrayStorage<f64, 6_usize, 1_usize>>>` is not implemented for `Matrix<f64, Const<6_usize>, Const<1_usize>, ArrayStorage<f64, 6_usize, 1_usize>>`

	/// Parameters:
	///
	/// * `cost_function`: cost function
	/// * `init_temp`: initial temperature
	pub fn new(
	search_region: (P, P),

	fn terminate(&mut self, state: &IterState<O>) -> TerminationReason {
	if state.get_grad().unwrap().norm() < std::f64::EPSILON.sqrt() {
	return TerminationReason::TargetPrecisionReached;
	}
	if (state.get_prev_cost() - state.get_cost()).abs() < std::f64::EPSILON {
	return TerminationReason::NoChangeInCost;
	}