julianlsolvers / nlsolversbase.jl Goto Github PK

View Code? Open in Web Editor NEW

30.0 30.0 28.0 203 KB

Base package for optimization and equation solver software in JuliaNLSolvers

License: Other

Julia 100.00%

equation-solver julia optimization

nlsolversbase.jl's People

Contributors

Stargazers

Watchers

nlsolversbase.jl's Issues

Queues for memoization

Shall we make `Abstract*Differentiable`s?

I was wondering if there would be some value to having abstract differentiable objects.
Something like this:

AbstractTwiceDifferentiable <: AbstractOnceDifferentiable <: AbstractNonDifferentiable <: AbstractObjective
TwiceDifferentiable <: AbstractTwiceDifferentiable 
OnceDifferentiable <: # etc.

This could possibly make dispatch easier, for example with FMinbox dispatching on
AbstractOnceDifferentiable, and hence allowing people to use TwiceDifferentiable (I'm not sure why they would want to, but whatever)

Error tagging new release

The tag name "V3.1.0" is not of the appropriate SemVer form (vX.Y.Z).
cc: @pkofod

`@inline` getters

E.g. https://github.com/JuliaNLSolvers/NLSolversBase.jl/blob/master/src/interface.jl#L59

non-modifying fg

This is more of a support request than an issue, I could not find this in the docs. I have a closure fg which calculates values and derivatives, eg

fx, gx = fg(x)

For an MWE, consider

fg(x) = 0.5*sum(abs2, x), x

How can I wrap this in OnceDifferentiable? I want to use this with Optim.jl.

Surprising runtime behaviour with objective types

Based on the discussion at https://discourse.julialang.org/t/surprising-runtime-behaviour-when-wrapping-functions/68140/3 the small test program

using BenchmarkTools
using NLSolversBase

function evaluate(f, x)
    return f(x)
end
function evaluate(od::OnceDifferentiable, x)
    return od.f(x)
end

function test(f, n)
    [evaluate(f, x) for x in 1:n]
end

f = x -> x
df = x -> [1, 1]
x_seed = [0.0, 0.0]
f_seed = [0.0, 0.0]
df_seed = [0.0 0.0; 0.0 0.0]
od = OnceDifferentiable(f, df, nothing, x_seed, f_seed, df_seed)
for n in [10, 100, 1000, 10000, 100000]
    test(f, n)
    @time test(f, n)

    test(od, n)
    @time test(od, n)
end
println()

@btime test(f, 10)
@btime test(od, 10)
@btime test(f, 100)
@btime test(od, 100)
@btime test(f, 1000)
@btime test(od, 1000)
@btime test(f, 10000)
@btime test(od, 10000)
@btime test(f, 100000)
@btime test(od, 100000)
println()

prints

  0.000011 seconds (3 allocations: 224 bytes)
  0.000013 seconds (3 allocations: 224 bytes)
  0.000001 seconds (3 allocations: 960 bytes)
  0.000003 seconds (3 allocations: 960 bytes)
  0.000001 seconds (4 allocations: 8.016 KiB)
  0.000024 seconds (493 allocations: 15.656 KiB)
  0.000006 seconds (5 allocations: 78.281 KiB)
  0.000265 seconds (9.49 k allocations: 226.547 KiB)
  0.000213 seconds (5 allocations: 781.406 KiB)
  0.013753 seconds (99.49 k allocations: 2.281 MiB, 79.70% gc time)

  267.021 ns (3 allocations: 224 bytes)
  445.685 ns (3 allocations: 224 bytes)
  282.657 ns (3 allocations: 960 bytes)
  1.940 μs (3 allocations: 960 bytes)
  783.750 ns (3 allocations: 8.00 KiB)
  20.600 μs (492 allocations: 15.64 KiB)
  4.740 μs (4 allocations: 78.27 KiB)
  223.000 μs (9493 allocations: 226.53 KiB)
  43.200 μs (4 allocations: 781.39 KiB)
  2.244 ms (99493 allocations: 2.28 MiB)

Maybe Michael Hatherly's solution improves this test case, too?

Higher dimensionnal Hessian

Hi,

I was wondering if there was a convenient way to produce an array (of size m ) of n*n Hessian matrices for an R^n -> R^m vector field out of TwiceDifferentiable.

Thanks!

Error tagging new release

The tag name "v.1.0.0" is not of the appropriate SemVer form (vX.Y.Z).
cc: @pkofod

Make show of BoundConstraints check emptiness of constraints

That is, if only variable bounds where set, just print the header for the variables.

Error tagging new release

Cannot tag a new version "v0.0.1" preceding all existing versions.
cc: @pkofod

Slow loading

This package is slow to load due to its dependence on FiniteDiff and ForwardDiff. This is somewhat problematic given it should be a "lightweight" base package, especially if packages above it do not need the functionality.

julia> @time using NLSolversBase
  9.662709 seconds (14.06 M allocations: 740.722 MiB, 3.58% gc time)

link to julianlsolvers fails

2nd line of README.md, as displayed on the web, is
NLSolversBase.jl is the core, common dependency of several packages in the JuliaNLSolvers family.

That includes a link to https://julianlsolvers.github.io/ which produces a "site not found" 404 error.

Add `value!` et al. interface to NDifferentiableConstraints

The new NDifferentiableConstraints objects introduced in #38 should also get an API for caching and calling value! et al. This would improve some of the code in IPNewton, and make it easier for others to access the constraints objects.

allow using other array types in TwiceOrderDifferentiable finite differences

xref JuliaNLSolvers/Optim.jl#924

Precompiling NLSolversBase for Julia v 1.0.0 fails.

I am trying to use the OnceDifferentiable() method from Julia v1.0.0. I get the following error:

[ Info: Precompiling NLSolversBase [d41bc354-129a-5804-8e4c-c37616107c6c]
ERROR: LoadError: LoadError: UndefVarError: start not defined
Stacktrace:
 [1] top-level scope at none:0
 [2] include at ./boot.jl:317 [inlined]
 [3] include_relative(::Module, ::String) at ./loading.jl:1038
 [4] include at ./sysimg.jl:29 [inlined]
 [5] include(::String) at /home/amit/.julia/packages/ForwardDiff/2v3a4/src/ForwardDiff.jl:3
 [6] top-level scope at none:0
 [7] top-level scope at none:2
in expression starting at /home/amit/.julia/packages/ForwardDiff/2v3a4/src/partials.jl:25
in expression starting at /home/amit/.julia/packages/ForwardDiff/2v3a4/src/ForwardDiff.jl:16
ERROR: LoadError: Failed to precompile ForwardDiff [f6369f11-7733-5829-9624-2563aa707210] to /home/amit/.julia/compiled/v1.0/ForwardDiff/k0ETY.ji.
Stacktrace:
 [1] top-level scope at none:2
in expression starting at /home/amit/.julia/packages/NLSolversBase/Cvvki/src/NLSolversBase.jl:5
ERROR: LoadError: Failed to precompile NLSolversBase [d41bc354-129a-5804-8e4c-c37616107c6c] to /home/amit/.julia/compiled/v1.0/NLSolversBase/NEDD6.ji.
Stacktrace:
 [1] macro expansion at ./logging.jl:313 [inlined]
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1184
 [3] macro expansion at ./logging.jl:311 [inlined]
 [4] _require(::Base.PkgId) at ./loading.jl:941
 [5] require(::Base.PkgId) at ./loading.jl:852
 [6] macro expansion at ./logging.jl:311 [inlined]
 [7] require(::Module, ::Symbol) at ./loading.jl:834
 [8] include(::String) at ./client.jl:388
 [9] top-level scope at none:0
in expression starting at /home/amit/WorkSpace/learning-julia/Driver.jl:2

Output of versioninfo()

julia> versioninfo()
Julia Version 1.0.0
Commit 5d4eaca0c9 (2018-08-08 20:58 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 4

Adding output of ] status:

julia> 
(v1.0) pkg> status
    Status `~/.julia/environments/v1.0/Project.toml`
  [c52e3926] Atom v0.7.4
  [7e4cc7a5] AutoDiffSource v0.2.3
  [6e4b80f9] BenchmarkTools v0.4.0
  [336ed68f] CSV v0.3.1
  [49dc2e85] Calculus v0.4.1
  [aaaa29a8] Clustering v0.12.0
  [f65535da] Convex v0.5.0
  [0fe7c1db] DataArrays v0.7.0
  [a93c6f00] DataFrames v0.13.1
  [864edb3b] DataStructures v0.11.1
  [0c46a032] DifferentialEquations v5.3.0
  [b4f34e82] Distances v0.7.3
  [aaf54ef3] DistributedArrays v0.5.1
  [31c24e10] Distributions v0.16.2
  [fa6b7ba4] DualNumbers v0.4.0
  [7a1cc6ca] FFTW v0.2.4
  [f6369f11] ForwardDiff v0.7.5
  [38e38edf] GLM v0.11.0
  [4086de5b] GLVisualize v0.7.0
  [c91e804a] Gadfly v0.7.0
  [fd0ad045] GeometricalPredicates v0.1.0
  [f67ccb44] HDF5 v0.10.0
  [50ceba7f] HyperDualNumbers v3.0.1
  [09f84164] HypothesisTests v0.7.0
  [7073ff75] IJulia v1.10.0
  [c601a237] Interact v0.9.0
  [42fd0dbc] IterativeSolvers v0.7.1
  [4076af6c] JuMP v0.18.2
  [e5e0dc1b] Juno v0.5.2
  [5ab0869b] KernelDensity v0.5.0
  [093fc24a] LightGraphs v1.0.2
  [d3d80556] LineSearches v7.0.0
  [d4b2101a] Lint v0.5.2
  [8c227e2d] MCMC v0.3.0
  [f0e99cf1] MLBase v0.7.0
  [6f286f6a] MultivariateStats v0.5.0
  [d41bc354] NLSolversBase v7.1.0
  [2774e3e8] NLsolve v2.1.0
  [6ef6ca0d] NMF v0.3.0
  [429524aa] Optim v0.17.0
  [91a5bcdd] Plots v0.20.0
  [c46f51b8] ProfileView v0.3.0
  [438e738f] PyCall v1.18.2
  [d330b81b] PyPlot v2.6.2
  [1fd47b50] QuadGK v2.0.1
  [f2b01f46] Roots v0.7.2
  [90137ffa] StaticArrays v0.8.3
  [2913bbd2] StatsBase v0.25.0
  [123dc426] SymEngine v0.4.3
  [24249f21] SymPy v0.8.0
  [6aa5eb33] TaylorSeries v0.7.4
  [9e3dc215] TimeSeries v0.12.0
  [37b6cedf] Traceur v0.1.1
  [72f80fcb] VoronoiDelaunay v0.2.0
  [44d3d7a6] Weave v0.6.1
  [8bb1440f] DelimitedFiles 
  [44cfe95a] Pkg

Joint Hessian and Jacobian evaluation using the TwiceDifferentiable type

I am working on the Gauss-Newton method, but need a bit more control over the computation than what is provided in LsqFit.jl. That I was thinking of is to use the Newton() algorithm in Optim.jl, but change the hessian and gradient as follows:

"""
    min 1/2 ||r(x)||^2
"""
# r(x) = ....  # predictive function
# J_r(x) = .... # Jacobian of r
function gauss_newton_fgh!(F, G, H, x)
    if any([!(H == nothing), !(G == nothing)])
        J = J_r(x)
        if !(H == nothing)
            # mutating calculations specific to h!
            H[:,:] = J'*J
        end
        if !(G == nothing)
            G[:] = J'*r(x)
            # mutating calculations specific to g!
        end
    end
    if !(F == nothing)
        r_k = r(x)
        return dot(r_k, r_k)/2.0
    end
end
TwiceDifferentiable(only_fgh!(gauss_newton_fgh!), x0)

However, from the TwiceDifferentiable type

mutable struct TwiceDifferentiable{T,TDF,TH,TX} <: AbstractObjective
    f
    df
    fdf
    h
    ....
end

and the constructor for inplace functions:

function TwiceDifferentiable(t::InPlaceFGH, x::AbstractVector, F::Real = real(zero(eltype(x))), G::AbstractVector = similar(x)) where {TH}

    H = alloc_H(x, F)
    f   =     x  -> t.fgh(F, nothing, nothing, x)
    df  = (G, x) -> t.fgh(nothing, G, nothing, x)
    fdf = (G, x) -> t.fgh(F, G, nothing, x)
    h   = (H, x) -> t.fgh(F, nothing, H, x)
    TwiceDifferentiable(f, df, fdf, h, x, F, G, H)
end

it seems that the gradient and hessian cannot be evaluated simultaneously, and the Gauss-Newton above evaluates the Jacobian of r one time more than needed.

Is there something I have missed, or would it make sensor to add fields to TwiceDifferentiable such that

mutable struct TwiceDifferentiable{T,TDF,TH,TX} <: AbstractObjective
    f
    df
    fdf
    h
    fdfh
    ....
end

function TwiceDifferentiable(t::InPlaceFGH, x::AbstractVector, F::Real = real(zero(eltype(x))), G::AbstractVector = similar(x)) where {TH}

    H = alloc_H(x, F)
    f   =     x  -> t.fgh(F, nothing, nothing, x)
    df  = (G, x) -> t.fgh(nothing, G, nothing, x)
    fdf = (G, x) -> t.fgh(F, G, nothing, x)
    h   = (H, x) -> t.fgh(F, nothing, H, x)
    fdfh = (H,x) -> t.fgh(F, G, H, x)
    TwiceDifferentiable(f, df, fdf, h, fdfh, x, F, G, H)
end

Edit: After some thinking, to avoid evaluating the jacobian of r one more time, one can also wrap the r function in a TwiceDifferentiable object:

"""
Minimize f(x) = 1/2||r(x)||^2
"""
function get_gauss_newton_df(dr, x0)
    function gauss_newton_fgh!(F, G, H, x)
        if !(H == nothing)
            jacobian!(dr, x)
            J_r = jacobian(dr)
            # mutating calculations specific to h!
            H[:,:] = J_r'*J_r
        end
        if !(G == nothing)
            jacobian!(dr, x)
            J_r = jacobian(dr)

            value!(dr)
            r = value(dr)
            G[:] = J_r'*r
            # mutating calculations specific to g!
        end
        if !(F == nothing)
            value!(dr)
            r = value(dr)
            return dot(r, r)/2.0
        end
    end
    TwiceDifferentiable(only_fgh!(gauss_newton_fgh!), x0)
end

But I think the original question is still relevant.

Jacobian dimensions are wrong

NLSolversBase.jl/src/objective_types/abstract.jl

Line 21 in 3923f84

 alloc_DF(x, F) = zeros(eltype(x), length(x), length(x)) #similar(x, length(x), length(x)) 

The Jacobian is not nxn, it's mxn where m is the number of functions, or you can store its nxm transpose depending on how you want to iterative over it.

1.0 development

Hi,

sorry for the dumb question (cf. https://discourse.julialang.org/t/guide-for-contributing-to-julia-package-on-github/17401) but I seem to be unable to figure out how to properly handle PRs and development with 1.0 + Pkg3. I forked and cloned the repo but activate . does not allow me to run the tests in the local instance of the repo since there is no Project.toml file (still runs the tests in .julia/... of the unmodified version) - is that intentional, am I missing something here? Thought the Project.toml was absolutely necessary.... I'd like to add a simple forward diff convenience call for TwiceDifferentiableConstraints() (cf. JuliaNLSolvers/Optim.jl#609 (comment))

Lots of method overwrites

Gadfly.jl depends on NLSolversBase and currently this package is outputting lots of method overwrites:

WARNING: Method definition (::Type{NLSolversBase.OnceDifferentiable{TF, TDF, TX, Tcplx} where Tcplx<:Union{Base.Val{true}, Base.Val{false}} where TX where TDF where TF})(Union{NLSolversBase.InplaceObjective{DF, FDF, FGH} where FGH where FDF where DF, NLSolversBase.NotInplaceObjective{DF, FDF, FGH} where FGH where FDF where DF}, AbstractArray{T, N} where N where T, AbstractArray{T, N} where N where T) in module NLSolversBase at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:82 overwritten at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:89.
WARNING: Method definition (::Type{NLSolversBase.TwiceDifferentiable{T, TDF, TH, TX} where TX where TH where TDF where T})(NLSolversBase.InplaceObjective{#s15, #s12, TH} where #s12<:Void where #s15<:Void, AbstractArray{T, 1}) in module NLSolversBase at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:96 overwritten at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:104.
WARNING: Method definition (::Type{NLSolversBase.TwiceDifferentiable{T, TDF, TH, TX} where TX where TH where TDF where T})(NLSolversBase.InplaceObjective{#s11, #s10, TH} where #s10<:Void where #s11<:Void, AbstractArray{T, 1}, Real) in module NLSolversBase at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:96 overwritten at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:104.
WARNING: Method definition (::Type{NLSolversBase.OnceDifferentiable{TF, TDF, TX, Tcplx} where Tcplx<:Union{Base.Val{true}, Base.Val{false}} where TX where TDF where TF})(Union{NLSolversBase.InplaceObjective{DF, FDF, FGH} where FGH where FDF where DF, NLSolversBase.NotInplaceObjective{DF, FDF, FGH} where FGH where FDF where DF}, AbstractArray{T, N} where N where T, AbstractArray{T, N} where N where T) in module NLSolversBase at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:82 overwritten at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:89.
WARNING: Method definition (::Type{NLSolversBase.TwiceDifferentiable{T, TDF, TH, TX} where TX where TH where TDF where T})(NLSolversBase.InplaceObjective{#s15, #s12, TH} where #s12<:Void where #s15<:Void, AbstractArray{T, 1}) in module NLSolversBase at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:96 overwritten at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:104.
WARNING: Method definition (::Type{NLSolversBase.TwiceDifferentiable{T, TDF, TH, TX} where TX where TH where TDF where T})(NLSolversBase.InplaceObjective{#s11, #s10, TH} where #s10<:Void where #s11<:Void, AbstractArray{T, 1}, Real) in module NLSolversBase at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:96 overwritten at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:104.
WARNING: Method definition (::Type{NLSolversBase.OnceDifferentiable{TF, TDF, TX, Tcplx} where Tcplx<:Union{Base.Val{true}, Base.Val{false}} where TX where TDF where TF})(Union{NLSolversBase.InplaceObjective{DF, FDF, FGH} where FGH where FDF where DF, NLSolversBase.NotInplaceObjective{DF, FDF, FGH} where FGH where FDF where DF}, AbstractArray{T, N} where N where T, AbstractArray{T, N} where N where T) in module NLSolversBase at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:82 overwritten at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:89.
WARNING: Method definition (::Type{NLSolversBase.TwiceDifferentiable{T, TDF, TH, TX} where TX where TH where TDF where T})(NLSolversBase.InplaceObjective{#s15, #s12, TH} where #s12<:Void where #s15<:Void, AbstractArray{T, 1}) in module NLSolversBase at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:96 overwritten at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:104.
WARNING: Method definition (::Type{NLSolversBase.TwiceDifferentiable{T, TDF, TH, TX} where TX where TH where TDF where T})(NLSolversBase.InplaceObjective{#s11, #s10, TH} where #s10<:Void where #s11<:Void, AbstractArray{T, 1}, Real) in module NLSolversBase at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:96 overwritten at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:104.
WARNING: Method definition (::Type{NLSolversBase.OnceDifferentiable{TF, TDF, TX, Tcplx} where Tcplx<:Union{Base.Val{true}, Base.Val{false}} where TX where TDF where TF})(Union{NLSolversBase.InplaceObjective{DF, FDF, FGH} where FGH where FDF where DF, NLSolversBase.NotInplaceObjective{DF, FDF, FGH} where FGH where FDF where DF}, AbstractArray{T, N} where N where T, AbstractArray{T, N} where N where T) in module NLSolversBase at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:82 overwritten at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:89.
WARNING: Method definition (::Type{NLSolversBase.TwiceDifferentiable{T, TDF, TH, TX} where TX where TH where TDF where T})(NLSolversBase.InplaceObjective{#s15, #s12, TH} where #s12<:Void where #s15<:Void, AbstractArray{T, 1}) in module NLSolversBase at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:96 overwritten at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:104.
WARNING: Method definition (::Type{NLSolversBase.TwiceDifferentiable{T, TDF, TH, TX} where TX where TH where TDF where T})(NLSolversBase.InplaceObjective{#s11, #s10, TH} where #s10<:Void where #s11<:Void, AbstractArray{T, 1}, Real) in module NLSolversBase at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:96 overwritten at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:104.
WARNING: Method definition (::Type{NLSolversBase.OnceDifferentiable{TF, TDF, TX, Tcplx} where Tcplx<:Union{Base.Val{true}, Base.Val{false}} where TX where TDF where TF})(Union{NLSolversBase.InplaceObjective{DF, FDF, FGH} where FGH where FDF where DF, NLSolversBase.NotInplaceObjective{DF, FDF, FGH} where FGH where FDF where DF}, AbstractArray{T, N} where N where T, AbstractArray{T, N} where N where T) in module NLSolversBase at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:82 overwritten at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:89.
WARNING: Method definition (::Type{NLSolversBase.TwiceDifferentiable{T, TDF, TH, TX} where TX where TH where TDF where T})(NLSolversBase.InplaceObjective{#s15, #s12, TH} where #s12<:Void where #s15<:Void, AbstractArray{T, 1}) in module NLSolversBase at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:96 overwritten at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:104.
WARNING: Method definition (::Type{NLSolversBase.TwiceDifferentiable{T, TDF, TH, TX} where TX where TH where TDF where T})(NLSolversBase.InplaceObjective{#s11, #s10, TH} where #s10<:Void where #s11<:Void, AbstractArray{T, 1}, Real) in module NLSolversBase at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:96 overwritten at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:104.
WARNING: Method definition (::Type{NLSolversBase.OnceDifferentiable{TF, TDF, TX, Tcplx} where Tcplx<:Union{Base.Val{true}, Base.Val{false}} where TX where TDF where TF})(Union{NLSolversBase.InplaceObjective{DF, FDF, FGH} where FGH where FDF where DF, NLSolversBase.NotInplaceObjective{DF, FDF, FGH} where FGH where FDF where DF}, AbstractArray{T, N} where N where T, AbstractArray{T, N} where N where T) in module NLSolversBase at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:82 overwritten at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:89.
WARNING: Method definition (::Type{NLSolversBase.TwiceDifferentiable{T, TDF, TH, TX} where TX where TH where TDF where T})(NLSolversBase.InplaceObjective{#s15, #s12, TH} where #s12<:Void where #s15<:Void, AbstractArray{T, 1}) in module NLSolversBase at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:96 overwritten at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:104.
WARNING: Method definition (::Type{NLSolversBase.TwiceDifferentiable{T, TDF, TH, TX} where TX where TH where TDF where T})(NLSolversBase.InplaceObjective{#s11, #s10, TH} where #s10<:Void where #s11<:Void, AbstractArray{T, 1}, Real) in module NLSolversBase at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:96 overwritten at /Users/tamasnagy/.julia/v0.6/NLSolversBase/src/objective_types/incomplete.jl:104

Any way we can limit these? I have NLSolversBase v4.4.0 installed

Better test case

Use a test case without exp to test the kwargs, as the finite and autodiff answers are too different which makes the test look stupid, but does at least test that finite diff and autodiff answers match

Message from JET

Running JET results in the following message

┌ @ C:\Users\Win10\source\repos\julia\NLSolversBase.jl-7.8.1\src\objective_types\incomplete.jl:46 
│ ArgumentError: invalid type for argument t in method definition for make_f at C:\Users\Win10\source\repos\julia\NLSolversBase.jl-7.8.1\src\objective_types\incomplete.jl:46
│ Stacktrace:
│  [1] evaluate_methoddef(frame::JuliaInterpreter.Frame, node::Expr)
│    @ JuliaInterpreter C:\Users\Win10\.julia\packages\JuliaInterpreter\4B89D\src\interpret.jl:283
│  [2] step_expr!(recurse::Any, frame::JuliaInterpreter.Frame, node::Any, istoplevel::Bool)
│    @ JuliaInterpreter C:\Users\Win10\.julia\packages\JuliaInterpreter\4B89D\src\interpret.jl:502
│  [3] step_expr!(interp::JET.ConcreteInterpreter{JET.JETAnalyzer{JET.BasicPass}, Expr}, frame::JuliaInterpreter.Frame, node::Any, istoplevel::Bool)
│    @ JET C:\Users\Win10\.julia\packages\JET\hidRI\src\virtualprocess.jl:924
│  [4] step_expr!(recurse::Any, frame::JuliaInterpreter.Frame, istoplevel::Bool)
│    @ JuliaInterpreter C:\Users\Win10\.julia\packages\JuliaInterpreter\4B89D\src\interpret.jl:611
│  [5] selective_eval!(recurse::Any, frame::JuliaInterpreter.Frame, isrequired::Vector{Bool}, istoplevel::Bool)
│    @ LoweredCodeUtils C:\Users\Win10\.julia\packages\LoweredCodeUtils\jZY56\src\codeedges.jl:850
│  [6] selective_eval_fromstart!
│    @ C:\Users\Win10\.julia\packages\LoweredCodeUtils\jZY56\src\codeedges.jl:875 [inlined]
│  [7] partially_interpret!(interp::JET.ConcreteInterpreter{JET.JETAnalyzer{JET.BasicPass}, Expr}, mod::Module, src::Core.CodeInfo)
│    @ JET C:\Users\Win10\.julia\packages\JET\hidRI\src\virtualprocess.jl:757
│  [8] _virtual_process!(toplevelex::Expr, filename::String, analyzer::JET.JETAnalyzer{JET.BasicPass}, config::JET.ToplevelConfig{Expr}, context::Module, res::JET.VirtualProcessResult)
│    @ JET C:\Users\Win10\.julia\packages\JET\hidRI\src\virtualprocess.jl:588
└─────────────────────────────────────────────────────────────────────────────────────────────────

[doc] explicit definition of arguments

Although I can often sort of guess, could you please provide an explicit definition of the function arguments for the functions defined here, both the type and substantive meaning?

Some examples I've seen suggest that some of the arguments can be omitted, perhaps to be filled in by automatic differentiation. Explicit discussion of that would be useful too.

Some of the source code (twicedifferentiable.jl, incomplete.jl) is similarly uncommented.

Deprecations and femtocleaner

att @Keno : I am wondering if FemtoCleaner shouldn't have removed

NLSolversBase.jl/REQUIRE

Line 2 in ddd6847

Compat 0.17.0

NLSolversBase.jl/src/objective_types.jl

Line 1 in ddd6847

@compat abstract type AbstractObjective end

and

NLSolversBase.jl/src/NLSolversBase.jl

Line 5 in ddd6847

using Compat

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Why is f_calls not the same shape as F?

NLSolversBase.jl/src/objective_types/nondifferentiable.jl

Line 27 in 3923f84

NonDifferentiable{TF,typeof(x),Val{iscomplex}}(f, F, x_of_nans(x), [0,])

The same thing is there with other objective types.

Himmelblau references in README.md

Text and code refer to Hosaki's function but the math image is Himmelblau's function.

Later Himmelblau's function is referenced again

sparse AD on OnceDifferentiableConstraints and TwiceDifferentiableConstraints

xref JuliaNLSolvers/Optim.jl#977, specially point 1 of the TODO list:

The passing of array type for the Jacobian of the constraints is ugly. I made it an argument to IPNewton, but it would be better as a field of TwiceDifferentiableConstraints, just like H is a field of TwiceDifferentiable. That requires changing NLSolversBase

Error tagging new release

The tag name "V7.0.0" is not of the appropriate SemVer form (vX.Y.Z).
cc: @anriseth

value! function is not specialized for Real and Vector TF in interface.jl

NLSolversBase.jl/src/interface.jl

Line 4 in 3923f84

obj.F = obj.f(real_to_complex(obj, x))

This function assumes the f function is always non-mutating. But in the case of TF<:AbstractVector, this may not be true and even if true, will cause allocations.

Why are`f_calls`, `df_calls` and `h_calls` all `Vector{Int}`?

It seems strange to me that these fields are Vector{Int} when the objective types (OnceDifferentiable etc.) are already mutable. Is there any use case where a vector is needed to hold multiple values? Would having these fields as Int be sufficient?

Also, the way to set/retrieve these counters are not explicitly made into the interface. But, there could be some functions defined just for that so that external packages do not have to hardcode the field names.

Is there any documentation?

For instance, how does one learn how to use TwiceDifferentiableConstraints? ?TwiceDifferentiableConstraints says "No documentation found". The documentation of Optim.jl uses it in the IPNewton tutorial, but doesn't explain what it is and what the options mean.

I learned that TwiceDifferentiableConstraints supports autodiff from a GitHub issue, and before that I was painfully calculating Hessians by hand, as shown in the Optim.jl documentation. I can only imagine how many mistakes people made when calculating constraints' Hessians by hand (unless it's widely known that TwiceDifferentiableConstraints supports autodiff).

img cache

Add tests for TwiceDifferentiableHV

We don't have any tests for TwiceDifferentiableHV

Complex-valued gradients broken

Due to the changes introduced in #112 for fixing #97, real-valued objective functions of complex parameters (thus complex-valued gradients) are broken.

For example:

x0 = randn(ComplexF64, 3)
OnceDifferentiable(x -> sum(abs2, x), x0)

does not work because of an attempt to copy the complex x0 into a real-valued array allocated in abstract.jl:21:

ERROR: InexactError: Float64(-0.3704615651702863 - 0.8842199776298206im)
Stacktrace:
 [1] Real at ./complex.jl:37 [inlined]
 [2] _broadcast_getindex_evalf at ./broadcast.jl:630 [inlined]
 [3] _broadcast_getindex at ./broadcast.jl:603 [inlined]
 [4] getindex at ./broadcast.jl:563 [inlined]
 [5] macro expansion at ./broadcast.jl:909 [inlined]
 [6] macro expansion at ./simdloop.jl:77 [inlined]
 [7] copyto! at ./broadcast.jl:908 [inlined]
 [8] copyto! at ./broadcast.jl:863 [inlined]
 [9] copy at ./broadcast.jl:839 [inlined]
 [10] materialize(::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,Type{Float64},Tuple{Array{Complex{Float64},1}}}) at ./broadcast.jl:819
 [11] x_of_nans(::Array{Complex{Float64},1}, ::Type) at .julia/packages/NLSolversBase/eVJ9t/src/NLSolversBase.jl:60
 [12] alloc_DF at .julia/packages/NLSolversBase/eVJ9t/src/objective_types/abstract.jl:21 [inlined]
 [13] OnceDifferentiable at .julia/packages/NLSolversBase/eVJ9t/src/objective_types/oncedifferentiable.jl:19 [inlined] (repeats 2 times)
 [14] top-level scope at REPL[21]:1

Error tagging new release

Cannot tag a new version "v0.0.1" preceding all existing versions.
cc: @pkofod

Add option to specify ForwardDiff chunk size

Hi, it would be nice to have this option available (and propagate it to Optim.jl as well), this setting can have a great impact on performances/allocations in some situations (see for ex: https://discourse.julialang.org/t/large-memory-allocation-in-forwarddiff/42581/14).

Thanks!

TwiceDifferentiable

Since I updated, TwiceDifferentiable(f, g!, fg!, h!) fails. I must either omit fg! or add a seed vector. I think a method is missing around line 95 in objective_types.jl.

Incorrect initialization of derivative storage

NLSolversBase.jl/src/objective_types/twicedifferentiable.jl

Line 18 in 12ceb15

 function TwiceDifferentiable(f, g, fg, h, x::TX, F::T = real(zero(eltype(x))), G::TG = similar(x), H::TH = alloc_H(x); inplace = true) where {T, TG, TH, TX} 

, similar(x) is used for initialization of the gradient array but that need not be the right element type. In a nested derivative computation such as

julia> myf(x,y) = x^2+y^2
myf (generic function with 1 method)

julia> ForwardDiff.gradient(s -> optimize(t -> myf(t[1], s[1]), [0.0], Newton(), autodiff=:forward).minimum, [0.4])
ERROR: MethodError: no method matching Float64(::ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1})
Closest candidates are:
  Float64(::Real, ::RoundingMode) where T<:AbstractFloat at rounding.jl:194
  Float64(::T<:Number) where T<:Number at boot.jl:741
  Float64(::Int8) at float.jl:60
  ...
Stacktrace:
 [1] convert(::Type{Float64}, ::ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1}) at ./number.jl:7
 [2] setproperty!(::DiffResults.MutableDiffResult{1,Float64,Tuple{Array{Float64,1}}}, ::Symbol, ::ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1}) at ./sysimg.jl:19
 [3] value!(::DiffResults.MutableDiffResult{1,Float64,Tuple{Array{Float64,1}}}, ::ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1}) at /Users/andreasnoack/.julia/packages/DiffResults/1LURj/src/DiffResults.jl:159
 [4] extract_gradient!(::Type{ForwardDiff.Tag{getfield(Main, Symbol("##38#40")){Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1},1}},Float64}}, ::DiffResults.MutableDiffResult{1,Float64,Tuple{Array{Float64,1}}}, ::ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##38#40")){Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1},1}},Float64},ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1},1}) at /Users/andreasnoack/.julia/packages/ForwardDiff/okZnq/src/gradient.jl:70
 [5] vector_mode_gradient!(::DiffResults.MutableDiffResult{1,Float64,Tuple{Array{Float64,1}}}, ::getfield(Main, Symbol("##38#40")){Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1},1}}, ::Array{Float64,1}, ::ForwardDiff.GradientConfig{ForwardDiff.Tag{getfield(Main, Symbol("##38#40")){Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1},1}},Float64},Float64,1,Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##38#40")){Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1},1}},Float64},Float64,1},1}}) at /Users/andreasnoack/.julia/packages/ForwardDiff/okZnq/src/gradient.jl:103
 [6] gradient!(::DiffResults.MutableDiffResult{1,Float64,Tuple{Array{Float64,1}}}, ::getfield(Main, Symbol("##38#40")){Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1},1}}, ::Array{Float64,1}, ::ForwardDiff.GradientConfig{ForwardDiff.Tag{getfield(Main, Symbol("##38#40")){Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1},1}},Float64},Float64,1,Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##38#40")){Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1},1}},Float64},Float64,1},1}}, ::Val{true}) at /Users/andreasnoack/.julia/packages/ForwardDiff/okZnq/src/gradient.jl:35
 [7] gradient!(::DiffResults.MutableDiffResult{1,Float64,Tuple{Array{Float64,1}}}, ::getfield(Main, Symbol("##38#40")){Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1},1}}, ::Array{Float64,1}, ::ForwardDiff.GradientConfig{ForwardDiff.Tag{getfield(Main, Symbol("##38#40")){Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1},1}},Float64},Float64,1,Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##38#40")){Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1},1}},Float64},Float64,1},1}}) at /Users/andreasnoack/.julia/packages/ForwardDiff/okZnq/src/gradient.jl:33
 [8] (::getfield(NLSolversBase, Symbol("##40#46")){getfield(Main, Symbol("##38#40")){Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1},1}},ForwardDiff.GradientConfig{ForwardDiff.Tag{getfield(Main, Symbol("##38#40")){Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1},1}},Float64},Float64,1,Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##38#40")){Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1},1}},Float64},Float64,1},1}}})(::Array{Float64,1}, ::Array{Float64,1}) at /Users/andreasnoack/.julia/dev/NLSolversBase/src/objective_types/twicedifferentiable.jl:122
 [9] value_gradient!!(::TwiceDifferentiable{Float64,Array{Float64,1},Array{Float64,2},Array{Float64,1}}, ::Array{Float64,1}) at /Users/andreasnoack/.julia/dev/NLSolversBase/src/interface.jl:82
 [10] initial_state(::Newton{LineSearches.InitialStatic{Float64},LineSearches.HagerZhang{Float64,Base.RefValue{Bool}}}, ::Optim.Options{Float64,Nothing}, ::TwiceDifferentiable{Float64,Array{Float64,1},Array{Float64,2},Array{Float64,1}}, ::Array{Float64,1}) at /Users/andreasnoack/.julia/dev/Optim/src/multivariate/solvers/second_order/newton.jl:45
 [11] #optimize#87 at /Users/andreasnoack/.julia/dev/Optim/src/multivariate/optimize/optimize.jl:33 [inlined]
 [12] (::getfield(Optim, Symbol("#kw##optimize")))(::NamedTuple{(:autodiff,),Tuple{Symbol}}, ::typeof(optimize), ::Function, ::Array{Float64,1}, ::Newton{LineSearches.InitialStatic{Float64},LineSearches.HagerZhang{Float64,Base.RefValue{Bool}}}, ::Optim.Options{Float64,Nothing}) at ./none:0 (repeats 2 times)
 [13] vector_mode_gradient(::getfield(Main, Symbol("##37#39")), ::Array{Float64,1}, ::ForwardDiff.GradientConfig{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1,Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1},1}}) at ./none:1
 [14] gradient(::Function, ::Array{Float64,1}, ::ForwardDiff.GradientConfig{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1,Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1},1}}, ::Val{true}) at /Users/andreasnoack/.julia/packages/ForwardDiff/okZnq/src/gradient.jl:17
 [15] gradient(::Function, ::Array{Float64,1}, ::ForwardDiff.GradientConfig{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1,Array{ForwardDiff.Dual{ForwardDiff.Tag{getfield(Main, Symbol("##37#39")),Float64},Float64,1},1}}) at /Users/andreasnoack/.julia/packages/ForwardDiff/okZnq/src/gradient.jl:15 (repeats 2 times)
 [16] top-level scope at none:0

the gradient will be a vector of duals which therefore causes an error.

`value` doesn't update x_f

NLSolversBase.jl/src/interface.jl

Line 6 in 3923f84

function value(obj::AbstractObjective, x)

Unlike

NLSolversBase.jl/src/interface.jl

Line 3 in 3923f84

copy!(obj.x_f, x)

, value doesn't update x_f which seems assumed as the function is checking if x != obj.x_f before evaluating the objective. This condition will never be true if x_f is not updated and only the function value is used in the algorithm, not value!.

Weird/wrong syntax for NonDifferentiable struct definition

NLSolversBase.jl/src/objective_types/nondifferentiable.jl

Line 2 in 3923f84

 mutable struct NonDifferentiable{TF,TX,Tcplx} <: AbstractObjective where {TF, TX, 

The syntax here looks off, am I missing something?

Sorting out f, df and fdf handling in incomplete.jl and abstract.jl

I have a few issues with incomplete.jl.

It introduces a parallel way to define f, df and fdf which I think can be merged with abstract.jl.
It defines an inplace version of make_f when it doesn't make sense, for example in

NLSolversBase.jl/src/objective_types/incomplete.jl

Line 32 in 3923f84

make_f(t::InplaceObjective, x, F::Real) = x -> fdf(t)(F, nothing, x)
The interface of fdf in terms of how many arguments and return types is inconsistent for example

NLSolversBase.jl/src/objective_types/incomplete.jl

Line 39 in 3923f84

# The contract with the user is that fdf returns (F, DF)

and

NLSolversBase.jl/src/objective_types/incomplete.jl

Line 46 in 3923f84

function make_fdf(t::NotInplaceObjective, x, F::Real)

.

Extending autodiff compatibilities

I was considering extending the autodiff compatibilities of NLSolverBase and PR the result back. I wanted to include ReverseDiff.jl, Zygote.jl and maybe also forward-over-reverse for the Hessian.

Is there any reason why this has not been done yet that I should be aware before starting? I don't want to implement it just to discover that there is a reason why it has not been done.

Also, is there a rationale for the fact that, for twicedifferentiable.jl when the gradient is provided, the hessian is calculated using the jacobian of the gradient for the finitediff case, but it calculated using the hessian of the function for forwardiff (lines 63 and 69)?

Hessian-vector products

JuliaNLSolvers/Optim.jl#416 is waiting for Hessian-vector product support in NLSolversBase, it seems (JuliaNLSolvers/Optim.jl#416 (comment) )

Just adding this issue to keep track of TODOs

Error tagging new release

The tag name "v.2.1.1" is not of the appropriate SemVer form (vX.Y.Z).
cc: @pkofod

julianlsolvers / nlsolversbase.jl Goto Github PK

nlsolversbase.jl's People

Contributors

Stargazers

Watchers

Forkers

nlsolversbase.jl's Issues

Recommend Projects

Recommend Topics

Recommend Org