d4l3k / go-bayesopt Goto Github PK

A library for doing Bayesian Optimization using Gaussian Processes (blackbox optimizer) in Go/Golang.

Home Page: https://godoc.org/github.com/d4l3k/go-bayesopt

License: MIT License

Go 100.00%

bayesianoptimization bayesopt machine-learning hyperparameter-optimization go optimization blackbox-optimizer gaussian-processes

go-bayesopt's Introduction

go-bayesopt

A library for doing Bayesian Optimization using Gaussian Processes (blackbox optimizer) in Go/Golang.

This project is under active development, if you find a bug, or anything that needs correction, please let me know.

Simple Example

package main

import (
  "log"
  "math"

  "github.com/d4l3k/go-bayesopt"
)

func main() {
  X := bayesopt.UniformParam{
    Max: 10,
    Min: -10,
  }
  o := bayesopt.New(
    []Param{
      X,
    },
  )
  // minimize x^2+1
  x, y, err := o.Optimize(func(params map[Param]float64) float64 {
    return math.Pow(params[X], 2) + 1
  })
  if err != nil {
    log.Fatal(err)
  }
  log.Println(x, y)
}

How does it work?

From https://github.com/fmfn/BayesianOptimization:

Bayesian optimization works by constructing a posterior distribution of functions (gaussian process) that best describes the function you want to optimize. As the number of observations grows, the posterior distribution improves, and the algorithm becomes more certain of which regions in parameter space are worth exploring and which are not, as seen in the picture below.

As you iterate over and over, the algorithm balances its needs of exploration and exploitation taking into account what it knows about the target function. At each step a Gaussian Process is fitted to the known samples (points previously explored), and the posterior distribution, combined with a exploration strategy (such as UCB (Upper Confidence Bound), or EI (Expected Improvement)), are used to determine the next point that should be explored (see the gif below).

This process is designed to minimize the number of steps required to find a combination of parameters that are close to the optimal combination. To do so, this method uses a proxy optimization problem (finding the maximum of the acquisition function) that, albeit still a hard problem, is cheaper (in the computational sense) and common tools can be employed. Therefore Bayesian Optimization is most adequate for situations where sampling the function to be optimized is a very expensive endeavor. See the references for a proper discussion of this method.

License

go-bayesopt is licensed under the MIT license.

go-bayesopt's People

Contributors

Stargazers

Watchers

Forkers

domenicrosati dtolpin kikuchy c-bata etsangsplk 5l1v3r1

go-bayesopt's Issues

UniformParams sometimes given as all zero values, even when Min is present

Example:

	var (
		a        = bayesopt.UniformParam{Name: "a", Min: 2, Max: 500}
		b        = bayesopt.UniformParam{Name: "b", Min: 0, Max: 100}
		c        = bayesopt.UniformParam{Name: "c", Min: 0, Max: 100}
		d        = bayesopt.UniformParam{Name: "d", Min: 0.01, Max: 1}
		e        = bayesopt.UniformParam{Name: "e", Min: 0, Max: 1000}
		bayesOpt = bayesopt.New([]bayesopt.Param{a, b, c, d, e})
	)

	bestParams, _, _ := bayesOpt.Optimize(
		func(params map[bayesopt.Param]float64) float64 {
			if params[a] == 0 {
				panic(fmt.Sprintf("params are all zero: %+v", params))
			}
			return params[a]
		},
	)

	log.Printf("bestParams:%+v", bestParams)

On latest master, using go 1.21.8, this more often than not panics at the if params[a] == 0 check:

panic: params are all zero: map[{Name:a Max:500 Min:2}:0 {Name:b Max:100 Min:0}:0 {Name:c Max:100 Min:0}:0 {Name:d Max:1 Min:0.01}:0 {Name:e Max:1000 Min:0}:0]

goroutine 1 [running]:
main.main.func1(0xc000385260)
        /tmp/tmp.yINnICZSgY/main.go:23 +0x165
github.com/d4l3k/go-bayesopt.(*Optimizer).Optimize(0xc0000c6000, 0xc0000be180)
        /home/mediocregopher/go/pkg/mod/github.com/d4l3k/[email protected]/bayesopt.go:348 +0x1f9
main.main()
        /tmp/tmp.yINnICZSgY/main.go:20 +0x379
exit status 2

The a param should never be zero, as it has a Min of 2. In fact all the params are zero, which makes me think there's definitely some kind of bug in the optimizer.

Interestingly the issue doesn't seem to happen on go 1.19.4, or if it does it's very rare. So perhaps some change in the scheduler is causing a race condition? I've tried building with -race but that hasn't popped up with anything.

Not sure if this project is still being maintained, but if so any help here would be much appreciated.

Provide a random seed for sampling

It would be great if we could provide a random seed for sampling so we can write determanistic tests.

What do you think @d4l3k ?

Add other informative priors

I have a function whose inputs are is normaly distrbuted,
I have another whose inputs are exponentially distributed,
We should be able to add the following informative prior distrubutions as parameters:

Normal
Expontential

How to we constraints the X

I have define this

	X := bayesopt.UniformParam{
		Max: 5,
		Min: 1,
	}
	o := bayesopt.New(
		[]bayesopt.Param{
			X,
		},
	)
	x, y, err := o.Optimize(func(params map[bayesopt.Param]float64) float64 {
		return ……
	})

I want to the minimal X is 1，but in the result, it perfroms map[{ 5 1}:0] 0
How could I contrain the X > min?

What is the status of this package?

I am interested in using this package, as I prefer not to use Python for my specific task, but I see that the last commits to this package were done many years ago.

What was the status of this package when it was last modified? Was it working with most cases, did you know if it was failing under some conditions, was there something planned to be added but never made it, etc? Or is the package considered complete?

Are there plans of continue working on this as main authors of the project? I could contribute, but we would need a tech lead on this with more experience on the whole package.

How do we set the confidence interval?

The default confidence interval is 95%, how do we change the default value?

How do we create constraints for the function output?

Hey, need some help.

I'd like to have some constraint for the function output.
So far I am doing something like this:

	function := func(sampleParams map[bayesopt.Param]float64) float64 {
		var cost float64
		for _, regressor := range regressions {
			var variables []float64
			for i := range params {
				variables = append(variables, sampleParams[params[i]])
			}
			p, _ := regressor.Predict(variables)
			cost += 500.0 - p
			if p <= 0 || p >= 500 {
				cost = math.Inf(0)
			}
		}
		return cost
	}

But that doesn't work when adding up costs for multiple regressions.
I was wondering if there was a simpler way in this pkg to have a constraint for y or the output like here my two constraints are:

p <= 0
p >= 500

Let me know if there is an easier way to implement this constraint with the pkg.

Dom