I am observing some strange behaviour relating to bounds.

Let's find the minimum of a standard simple function:

def ackley( x, a=20, b=0.2, c=2*pi ):
    x = np.asarray_chkfinite(x)  # ValueError if any NaN or Inf
    n = len(x)
    s1 = sum( x**2 )
    s2 = sum( cos( c * x ))
    return -a*exp( -b*sqrt( s1 / n )) - exp( s2 / n ) + a + exp(1)

I'd like to use Nevergrad to optimise this. The true minimum value is at

minimum = np.zeros(n)

which gives a calculated value of

4.440892098500626e-16.

I'm using Nevergrad as follows:

from nevergrad import instrumentation as inst
from nevergrad.optimization import optimizerlib

n = 5

# Define the variable ranges...
instrum = inst.Instrumentation(*[inst.var.Array(1).asfloat().bounded(-10, 10) for _ in range(5)])

# Optimise the function...
optimizer = optimizerlib.TwoPointsDE(instrumentation=instrum, budget=10000)
recommendation = optimizer.optimize(lambda *args: ackley(np.array(args)))
print(recommendation)

This rightly returns:

>>> Candidate(args=(-1.4223567512956928e-11, 1.161667735276156e-11, 1.5909965473446878e-11, 3.142964723880178e-11, 2.1667036390793597e-12), kwargs={})

I.e., roughly zero. However, if I change the bounds to [-10, k], with k<0 I get:

k=-2
>>> Candidate(args=(-2.97922395034131, -2.979223956777669, -2.9792239575180597, -2.979223949490683, -2.979223956813635), kwargs={})

k=-4
>>> Candidate(args=(-4.986180458441733, -4.9861804545723105, -4.986180452383419, -4.986180455508544, -4.986180459693069), kwargs={})

...the minimum value is always claimed to be (k-1)*np.ones(n), however in all cases it should be at k*np.ones(n).

My question is: what's going on here? Why is Nevergrad getting these answers so consistently wrong? Have I done something wrong with the bounds? Any help would be appreciated!

Thanks!

The test framework uses nosetests, and genty to create multiple test cases easily.
Since both projects are not maintained anymore, nosetests should be updated to pytest, and genty replaced by pytest.mark.parametrize

Hi, I have prevously worked¹ on a gradient-free optimization algorithm called SPSA²-³, and I have Matlab/mex code⁴ that I can port to python easily. I am interested in benchmarking SPSA against other zero-order optimization algorithms using nevergrad.

I am following the instructions for benchmarking a new optimizer given in adding-your-own-experiments-andor-optimizers-andor-function. My understanding is that I can just add a new SPSA class in nevergrad/optimization/optimizerlib.py and implement

• __init
• _internal_ask
• _internal_provide_recommendation
• _internal_tell

functions and then add SPSA to the optims variable in the right experiment function in the nevergrad/benchmark/experiments.py module and then I should be able to generate graphs like
docs/resources/noise_r400s12_xpresults_namecigar,rotationTrue.png

However, SPSA itself uses an rng in the _internal_ask function. But the optimizer base class does not take any seed in the __init__ function. What will be a good way to make the experiments reproducible in such situation?

[1] Pushpendre Rastogi, Jingyi Zhu, James C. Spall (2016). Efficient implementation of Enhanced Adaptive Simultaneous Perturbation Algorithms. CISS 2016, pdf
[2] https://en.wikipedia.org/wiki/Simultaneous_perturbation_stochastic_approximation
[3] https://www.chessprogramming.org/SPSA
[4] https://github.com/se4u/FASPSA/blob/master/src/SPSA.m

Steps to reproduce

install pytorch1.0, and run the code, should reproduce the result
nevergrad is installed from master branch.
Once I call torch.from_numpy(nparray).float().to(device='cuda:1') or torch.from_numpy(nparray).float().cuda(), it will give me assertion error.

Observed Results

• What happened? This could be a description, log output, etc.

Assert error:

Traceback (most recent call last):
  File "bug.py", line 26, in <module>
    recommendation = optim.optimize(get_error_measurement, executor=executor)
  File "/home/jiangwei/miniconda3/envs/torch1.0/lib/python3.6/site-packages/nevergrad-0.1.1-py3.6.egg/nevergrad/optimization/base.py", line 237, in optimize
    self.tell(x, job.result())
  File "/home/jiangwei/miniconda3/envs/torch1.0/lib/python3.6/site-packages/nevergrad-0.1.1-py3.6.egg/nevergrad/optimization/base.py", line 142, in tell
    self._internal_tell(x, value)
  File "/home/jiangwei/miniconda3/envs/torch1.0/lib/python3.6/site-packages/nevergrad-0.1.1-py3.6.egg/nevergrad/optimization/optimizerlib.py", line 499, in _internal_tell
    assert x == point, str(x) + f"{x} vs {point}     {self.pop}"
AssertionError: (0.4911501931899773, -0.43302940808355883, -0.4372678287393592)(0.4911501931899773, -0.43302940808355883, -0.4372678287393592) vs (-0.3768329253449958, 0.09104643193495207, -0.10307932721079391)     [[0.5009757644092511, 0.6014841026283654, 0.6859709631240618], [0.6346080073994873, 0.7859865345869301, 0.9003882554639443], [0.7826725205681747, 0.37854971712039376, 0.6797047208048654], [0.9999999999, 0.5543576703897788, 1e-10], [0.7719219458607323, 0.42917849791820384, 1e-10], [0.7459578005791211, 1e-10, 0.9339107460996932], [0.6344706146034592, 0.9999999999, 0.5211357955736347], [0.9999999999, 0.6901278475780924, 0.39476172833171197], [0.27337963103804097, 0.24950741880316007, 0.6558451292516041], [0.4045170571446972, 0.44142018839838293, 0.8774103969720204], [0.2931404727886101, 0.4044555769698123, 0.4453981392820737], [0.5845806766378412, 0.720647952930092, 0.36691038396860787], [0.5482289873555184, 0.16233800491965578, 0.3214338601295012], [0.5442985098098989, 0.8139112813106466, 0.5338095598114053], [0.9285733747635484, 0.7721764885765621, 0.20233859057625267], [0.1717733937058818, 0.7397147758717426, 0.418468841499127], [0.47122042893368676, 0.34472409317730535, 0.15399366514420143], [1e-10, 0.3220628244980885, 0.5489131256512875], [0.16444151902631624, 0.5934854190025691, 0.5515267912162783], [0.4770005853392675, 0.42229850110668654, 0.4244852075218794], [0.34238742281461815, 0.42978664284006857, 0.8560789149377966], [0.353708077397192, 0.06932204470507108, 0.5195145124993547], [0.5780290925935871, 0.5138362412952671, 0.8322672950830423], [0.8117330114595815, 0.7950609196755931, 0.4991766093681165], [0.3531488852707311, 0.5362721515556028, 0.45895000605441877], [0.6802878404050181, 0.5851049726695651, 0.8847457111705908], [0.2469549677002738, 0.5306698658917348, 0.09330858299535116], [0.4963893894056529, 0.7295241445981984, 0.9999999999], [0.9782754185140657, 0.6190352998443717, 0.41512625078817084], [0.46593092943100056, 0.9955305520212495, 1e-10], [0.9999999999, 0.20854771969373925, 0.15305413168939158], [1e-10, 0.6313105651191179, 0.9213218538340955], [0.9019313293100729, 0.9999999999, 0.564384035404081], [0.6398750490075242, 1e-10, 0.8838234821306931], [0.36493787003029377, 0.5228582230934327, 0.7896191877081742], [0.9423521479851141, 0.8605461693648666, 0.4919352303102883], [0.5514082000961823, 0.1329081294788267, 0.23092038755143207], [0.514187105709132, 0.33944577585012803, 0.8222953832603148], [0.42020466212844065, 0.3516550689435945, 0.8303095770121608], [0.2799385916944856, 0.28369358425021557, 0.5741640442586036]]

Expected Results

• What did you expect to happen?
  This code works fine when num_workers=10, but doesn't work when num_workers=40
  Deepcopy doesn't help...

Relevant Code

from concurrent import futures
import torch
import nevergrad.optimization as optimization
import numpy as np
from numpy import linalg as LA
from copy import deepcopy
import time

optim = optimization.registry['PSO'](dimension=3, budget=1000, num_workers=40)

def get_error_measurement(in_x):
    x = deepcopy(in_x)
    x = np.array(x)
    if len(x.shape) == 1:
        x = np.expand_dims(x, axis=0)
    another_mem_np = np.ones(3)
    another_mem_torch = torch.from_numpy(another_mem_np).cuda()
    estimated_error = 0
    return estimated_error

with futures.ThreadPoolExecutor(max_workers=optim.num_workers) as executor:
    recommendation = optim.optimize(get_error_measurement, executor=executor)
print(recommendation, get_error_measurement(recommendation))

I'm currently using hyperopt to run an objective function to optimize hyperparameters for FastText so the optimization we'll have hyperparameters (like learning rate, dimension, epoch, etc.) as input and going to optimize the resulting model with precision and recall. How to achieve the same using nevergrad?

from fastText import train_supervised
from hyperopt import hp, fmin, tpe, space_eval, STATUS_OK

def objective_function(configuration, return_model=False):
    model = train_supervised(input=FILE_TRAIN, label=u'__label__', thread=12,
        epoch=configuration['epoch'],
        lr=configuration['learning_rate'],
        dim=configuration['dimension'],
        ws=configuration['window_size'],
        wordNgrams=configuration['word_ngrams']
        )
    samples_number, precision, recall = model.test(FILE_VALID, k=1)
    if return_model:
        return {'loss': -precision, 'status': STATUS_OK}, model
    return {'loss': -precision, 'status': STATUS_OK }

if __name__ == "__main__":
    space = {
        'epoch': hp.choice('epoch_', [10, 20, 30, 40, 50, 60, 70, 80]),
        'learning_rate': hp.choice('learning_rate_', [0.1, 0.3, 0.5, 0.7, 1]),
        'dimension': hp.choice('dimension_', [10, 20, 50, 70, 100, 150, 200, 250, 300]),
        'window_size': hp.choice('window_size_', range(5, 15, 1)),
        'word_ngrams': hp.choice('word_ngrams_', range(1, 5, 1))
    }

    # Optimization
    best = fmin(fn=objective_function, space=space, algo=tpe.suggest, max_evals=5) #max_evals=200

    # best model
    configuration_best = space_eval(space, best)
    print(datetime_now(), "Best (configuration): ", configuration_best)

    print(datetime_now(), "Training best config...")
    _, model = objective_function(configuration_best, return_model=True)
    model.save('model_opt.bin')

    print(datetime_now(), "Test set evaluation")
    samples_number, precision, recall = model.test(FILE_TEST, k=1)
    print_results(samples_number, precision, recall)

Steps to reproduce

1. Install nevergrad on ubuntu 18.04 , python 3.6.7 (Also seen using anaconda on Red Hat 6.2, python 3.6.8)
2. Ask and tell with particular optimizers (seen so far with Cobyla, Powell, maybe just the scipy ones)

Observed Results

Script does not terminate properly, requires Ctrl+C.

^CException ignored in: <module 'threading' from '/usr/lib/python3.6/threading.py'>
Traceback (most recent call last):
 File "/usr/lib/python3.6/threading.py", line 1294, in _shutdown
   t.join()
 File "/usr/lib/python3.6/threading.py", line 1056, in join
   self._wait_for_tstate_lock()
 File "/usr/lib/python3.6/threading.py", line 1072, in _wait_for_tstate_lock
   elif lock.acquire(block, timeout):
KeyboardInterrupt

Expected Results

Optimizer (Cobyla) should close normally. Also, if I stop the messaging thread the code terminates, but I still get the following message:

capi_return is NULL
Call-back cb_calcfc_in__cobyla__user__routines failed.

which I think is related to scipy, not sure. Regardless, I don't think I should have to go in and kill the thread.

Relevant Code

Bit redundant with the print statements, but here:

from nevergrad.optimization import optimizerlib
from nevergrad.optimization import registry
from nevergrad.optimization.recaster import _MessagingThread as mt
import threading

def beale2(p):
   x,y = p[0],p[1]
   return (1.5-x+x*y)**2+(2.25-x+x*(y**2))**2+(2.625-x+x*(y**3))**2

print('Initial Threading')
print(threading.enumerate())
a = registry['OnePlusOne'](2,budget=500)
print('Threading after getting 1+1')
print(threading.enumerate())

print('Threading after each step: ')
for i in range(3):
   x = a.ask()
   a.tell(x,beale2(*x.args))
   print(threading.enumerate())

print('Threading after getting Cobyla')
b = registry['Cobyla'](2,budget=5)
print(threading.enumerate())
print('Threading after each step: ')
for i in range(5):
   x = b.ask()
   b.tell(x,beale2(*x.args))
   print(threading.enumerate())
print('Done. Current threads: ')
print(threading.enumerate())
for t in reversed(threading.enumerate()):
   if type(t)==type(mt(None)):
       pass
       #t.stop() 
   else:
       pass
print('Final.')
print(threading.enumerate())  

Steps to reproduce

1. install nevergrad v0.2.0
2. run the below code

Observed Results

ValueError: Range cannot be empty (low >= high) unless no samples are taken

Expected Results

there should be no errors running the optimization

Relevant Code

import math
import matplotlib.pyplot as plt
import numpy as np
from nevergrad.optimization import optimizerlib
import nevergrad.instrumentation as instru

np.random.seed(123)

# from: https://www.sfu.ca/~ssurjano/michal.html
def f(x, dim=1, m=10):
  if isinstance(x, float):
      x = [x]

  res = 0
  for i in range(dim):
      res += math.sin(x[i]) * math.sin((i+1) * x[i] * x[i] / math.pi)**(2*m)
  return -res

x = np.linspace(-10, 10, 10**4)
y = [f(i) for i in x]
plt.plot(x, y)

x = instru.var.Array(1).asfloat().bounded(-10, 10, transform="arctan")
my_instrument = instru.Instrumentation(x)

optimizer = optimizerlib.PortfolioDiscreteOnePlusOne(instrumentation=my_instrument, budget=70, num_workers=1)
recomendation = optimizer.optimize(f)

for i in optimizer.archive.items_as_array():
  cur_candidate = optimizer.create_candidate.from_data(i[0])
  func_eval = i[1].mean

  plt.plot(cur_candidate.args[0], func_eval, 'bo')
  print(cur_candidate, func_eval)

plt.show()

Steps to reproduce

1. Optimize a function using the CMA algorithm
2. Run the same optimization multiple times and log results

Observed Results

• Different runs return different recommendation results

Expected Results

• Consistent results since my assumption was that nevergrad sets the seed for the optimizer internally since that is not provided in the interface.

Suggestion

• Closer inspection reveals that CMA is seeded with np.nan in nevergrad which doesn't ensure that identical results get returned from CMA. Is this a conscious decision? Are there plans to have an optional seed variable available during initializing the optimizers? If not, I suggest we set the seed for CMA to something that ensures consistent results.

Relevant code

Code snippet from optimizerlib.py:

class _CMA(base.Optimizer):
    ...

    @property
    def es(self) -> cma.CMAEvolutionStrategy:
        if self._es is None:
            popsize = max(self.num_workers, 4 + int(3 * np.log(self.dimension)))
            diag = self._parameters.diagonal
            self._es = cma.CMAEvolutionStrategy(x0=np.zeros(self.dimension, dtype=np.float),
                                                sigma0=self._parameters.scale,
                                                inopts={"popsize": popsize, "seed": np.nan, "CMA_diagonal": diag})
        return self._es

Steps to reproduce

I use the optimization demo provide in the README.

from nevergrad.optimization import optimizerlib

def square(x):
    return (x - .5)**2

optimizer = optimizerlib.OnePlusOne(dimension=1, budget=100, num_workers=5)
# alternatively, you can use optimizerlib.registry which is a dict containing all optimizer classes
recommendation = optimizer.optimize(square, executor=None, batch_mode=True)

But it raises an error as follows

• raise TypeError(f'"tell" method only supports float values but the passed value was: {value} (type: {type(value)}.')
  TypeError: "tell" method only supports float values but the passed value was: [0.25] (type: <class 'numpy.ndarray'>.

Steps to reproduce

Install nevergrad

pip3 install -e [email protected]:facebookresearch/nevergrad@master#egg=nevergrad
Obtaining nevergrad from [email protected]:facebookresearch/nevergrad@master#egg=nevergrad
  Cloning [email protected]:facebookresearch/nevergrad (to revision master) to ./src/nevergrad
Warning: Permanently added the RSA host key for IP address '192.30.255.112' to the list of known hosts.
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Command "git clone -q [email protected]:facebookresearch/nevergrad /Users/ME/Documents/workspace/temp/src/nevergrad" failed with error code 128 in None

ok. trying git clone and python3 setup.py install method. That seems to work.

run the sample program:

python3 mynevergrad.py

from nevergrad.optimization import optimizerlib

def square(x):
    return (x - .5)**2

optimizer = optimizerlib.OnePlusOne(dimension=1, budget=100, num_workers=5)
recommendation = optimizer.optimize(square, executor=None, batch_mode=True)

Observed Results

Traceback (most recent call last):
  File "mynevergrad.py", line 6, in <module>
    from nevergrad.optimization import optimizerlib
ModuleNotFoundError: No module named 'nevergrad.optimization'

Expected Results

It should run the sample

Relevant Code

import pkg_resources
for d in pkg_resources.working_set:
	print(d)

DOES include nevergrad 0.1.0

This is very likely an install, python3, homebrew "installed in user directory", or paths issue, but given that nevergrad 0.1.0 shows up in the list, it is odd...

Trying to run examples like:

https://github.com/facebookresearch/nevergrad/blob/master/docs/machinelearning.md

gives errors.

There are several overlapping structures which should be named more appropriately: Instrumentation tokens (for external files), Instrumentation, Variables (such as Gaussian etc...), (benchmark function) transforms.

I do not have any clear idea on this, help is welcome!

Expected Results

Nevergrad includes a pretty comprehensive list of implemented functions to test algorithms on. However, is it possible to include some documentation to the functions implemented inside corefuncs.py regarding the nature of these functions (multimodal, monotonous etc.)? If not some detailed documentation of the function properties, at least a link to some white paper/publication from where the functions are sourced (any of the past CECs? Since nevergrad is organising a competition in CEC19)?

Relevant Code

from nevergrad.functions.corefunc import registry as func_registry
print(sorted(func_registry.keys()))

Steps to reproduce

1. Install Nevergrad
2. Run TwoPointsDE optimization on any problem and record the params used and the values produces
3. Terminate the script
4. Reload the script and for each previously recorded arg, call optimizer.tell(cand, value)
5. Ask optimizer for new points

Observed Results

optimizer returns same initial points (same seed used) without taking into consideration the previously recorded args / points and values

Expected Results

the optimizer should provide new points for evaluation and take into consideration the previously evaluated points

The library seems very promising!
Do you plan or have any examples of Dask integration?
It would be a very good match for a dask executor.

Once optimizer.optimize() is called the system just crunches numbers without having a programmatic way to react and stop the optimization process.

Now a way to allow to calculate for either x seconds, or for n function evaluations before giving control back to the library-user would be nice.

The user could decide to continue the optimization after he got updated by just calling optimizer.optimize() again. The optimizer then could continue on the spot it got "interrupted".

edit: there is related closed issue: #49 which suggest a solution which is a bit tedious.

In the class of "class BO(recaster.SequentialRecastOptimizer)", in <recastlib.py>. The framwork operates the code " bo.explore(points_dict)". But there is no explore function in BayesionOptimization class.

The code I use to test BO classs is:

from nevergrad.optimization import recastlib

def square(x):
    return sum((x - .5)**2)

bboo = recastlib.BO(dimension=2, budget=10, num_workers=1)
bboo.qr = "lhs"
bboo._optimization_function(square)

The error raised in the terminal is :

    bo.explore(points_dict)
AttributeError: 'BayesianOptimization' object has no attribute 'explore'

Steps to reproduce

Note: I don't know if this is properly handled (doesn't cause the optimizer to mis-behave) but here is my observation:

1. Define a bounded instrumentation variable (ex. instru.var.Array(1).asfloat().bounded(0,5))
2. Create a candidate from out of space args (ex. 10)
3. Optimizer.tell(candidate, arbitrary value)

Observed Results

the candidate is created normally and the optimizer accepts it.

Expected Results

throwing an exception due to output of bounds args.

Both chocolate and optuna support asynchronous distributed optimization (e.g. via a shared SQLite database).

This is very handy when one has access to lots of machines sharing a same file system.
Each machine can ask for a new trial and tell the result (ie write to the on-disk database) in parallel.

Is this feature available in nevergrad out of the box?
Or does one need to build this oneself?

Variable should be updated to have method data_to_argument instead of process and argument_to_data instead of process_arg. This will mirror Instrumentation methods data_to_arguments  and arguments_to_data.

I'd like to run multiple instances of optimizers (e.g. 2 instances of TBPSA + several instances of CMA), each responsible for theoretically independent sets of variables. Are there any known limitations/assumptions in the NeverGrad code that would break if I tried it?

Background info: each evaluation I run takes 90s during which I could gather GBs' worth of measurements. Some of the sets of variables I'm trying to optimize affect non-overlapping measurements. e.g. they might affect different frames in a video. I would like to break down the complexity of the optimization problem using the approach above. Your advice would be very appreciated!

BTW if this already is supported by NeverGrad, I think it would be worth documenting it as I'm sure others will run into similarly complex optimization problems. I'll be happy to document it if you recommend where, e.g. in machinelearning.md?

Steps to reproduce

1. Create script with custom extensions (one that is not part of the defaults).
2. Place inside some line which should be interpolated, e.g. // @nevergrad@ var = NG_G{0, 1};.
3. Create a corresponding FolderFunction.
4. (By the way, it looks like the extension .m is missing the . here.)

Observed Results

The following exception is raised (location in code):

Traceback (most recent call last):
  File "test.py", line 15, in <module>
    func = FolderFunction(folder, command, extensions=['.custom'])
  File ".../lib/python3.6/site-packages/nevergrad/instrumentation/folderfunction.py", line 61, in __init__
    self.instrumentized_folder = InstrumentizedFolder(folder, extensions=extensions, clean_copy=clean_copy)
  File ".../lib/python3.6/site-packages/nevergrad/instrumentation/instantiate.py", line 127, in __init__
    instru_f = InstrumentizedFile(fp)
  File ".../lib/python3.6/site-packages/nevergrad/instrumentation/instantiate.py", line 63, in __init__
    lines = [(l if LINETOKEN not in l else uncomment_line(l, ext)) for l in lines]
  File ".../lib/python3.6/site-packages/nevergrad/instrumentation/instantiate.py", line 63, in <listcomp>
    lines = [(l if LINETOKEN not in l else uncomment_line(l, ext)) for l in lines]
  File ".../lib/python3.6/site-packages/nevergrad/instrumentation/instantiate.py", line 42, in uncomment_line
    pattern += r'(?P<linetoken>' + comment_chars[extension] + r" *" + LINETOKEN + r" *)"
KeyError: '.custom'

Expected Results

Since FolderFunction supports custom extensions, I'd expect this class to work with uncommenting the relevant lines in corresponding scripts. However since the comment chars are hardcoded here it fails to do so.

Would be nice to have some extra module or list / dict that stores available file types, together with possible extensions and comment chars, so the user can extend this list as needed. Or alternatively pass a dict to extensions which contains the relevant information. For example:

file_types = {'.c':  '//', '.py': '#', ...}

or

FolderFunction(..., extensions={'.custom': '//'})

and then InstrumentedFile would take the same dict of custom extensions and comment chars.

Right now the defaults are anyway duplicated here and here.

Relevant Code

import os
from pathlib import Path
import tempfile
from nevergrad.instrumentation import FolderFunction

folder = Path(tempfile.gettempdir()).joinpath('test')
script = folder.joinpath('test.custom')
os.mkdir(folder)

with open(script, 'w') as fh:
    fh.write('// @nevergrad@ var = NG_G{0, 1};\nprint(var);')

command = ['custom_app', os.path.join(*script.parts[-2:])]
func = FolderFunction(folder, command, extensions=['.custom'])

Steps to reproduce

1. pip install nevergrad
2. run relevant code below in jupyter notebook

Observed Results

---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
~/.conda/envs/ifp_design/lib/python3.6/site-packages/nevergrad/optimization/recaster.py in run(self)
     87         try:
---> 88             self.output = self._caller(self._fake_callable, *self._args, **self._kwargs)
     89         except StopOptimizerThread:  # gracefully stopping the thread

~/.conda/envs/ifp_design/lib/python3.6/site-packages/nevergrad/optimization/recastlib.py in _optimization_function(self, objective_function)
    119             bo.explore(points_dict)
--> 120         assert budget is not None
    121         assert self.budget is not None

UnboundLocalError: local variable 'budget' referenced before assignment

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
<ipython-input-14-f517d0f57590> in <module>
----> 1 optimizerlib.BO(dimension=1).ask()

~/.conda/envs/ifp_design/lib/python3.6/site-packages/nevergrad/optimization/base.py in ask(self)
    150         This function can be called multiple times to explore several points in parallel
    151         """
--> 152         suggestion = self._internal_ask()
    153         assert suggestion is not None, f"{self.__class__.__name__}._internal_ask method returned None instead of a point."
    154         self._num_suggestions += 1

~/.conda/envs/ifp_design/lib/python3.6/site-packages/nevergrad/optimization/recaster.py in _internal_ask(self)
    200             warnings.warn("Underlying optimizer has already converged, returning random points",
    201                           FinishedUnderlyingOptimizerWarning)
--> 202             self._check_error()
    203             return np.random.normal(0, 1, self.dimension)
    204         message = messages[0]  # take oldest message

~/.conda/envs/ifp_design/lib/python3.6/site-packages/nevergrad/optimization/recaster.py in _check_error(self)
    209         if self._messaging_thread is not None:
    210             if self._messaging_thread.error is not None:
--> 211                 raise RuntimeError("Recast optimizer raised an error") from self._messaging_thread.error
    212 
    213     def _internal_tell(self, x: base.ArrayLike, value: float) -> None:

RuntimeError: Recast optimizer raised an error

Expected Results

Return a 10-tuple of floats.

Relevant Code

import nevergrad.optimization.optimizerlib as optimizerlib
bo_optimizer = optimizerlib.BO(budget=20, dimension=10)
bo_optimizer.ask()

Hello,

Is there a way to make optimization results reproducible when using the ask and tell interface? We've tried to set seeds using random.seed() and numpy.random.seed() just before using ask and tell with but did not succeed in fixing the output.

Thank you

Hi,

For the minimize function, there is no argument for the callback, even if in the comment I can find https://github.com/facebookresearch/nevergrad/blob/master/nevergrad/optimization/base.py#L467-L468. Thus, I got the error of TypeError: minimize() got an unexpected keyword argument 'callback'. Could one tell me how to add callback function to the minimize function?

Thank you for the library. However I have an issue with recommendation estimate for variables with discrete range of values.

Steps to reproduce

`from nevergrad.optimization import optimizerlib
from nevergrad import instrumentation as instru

updates=[]
def square(x,y,z):
updates.append([x,y,z])
return (x-2)**2+(y+3)**2+(z-1)**2

x = instru.variables.OrderedDiscrete(np.arange(-100,100))#instru.variables.np.arange(-10,10)
y = instru.variables.OrderedDiscrete(np.arange(-100,100))#instru.variables.np.arange(-10,10)
z = instru.variables.OrderedDiscrete(np.arange(-100,100))#instru.variables.np.arange(-10,10)
ifunc = instru.InstrumentedFunction(square, x, y, z)

optimizer = optimizerlib.OnePlusOne(dimension=3, budget=1000, num_workers=1)
recommendation = optimizer.optimize(ifunc, executor=None, batch_mode=True,verbosity=0)`

Observed Results

Recommendation is:
(0.035893338998232931, -0.037160830995829099, 0.023447903097599414)

Whereas updates list has correct solutions : [2,-3,1]

Expected Results

Recommended value should be: [2,-3,1]

The arguments to setup.py add files directly to the root of sys.prefix, instead of in a nevergrad-specific subsubdirectory.

Steps to reproduce

jwnimmer@call-cc:~/issue$ python3 -m virtualenv --python python3 scratch
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/jwnimmer/issue/scratch/bin/python3
Also creating executable in /home/jwnimmer/issue/scratch/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.

jwnimmer@call-cc:~/issue$ scratch/bin/pip install 'nevergrad == 0.2.2'
Collecting nevergrad==0.2.2
  Downloading https://files.pythonhosted.org/packages/46/04/b2f4673771fbd2fd07143f44a4f2880f8cbaa08a0cc32bdf287eef99c1d7/nevergrad-0.2.2-py3-none-any.whl (170kB)
... etc ...
... etc ...
Installing collected packages: typing-extensions, numpy, scipy, joblib, scikit-learn, bayesian-optimization, cma, six, python-dateutil, pytz, pandas, nevergrad
Successfully installed bayesian-optimization-1.0.1 cma-2.7.0 joblib-0.13.2 nevergrad-0.2.2 numpy-1.16.4 pandas-0.24.2 python-dateutil-2.8.0 pytz-2019.1 scikit-learn-0.21.2 scipy-1.3.0 six-1.12.0 typing-extensions-3.7.4

jwnimmer@call-cc:~/issue$ ls -l scratch
total 80
-rw-rw-r-- 1 jwnimmer jwnimmer   84 Jul 23 14:08 bench.txt
drwxrwxr-x 2 jwnimmer jwnimmer 4096 Jul 23 14:08 bin
-rw-rw-r-- 1 jwnimmer jwnimmer  102 Jul 23 14:08 dev.txt
drwxrwxr-x 2 jwnimmer jwnimmer 4096 Jul 23 14:07 include
drwxrwxr-x 3 jwnimmer jwnimmer 4096 Jul 23 14:07 lib
-rw-rw-r-- 1 jwnimmer jwnimmer 1086 Jul 23 14:08 LICENSE
-rw-rw-r-- 1 jwnimmer jwnimmer   94 Jul 23 14:08 main.txt
drwxrwxr-x 3 jwnimmer jwnimmer 4096 Jul 23 14:08 nevergrad
-rw-rw-r-- 1 jwnimmer jwnimmer   59 Jul 23 14:07 pip-selfcheck.json
drwxrwxr-x 3 jwnimmer jwnimmer 4096 Jul 23 14:07 share

Observed Results

The sys.prefix contains spurious files:

LICENSE
bench.txt
dev.txt
main.txt
nevergrad/*

Expected Results

Only standardized files and folders (bin, lib, share, ...) exist in the root of sys.prefix.

Relevant Code

Additional thoughts

I am sure why the LICENSE or *.txt files are being explicitly installed in the first place. I believe that information is already available in the metadata. If they are to be installed, they should be in a nevergrad-specific subfolder.

I suspect the nevergrad examples are probably better installed using packages= or package_data= argument, not data_files=.

Currently each type of variable implements a token that can be used in a non-python file for "external" instrumentation. This process should be modified so that the type of variables is specified within python instead. Tokens could then be simple (named) placeholders, and the output of the instrumentation would be a function taking the corresponding named arguments as input. This function can then be instrumented as any other function.

Steps to reproduce

1. git clone https://github.com/claforte/nevergrad.git
2. cd nevergrad
3. python setup.py develop
4. nosetests nevergrad

I reproduced the issue on 2 different PCs running Windows 10, recent versions of anaconda, python 3.6.6. One environment has lots of packages (e.g. pytorch 0.4), while the other only contains nevergrad and its dependencies:

1. conda create --name test python=3.6.6
2. conda activate test
3. conda install -c anaconda numpy (otherwise python setup.py develop fails due to missing numpy)
4. continue to step 3 of the reproduction steps above.

Observed Results

• 93 tests failed, the vast majority report the same error:

======================================================================
ERROR: nevergrad.optimization.test_sequences.test_rescaler_on_hammersley
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\ProgramData\Anaconda3\envs\deepr\lib\site-packages\nose-1.3.7-py3.6.egg\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "D:\cl\amd_code\dl\nevergrad\nevergrad\optimization\test_sequences.py", line 78, in test_rescaler_on_hammersley
    sampler = sequences.ScrHammersleySampler(dimension=3, budget=4)
  File "D:\cl\amd_code\dl\nevergrad\nevergrad\optimization\sequences.py", line 176, in __init__
    super().__init__(dimension, budget, scrambling=True)
  File "D:\cl\amd_code\dl\nevergrad\nevergrad\optimization\sequences.py", line 165, in __init__
    super().__init__(dimension-1, budget, scrambling)
  File "D:\cl\amd_code\dl\nevergrad\nevergrad\optimization\sequences.py", line 134, in __init__
    self.permgen = HaltonPermutationGenerator(dimension, scrambling)
  File "D:\cl\amd_code\dl\nevergrad\nevergrad\optimization\sequences.py", line 118, in __init__
    self.seed = np.random.randint(np.iinfo(np.uint32).max)
  File "mtrand.pyx", line 991, in mtrand.RandomState.randint
ValueError: high is out of bounds for int32

----------------------------------------------------------------------
Ran 335 tests in 21.589s

FAILED (SKIP=1, errors=93)

I tried to attach a complete log produced using nosetests nevergrad > nosetests_error.txt 2>&1, but I wasn't able to drag-and-drop it on this issue. I will try again once it's been submitted.

While some tests fail, the examples in the README run successfully, and I was able to use the library with my own code.

Expected Results

• What did you expect to happen?

All tests to pass, or a mention of which tests are known to fail (e.g. in the README).

Note: I'm not an expert in pip and anaconda, it's very possible I'm doing something wrong. I just wanted to get clarification before I spend more time debugging this issue:

• What exact latest versions of Python, numpy are you using?
• Do all tests run successfully?
• Any advice?

Relevant Code

n/a

Optimizer's archive is a dict of type Dict[Tuple[float,...], Value] where Value is a light-weighted class. It keeps track of all function evaluations.
This is quite resourceful in term of memory for several reasons:

1. it's a dict.
2. the keys are tuples of float of the function dimension size (which could be millions).
3. it keeps track of all the estimations.

While I cannot think of a way to get rid of the dict, we can deal with the other 2 issues:
2. use some kind of hashable array instead of tuple of floats. This can lead to a factor 4 improvement at least. See some examples here https://stackoverflow.com/questions/16589791/most-efficient-property-to-hash-for-numpy-array
3. implement pruning on the archive, getting rid of the lowest values for instance.

Thanks for all the hard work on nevergrad, it's very useful.

It would be nice to have some documentation on the currently supported optimizers. The comments in optimizerlib already contain some details - maybe would it be possible to automatically generate some documentation out of it (using some tools from this list ?).

Ideally some documentation could also be added to the optimizers from recastlib.

And how to initialize ng.var.Array( )?

Steps to reproduce

1. Download nevergrad-0.1.1.tar.gz from PyPI
2. Extract and install it

Observed Results

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "setup.py", line 10, in <module>
    with open('requirements.txt') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'
*** Error code 1

Solution

Add requirements.txt to the tarball

Steps to reproduce

1. I have a function to minimize, which is extremely slow.
2. Due to the fact, that function has several steps inside, I could detect bad outcome early and return some big constant, say 10e5 (possible optima is 0).
3. Also, for some parameters combination it's impossible to get any result, so I also return 10e5.
4. To optimize such a function with nevergrad I need an ability to call ask and tell completely independent. Is it possible?

Relevant Code

What do I want could be coded like:

for param, value in somehow_before_obtained_results:
  optimizer.tell(param, value)

for _ in range(optimizer.budget):
  x = optimizer.ask()
  value = square(*x.args, **x.kwargs)
  if value<10e5:
      optimizer.tell(x, value)
recommendation = optimizer.provide_recommendation()

We should add a visualization tool to help understand the behavior of the function with regard to its arguments.
In my opinion, good example of such visualization is: https://bl.ocks.org/syntagmatic/482706e0638c67836d94b20f0cb37122

Other ideas are more than welcome ;)

Steps to reproduce

1. install nevergrad-0.2.0
2. initialize a RBO optimizer with small budget (I used 10 in this case)
3. run optimization

• note that this happens randomly so sometimes I run and get no errors

Observed Results

Traceback (most recent call last):
  File ".\mnist.py", line 51, in <module>
    recommendation = optimizer.optimize(ifunc)
  File "C:\Users\robert\AppData\Local\Programs\Python\Python36\lib\site-packages\nevergrad\optimization\base.py", line 402, in optimize
    self.tell(x, job.result())
  File "C:\Users\robert\AppData\Local\Programs\Python\Python36\lib\site-packages\nevergrad\optimization\base.py", line 244, in tell
    self._internal_tell_candidate(candidate, value)
  File "C:\Users\robert\AppData\Local\Programs\Python\Python36\lib\site-packages\nevergrad\optimization\base.py", line 328, in _internal_tell_candidate
    self._internal_tell(candidate.data, value)
  File "C:\Users\robert\AppData\Local\Programs\Python\Python36\lib\site-packages\nevergrad\optimization\optimizerlib.py", line 1025, in _internal_tell
    self._fake_function.register(y, -value)  # minimizing
  File "C:\Users\robert\AppData\Local\Programs\Python\Python36\lib\site-packages\nevergrad\optimization\optimizerlib.py", line 967, in register
    raise RuntimeError("Only one call can be registered at a time")

Expected Results

There should be no errors running the optimization

Relevant Code

from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC

from nevergrad.optimization import optimizerlib
from nevergrad import instrumentation as instru
from nevergrad.optimization import base

X, y = load_digits(return_X_y=True)
Xtrain, XTest, yTrain, yTest = train_test_split(X, y, random_state=42)

print('training with {} examples, testing with {}'.format(len(Xtrain), len(XTest)))

def train_and_score(args):
  if args['C_in'] > 0:
      model = SVC(C=args['C_in'], kernel=args['kernel_in'], degree=args['degree_in'])
      model.fit(Xtrain, yTrain)
      res = model.score(XTest, yTest)
  else:
      res = 1000
  
  if res > 0.98 and res < 1:
      print('args: {}'.format(args), end=', ')
      print('res: {}'.format(res))
  return -res

def train_and_score_wrapper(C_in, kernel_in, degree_in):
  return train_and_score({'C_in':C_in, 'kernel_in':kernel_in, 'degree_in':degree_in})

kernel_in = instru.variables.SoftmaxCategorical(['linear', 'poly', 'rbf', 'sigmoid'])
C_in = instru.variables.Gaussian(mean=1, std=2)
degree_in = instru.variables.OrderedDiscrete(list(range(1, 10)))
ifunc = instru.InstrumentedFunction(train_and_score_wrapper, C_in, kernel_in, degree_in)

optimizer = optimizerlib.RBO(instrumentation=ifunc.dimension, budget=10, num_workers=1)
recommendation = optimizer.optimize(ifunc)
print(recommendation)

Hi, I hope you don't mind me asking this question. I think I read all the available documentation and spent a couple of hours examining the code but I can't figure this one out.

How do you make use of an instrumented function in a benchmark experiment? I couldn't find any example that link both concepts together. For example, starting from the instrumentation example code in the README, I tried passing the instrumented function (ifunc) or the original function (myfunction) to an Experiment...

@xpregistry.register  # register experiments in the experiment registry
def DriverTunerExperiment(seed: Optional[int] = None) -> Iterator[Experiment]:
    func = myfunction  # ifunc
    for budget in [100]:
        for optimizer in ["TBPSA"]:
            yield Experiment(func, optimizer_name=optimizer, budget=budget, num_workers=1)

... but this asserts: All experiment functions should derive from BaseFunction

I also thought that the concepts of _descriptors and the instrumented variables would somehow be related, e.g. the _descriptors would be automatically populated based on the instrumented variables. Can you clarify how these 2 fit together, currently or in the roadmap?

Thank you very much! I'm excited about this project and I hope to soon contribute! (I'm preparing a PR with dozens of fixed typos to start with.)

All variations of optimizers are different classes, even for subtle changes (eg.: CMA). This process is not scalable and we should create some Optimizer factories.
It's not clear to me yet what the naming should be nor their API, or how it can interface with benchmarking which assumes all optimizers are different classes.

Currently on fight plot, the winning rate is computed from a mean over different experiments of best algorithms on one experiment after averaging the results on several runs.

It would be more accurate to compute it as the mean over different experiments of the mean winning rate between two algorithms on the same settings. This would require computing the average winning rate on one set of settings by comparing each run of each algorithm.
A preliminary example is below, but this takes too much time to compute and therefore does not seem a good idea for now.

def _make_winners_df(df: pd.DataFrame, all_optimizers: List[str]) -> tools.Selector:
    """Finds mean loss over all runs for each of the optimizers, and creates a matrix
    winner_ij = 1 if opt_i is better (lower loss) then opt_j (and .5 for ties)
    """
    if not isinstance(df, tools.Selector):
        df = tools.Selector(df)
    all_optim_set = set(all_optimizers)
    assert all(x in all_optim_set for x in df.unique("optimizer_name"))
    assert all(x in df.columns for x in ["optimizer_name", "loss"])
    # winners_ij = 1 means opt_i beats opt_j once (beating means getting a lower loss/regret)
    winners = tools.Selector(index=all_optimizers, columns=all_optimizers, data=0.)
    for optim1, optim2 in itertools.combinations(all_optimizers, r=2):
        if optim1 == optim2:
            winners[optim1, optim2] = .5
        vals1, vals2 = [df.select(optimizer_name=x).loc[:, "loss"].tolist() for x in (optim1, optim2)]
        if not vals1:
            winners.loc[optim2, optim1] = 1
        elif not vals2:
            winners.loc[optim1, optim2] = 1
        else:
            victories = sum(.5 if x == y else x > y for x, y in itertools.product(vals1, vals2))
            winners.loc[optim1, optim2] = victories
            winners.loc[optim2, optim1] = len(vals1) * len(vals2) - victories
    return winners.T

Especially in the "Adding your own experiments and/or optimizers and/or functions" section

When trying to test the results of running the same optimizer on different functions, how do we ensure consistency of representation? By consistency I mean since different function output values in different ranges, does nevergrad involve some normalisation step where all of these values are scaled to a [0, 1] range so that it is easier to visualise how far the optimizer reached with a particular function and budget? Is there any suggested method to do this otherwise?

InstrumentedFunction tend to be impractical to work with, and optimizer may eventually need information on the underlying problem structure. It seems therefore necessary to handle Instrumentation on the optimizer side.

As a preliminary version, an InstrumentedOptimizer should be added. More should come eventually, but the best way to tackle this problem is not yet clear to me (inputs are welcome).

in instrumentation/instantiate.py testing is imported .. which depends on pytest but it's not installed with nevergrad as a dependency with pip

I am trying the example provided in this issue.
I am just doing a simple trial with variable epochs and all the rest of the parameters set to their default values.

epoch = instru.variables.OrderedDiscrete([10, 20, 30, 40, 50, 60, 70, 80])
 ifunc = instru.InstrumentedFunction(train_supervised,
                                    input=FILE_TRAIN,
                                    label='__label__',
                                    thread=12,
                                    epoch=epoch) 

but the code stops here with an error that I did not manage to solve:

python3.6 train.py 
ifunc.dimension: 1
OPTIMIZER Instance of TwoPointsDE(dimension=1, budget=100, num_workers=1)
Read 0M words
Number of words:  93768
Number of labels: 13
Progress: 100.0% words/sec/thread:  164210 lr:  0.000000 loss:  0.160289 ETA:   0h 0m
Traceback (most recent call last):
  File "train.py", line 158, in <module>
    train_opt(FILE_TRAIN)
  File "train.py", line 139, in train_opt
    recommendation = optimizer.optimize(ifunc)
  File "/usr/local/lib/python3.6/dist-packages/nevergrad/optimization/base.py", line 237, in optimize
    self.tell(x, job.result())
  File "/usr/local/lib/python3.6/dist-packages/nevergrad/optimization/base.py", line 118, in tell
    if np.isnan(value) or value == np.inf:
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'' 

I have installed nevergrad with pip and I am using python3.6.
I am sure I do not have NANs in my data, and as you can see, the train_supervised method is working fine, something happens after this.

As notified in #70 the bayesian-optimization package has a new version with a different interface.

From a quick look at it, the new API may provide an ask and tell interface which could make it simpler to interface (and may help move it out of the recastlib which is mostly a hack).

In any case, we need to move to the next version, but the update may be a bit tricky.

Convergence would be helpful for simplicity's sake and to have more generic population based optimizer code.

Some problems require tuning the underlying bayes_opt params such as the utility function being used or even the underlying gp params ... it seems that there is no way to change them using nevergrad

Hi, I have several questions/ideas for which I would really appreciate your advice. Even a short answer of a couple of sentences would be really helpful and appreciated! I will break down my questions in several GitHub "issues" in the hope that they can be helpful for others and hopefully reusable in your documentation.

Q1: I in my project, each evaluation of a fitness function takes about 90 seconds and I can only run one at at time on my test PC. I'd like to run as many evaluations as I can during the night, then save the intermediate results (optimizer state?), work on something else (that often require rebooting the PC) during the day, and finally, run more of the experiment the following night.

Is there any gotchas I should be aware of, or can I simply save all the variables associated with an optimizer object? IMHO it would be ideal if methods were provided for this, especially if there are interdependencies between an optimizer and other objects' states.

(To make things more complicated, sometimes an evaluation can fail due to bad luck, e.g. computer freezes/crashes as I'm testing weird combinations of graphics driver settings. I'd like to be able to try again and resume if that happens. I could run the optimization on a separate PC but being able to save/resume would be ideal.)

Q2: Let's say after several days I'd like to get an even better optimized result. Is there a recommended way to provide additional budget to an optimizer, to explore/exploit more of the state, without starting from scratch? Are there some optimizers for which this would work and others where it wouldn't work?

While all optimizer methods use a unique internal random state to help with reproducibility since #217 , instrumentation does not, so stochastic variables such as SoftmaxCategorical can lead to non-reproducible results.

RandomState should probably be moved inside the instrumentation, and optimizers will be able to pull from it as well.

This may be related to #181, but I doubt it since external seeding would still lead to reproducible results.

They are basically the same thing, but it is unclear to me how to make Instrumentation work easily as BenchmarkFunction transforms.

Transforms:

Instrumentation: