claesenm / optunity Goto Github PK

I tried to add the argument 'class_weight' to search space for maximize_structured function. Although it works well when I put a normal value such as 'class_weight':[1,10], the search space gets to be broken when I put 'None' like 'class_weight':None which is the default value for 'class_weight'.

In some cases PR curves are incorrect.

Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/queues.py", line 266, in _feed
send(obj)
PicklingError: Can't pickle optunity.solvers.ParticleSwarm.Particle: attribute lookup optunity.solvers.ParticleSwarm.Particle failed

The following list of errors should be caught at the highest level:

• TypeError
• ValueError

sklearn has a tendency of returning numpy stuff, which in turn have a tendency not to iterate as one might expect.

An error with make_solver and random search. Expecting instead a JSON message.

echo '{"make_solver":{"solver_name":"random search"}}' | python -m optunity.piped
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/jaak/git/optunity/optunity/piped.py", line 482, in <module>
    make_solver(startup_msg['make_solver'])
  File "/home/jaak/git/optunity/optunity/piped.py", line 389, in make_solver
    optunity.make_solver(**solver_config)
  File "optunity/api.py", line 282, in make_solver
    return solvercls(*args, **kwargs)
TypeError: __init__() takes exactly 2 arguments (1 given)

When using grid search the make_solver works:

echo '{"make_solver":{"solver_name":"grid search"}}' | python -m optunity.piped
{"success": "true"}

PSO solver sometimes stops early, with less function evaluations than we allow.

Hi,

I'd like to know if it is possible to use optunity to find hyperparameter solutions, when these parameters are discrete.

Then is it possible with non-ordinal hyperparameters ?

Finally, if it is not possible, do you plan to implement it and in case the answer is affirmative, for when ?

Hello. When i am working on an example of Optunity+e1071 in R link, cv.setup give me an error:

Error in if (is.na(port) || port == 0) { :
missing value where TRUE/FALSE needed

How can i fix this issue?

Random search seems to ignore bounds. The problem is either in suggest_from_box or the static key order used in function evaluation.

Seed is taken as input but not passed to Python API in generate_folds call.

In optunity/api.py line 176 sys.float_info.min is set as default:

f = _wrap_hard_box_constraints(f, kwargs, sys.float_info.min)

However, sys.float_info.min is the "minimum positive normalized float" [1] which evaluates to 2.2250738585072014e-308 in my case. This causes maximize to to choose parameters that violate constraints if something like a negative mean squared error should be maximized. I suggest to replace this line by:

f = _wrap_hard_box_constraints(f, kwargs, -sys.float_info.max)

[1] https://docs.python.org/2/library/sys.html#sys.float_info

If Use 'TPE' solver, the code will return below error:

optunity\solvers\TPE.pyc in algo(_args, *_kwargs)
117 def algo(_args, *_kwargs):
118 seed = self.seed if self.seed else random.randint(0, 9999999999)
--> 119 return hyperopt.tpe.suggest(_args, seed=seed *_kwargs)
120
121 space = [hyperopt.hp.uniform(k, v[0], v[1]) for k, v in self.bounds.items()]

TypeError: suggest() got multiple values for keyword argument 'seed'

Is it possible to tune a classifier's class weight vector using optunity?

Currently the parameter space is defined in optunity's 'minimize' and 'maximize' by passing it the lower and upper bounds. This assumes the parameter is a single number (such as 'C' or 'gamma'). I am wondering whether it is possible to optimize a weight vector, for example, where the length of the vector equals the number of classes.

Thanks

Typically these loss functions are compatible with multiclass problems. To make this work, we would need to provide a more generic interface (or require preprocessing on the user end).

I'm currently leaning towards preprocessing on the user's end, e.g. have users select the predicted probability of the correct class prior to calling brier() or logloss().

When running 'grid search' from command line gives error when 'random search' works.

Grid search

echo '{"maximize" : {"solver_name" : "grid search", "num_evals": 5, "x":[0,10]} }' | python -m optunity.piped
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/jaak/git/optunity/optunity/piped.py", line 495, in <module>
    startup_msg.get('call_log', None))
  File "/home/jaak/git/optunity/optunity/piped.py", line 421, in max_or_min
    solution, rslt, solver = solve_fun(func, pmap=mgr.pmap, **kwargs)
  File "optunity/api.py", line 172, in maximize
    pmap=pmap)
  File "optunity/api.py", line 245, in optimize
    optimum = f.call_log[t(**solution)]
TypeError: __new__() got an unexpected keyword argument 'x'

Random search (works)

echo '{"maximize" : {"solver_name" : "random search", "num_evals": 5, "x":[0,10]} }' | python -m optunity.piped
[{"x": 4.90398335468162}, {"x": 2.9585346492371967}, {"x": 8.75973165625038}, {"x": 0.934354531998024}, {"x": 5.092791699936537}]
{"error_msg": "Broken pipe."}

Remove all traces of numpy dependency from optunity._cross_validation.

In Linux/mac we have a solution python -m optunity > /tmp/py2r. However, Windows does not support that (http://superuser.com/questions/430466/in-windows-can-i-redirect-stdout-to-a-named-pipe-in-command-line).

So an option is to pass the name of the pipe to python with a paramater, like:

python -m optunity.piped -p py2r

And in python use

f = open(r'\\.\pipe\py2r', 'w', 0)
...
f.write(...)

This would be only used in windows. Seems like the easiest approach or are there other options?

• links to notebooks in cross-validation guidepage

Not sure what happened but builds suddenly broke on both travis & rtd. Needs fixing ASAP (tomorrow).

Typical error when I try to run any function is:-

octave:1> optunity_example
error: 'optunity' undefined near line 5 column 11
error: called from:
error: /home/andrew/my_source_makes/optunity/wrappers/matlab/optunity_example.m at line 5, column 9

Output of debug is:-

octave:2> global DEBUG_OPTUNITY
octave:3> DEBUG_OPTUNITY=true
DEBUG_OPTUNITY = 1

Al the relevant folders in the optunity directory are in Octave's path environment. Any suggestions?

Currently nosetests only runs the tests in optunity/solvers/. For some reason those in optunity/ are being ignored.

We checked Windows setup with Dusan and found out there is an issue:

• the Python installation in Windows does not put python.exe into the system path
• however, it links .py files to python, so running any .py file will work

So python -m optunity.piped will not work in Windows, but a simple solution is to call optunity.piped from a separate script:

1. make optunity.piped's main code into a separate method (currently it is just under if __name__=='__main__'), like explained here http://www.artima.com/weblogs/viewpost.jsp?thread=4829
2. then create a script in the top folder that launches the new main method, e.g. run.py:

#!/usr/bin/env python
import optunity.piped
optunity.piped.main()

Then we will just execute run.py in Windows (or also in other systems). Note: shebang is ignored in Windows.

@claesenm if the solution looks fine to you let me know, I can easily do the implementation or you can do it :).

Include DEAP in the Optunity package and automate code conversion from 2 to 3 on install (setup.py).

Note: the fact DEAP needs 2 to 3 conversion may cause issues for non-Python users (e.g. in wrappers).

I got an error when I tried to save a value of which class is 'optunity.util.NT' with pickling. My OS is windows 7 64 bit and use Python 2.7.2. I show the code below. Note that I got a similar error when I tried parallelization using optunity.pmap: pickling error.

---code
optimal_pars, info, optimal_pars2 = optunity.maximize_structured(svm_acc,search_space=args,num_evals=num_evals)

---this works (type:dictionary)
with open('test1','wb') as f:
pickle.dump(optimal_pars,f)

---this doesn't work (type:optunity.util.NT)
with open('test2','wb') as f:
pickle.dump(info,f)

---error
Traceback (most recent call last):
File "", line 2, in
File "C:\Python27\lib\pickle.py", line 1370, in dump
Pickler(file, protocol).dump(obj)
File "C:\Python27\lib\pickle.py", line 224, in dump
self.save(obj)
File "C:\Python27\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 401, in save_reduce
save(args)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 562, in save_tuple
save(element)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 748, in save_global
(obj, module, name))
PicklingError: Can't pickle <class 'optunity.util.NT'>: it's not found as optunity.util.NT

optunity.manual()
Traceback (most recent call last):
File "", line 1, in
File "/data/svn/claesenm/python/optunity/optunity/api.py", line 92, in manual
man = solver_registry._manual_lines()
AttributeError: 'module' object has no attribute '_manual_lines'

IPython crashes if the parallelized function outputs anything to stdout or stderr. This is an IPython issue that we can't fix ourselves.

More info at: ipython/ipython#2438

A workaround is to use IPython's own parallel features:
http://nbviewer.ipython.org/github/vals/scilife-python-course/blob/master/parallel%20python.ipynb

Would be useful to have call example for optimize function in piped
http://optunity.readthedocs.org/en/latest/api/optunity.piped.html#optimize

http://optunity.readthedocs.org/en/latest/api/optunity.solvers.html#optunity.solvers.ParticleSwarm

Might be useful to have an example of ParticleSwarm. Mainly for showing that the contents of kwargs is box constraints (currently there is also no comment about kwargs under ParticleSwarm).

optunity.piped works but uses way too many resources currently when doing vector evaluations (1 thread per evaluation, solvers like grid search can easily have thousands ...). Need to use concurrent.future backport in 2.x

cfr. http://stackoverflow.com/a/8797256/2148672

I am getting the following error by setting the aggregator option to opt.cross_validation.identity

---------------------------------------------------------------------------
     33 # Define Parameter Tuning
---> 34 optimal_pars_clf_sgd, _, _ = opt.maximize(clf_sgd_cv, num_evals=n_hyperparams_evals, alpha=[0.001, .1], l1_ratio=[0., 1.])
     35 
     36 # Train model on the Inner Training Set with Tuned Hyperparameters

../local/lib/python2.7/site-packages/optunity/api.pyc in maximize(f, num_evals, solver_name, pmap, **kwargs)
    179     solver = make_solver(**suggestion)
    180     solution, details = optimize(solver, f, maximize=True, max_evals=num_evals,
--> 181                                  pmap=pmap)
    182     return solution, details, suggestion
    183 

../local/lib/python2.7/site-packages/optunity/api.pyc in optimize(solver, func, maximize, max_evals, pmap)
    243     time = timeit.default_timer()
    244     try:
--> 245         solution, report = solver.optimize(f, maximize, pmap=pmap)
    246     except fun.MaximumEvaluationsException:
    247         # early stopping because maximum number of evaluations is reached

../local/lib/python2.7/site-packages/optunity/solvers/ParticleSwarm.pyc in optimize(self, f, maximize, pmap)
    257             fitnesses = pmap(evaluate, list(map(self.particle2dict, pop)))
    258             for part, fitness in zip(pop, fitnesses):
--> 259                 part.fitness = fit*fitness
    260                 if not part.best or part.best_fitness < part.fitness:
    261                     part.best = part.position

TypeError: can't multiply sequence by non-int of type 'float'

Here is my code

import optunity as opt
from optunity.metrics import _recall, contingency_table
from sklearn.linear_model import SGDClassifier
import numpy as np

n_in = 1
k_in = 2
n_hyperparams_evals = 10

clf_sgd = SGDClassifier(
            penalty="elasticnet",
            shuffle=True,
            n_iter=500,
            fit_intercept=True,
            learning_rate="optimal")

# Define Inner CV
cv_decorator = opt.cross_validated(x=X, y=Y.values, 
                                   num_folds=k_in, num_iter=n_in,
                                   strata=[Y[Y==1].index.values], 
                                   regenerate_folds=True,
                                   aggregator=opt.cross_validation.identity)

def obj_fun_clf_sgd(x_train, y_train, x_test, y_test, alpha, l1_ratio):
    model = clf_sgd.set_params(l1_ratio=l1_ratio, alpha=alpha).fit(x_train, y_train)
    y_pred = model.predict(x_test)
    score = _recall(contingency_table(y_test,y_pred))
    return score

clf_sgd_cv = cv_decorator(obj_fun_clf_sgd)

# Define Parameter Tuning
optimal_pars_clf_sgd, _, _ = opt.maximize(clf_sgd_cv, num_evals=n_hyperparams_evals, alpha=[0.001, .1], l1_ratio=[0., 1.])

# Train model on the Inner Training Set with Tuned Hyperparameters
optimal_model_clf_sgd = clf_sgd.set_params(**optimal_pars_clf_sgd).fit(X, Y.values)

The objective is to keep track of all the scores from the various folds. Is it a bug? or am I using incorrectly the API ?

Thanks in advance

$jython -Dpython.path=$PYTHONPATH examples_parabola.py 
Traceback (most recent call last):
  File "examples_parabola.py", line 6, in <module>
    optimal_pars, details, _ = optunity.maximize(f, num_evals=200, x=[-5, 5], y=[-5, 5])
  File "/data/svn/claesenm/python/optunity/optunity/api.py", line 170, in maximize
    solution, details = optimize(solver, f, maximize=True, max_evals=num_evals,
  File "/data/svn/claesenm/python/optunity/optunity/api.py", line 244, in optimize
    optimum = f.call_log[t(**solution)]
KeyError: args(x=0.04816650353569685, y=0.010200396673881285)

Maybe a rounding issue? Maybe something else.

Having some questions about the usage of Optunity, i was not able to find any mailing list or discussion board. Am i missing something?

MATLAB blocks at accepting the socket connection. Strange, because it works on other pairs of Windows/MATLAB versions.

Further testing required to diagnose the problem. Maybe due to Java.

Hi,
installed optunity and deap through pip and tried to run git parabola.py example but got a few errors, i am using Python 3.4, Numpy 1.10.0.dev-9671491 (git-version) and DEAP 1.0.1.

with cma-es algorithm i got this error:

File "/usr/local/lib/python3.4/dist-packages/deap/cma.py", line 84, in init
self.dim = len(self.centroid)
TypeError: len() of unsized object*

After an investigation i tried to correct it by passing a list instead of a dict_values type at the CMAES.py, looked like this:

if self.lambda_:
            strategy = deap.cma.Strategy(centroid=list(self.start.values()),
                                            sigma=self.sigma, lambda_=self.lambda_)
        else:
            strategy = deap.cma.Strategy(centroid=list(self.start.values()),
                                            sigma=self.sigma)

then, i got another error:

Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/optunity/api.py", line 245, in optimize
solution, report = solver.optimize(f, maximize, pmap=pmap)
File "/usr/local/lib/python3.4/dist-packages/optunity/solvers/CMAES.py", line 156, in optimize
halloffame=hof, verbose=False)
File "/usr/local/lib/python3.4/dist-packages/deap/algorithms.py", line 428, in eaGenerateUpdate
for ind, fit in zip(population, fitnesses):
File "/usr/local/lib/python3.4/dist-packages/optunity/solvers/CMAES.py", line 149, in evaluate
individual)])),)
File "/usr/local/lib/python3.4/dist-packages/optunity/functions.py", line 286, in wrapped_f
value = f(_args, *_kwargs)
File "/usr/local/lib/python3.4/dist-packages/optunity/functions.py", line 338, in wrapped_f
raise MaximumEvaluationsException(max_evals)
optunity.functions.MaximumEvaluationsException: 100

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "parabola.py", line 24, in
solver_name=solver)
File "/usr/local/lib/python3.4/dist-packages/optunity/api.py", line 212, in minimize
pmap=pmap)
File "/usr/local/lib/python3.4/dist-packages/optunity/api.py", line 254, in optimize
solution = operator.itemgetter(index)(f.call_log.keys())._asdict()
TypeError: 'KeysView' object does not support indexing

tried to run /usr/local/lib/python3.4/dist-packages/optunity/tests/test_solvers.py but got same errors:

Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/optunity/api.py", line 245, in optimize
solution, report = solver.optimize(f, maximize, pmap=pmap)
File "/usr/local/lib/python3.4/dist-packages/optunity/solvers/CMAES.py", line 156, in optimize
halloffame=hof, verbose=False)
File "/usr/local/lib/python3.4/dist-packages/deap/algorithms.py", line 428, in eaGenerateUpdate
for ind, fit in zip(population, fitnesses):
File "/usr/local/lib/python3.4/dist-packages/optunity/solvers/CMAES.py", line 149, in evaluate
individual)])),)
File "/usr/local/lib/python3.4/dist-packages/optunity/functions.py", line 286, in wrapped_f
value = f(_args, *_kwargs)
File "/usr/local/lib/python3.4/dist-packages/optunity/functions.py", line 338, in wrapped_f
raise MaximumEvaluationsException(max_evals)
optunity.functions.MaximumEvaluationsException: 100

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/optunity/tests/test_solvers.py", line 16, in
solver_name=solver)
File "/usr/local/lib/python3.4/dist-packages/optunity/api.py", line 181, in maximize
pmap=pmap)
File "/usr/local/lib/python3.4/dist-packages/optunity/api.py", line 254, in optimize
solution = operator.itemgetter(index)(f.call_log.keys())._asdict()
TypeError: 'KeysView' object does not support indexing

In the doc the first CV example returns 0.0. Maybe we could have a more practical example there?
http://optunity.readthedocs.org/en/latest/user/cross_validation.html

@opt.cross_validated(x=data, y=labels, num_folds=3)
def cved(x_train, y_train, x_test, y_test):
    train(x_train, y_train)
    predict(x_test)
    return 0.0

cved()

As explained here: http://stackoverflow.com/q/30539388/2148672

I think it's a matter of missing dependencies. I issued the following commands:

git clone https://github.com/claesenm/optunity.git
cd optunity/wrappers
R CMD build R
R CMD INSTALL optunity_0.6.tar.gz 

Error:

* installing to library ‘/home/badn/R/x86_64-pc-linux-gnu-library/3.1’
ERROR: dependencies ‘ROCR’, ‘enrichvs’ are not available for package ‘optunity’
* removing ‘/home/badn/R/x86_64-pc-linux-gnu-library/3.1/optunity’```

DIRECT solver not fully implemented. Should not be used yet.

We use Python's (standard) multiprocessing library for interthread communication (via multiprocessing.Queue). Unfortunately, Jython currently doesn't support this library.

A transition to something else would solve this problem.

A python problem: lambda is a reserved keyword. Would be nice to have a workaround. For at least API calls.

echo '{"optimize" : {"max_evals": 0}, "solver": {"solver_name" : "grid search", "lambda":[0,10]}}' | python -m optunity.piped
Exception in thread FutureThread:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "optunity/parallel.py", line 131, in Wrapper
    self.__result=func(*param)
  File "optunity/communication.py", line 157, in wrap
    result = f(*args)
  File "optunity/functions.py", line 357, in wrapped_f
    return f(**dict([(k, v) for k, v in zip(keys, args)]))
  File "optunity/functions.py", line 232, in wrapped_f
    wrapped_f.argtuple = collections.namedtuple('args', wrapped_f.keys)
  File "/usr/lib/python2.7/collections.py", line 334, in namedtuple
    'keyword: %r' % name)
ValueError: Type names and field names cannot be a keyword: 'lambda'