Via setup.py.

Instead of runsolver_wrapper -> runsolver -> run_instance -> algorithm_wrapper -> algorithm it should be runsolver_wrapper -> runsolver -> algorithms_wrapper (via CLI) -> algorithm

Th algorithm_wrapper should return a SMAClike string and gets configuration, seed, number of folds, current fold as an input. The algorithm_wrapper is responsible to load its data and run the algorithm on one fold. Depending on the benchmark a user need to define her own wrapper.

By this we can save a lot of custom code in every parse function. The way to go is to first parse the default config, then parse the experiment config and then add the cli arguments.

There are three kinds of plots:

• Can plot one optimizer run on one benchmark
• Can compare multiple runs on one benchmark
• Can compare multiple runs of one optimizer with multiple runs of another optimizer on one benchmark

To manage all possible plotfunctions, implement them as Plugins and HPOlib-plot can easily select and create possible plots.

Having optimized an algorithm one might be interested in the test performance of the found configuration.
HPOlib-run could have an additional --validate flag that calls the target algorithm with the best found configuration params, e.g. <anyBenchmarkWrapper>.py --params -a 23 -b -0.1234 --test. This wrapper now loads it's testdata and returns test performance.
This is optional for a benchmark, but can be useful for evaluation.

Questions:

• How do we handle the computing time of this single run? Training on the whole training/validation data might take longer than any of previous runs?
• If specified number of runs, is this a run?
• If specified a wallclock-time, how to know when to start calculating the test performance
• Rather implement a HPOlib-test script that calls the benchmark with the best configuration found in a .pkl file?
• Only validate offline?

Add badges like

• code quality
• version number
• many more

When analyzing results, provide a mechanism to automatically generate a .tex table in a flexible way (e.g. by table templates).

e.g. via a configuration parameter

Right now directories are created in the following fashion:

OPTIMIZER_SEED_TIMESTRING

change to:

• a custom directory path supplied via CLI
• CUSTOMSTRING1_OPTIMIZER_SEED_TIMESTRING

Looks like the best current guide to making a Python package is this one:
http://python-packaging-user-guide.readthedocs.org/en/latest/

How would you guys feel about making hpolib a Python package in this sense? I have wanted to import some of the benchmark functions, for example, for use by other modules.

Mainly it means moving the code to a subdir called e.g. hpolib and drafting a setup.py (which is mostly boilerplate).

This can be easily done in wrapping.py.

Useful for e.g.

• creating a compile directory for theano based experiments
• deleting this compile directory after the experiment
• check if all dependencies for an experiment are installed

Problematic, because a benchmark is not necessarily written in python, so we must probably call a whole script?

We currently have this in wrapping.py for the optimizer and in runsolver_wrapper.py for the runsolver. Also think about a useful default.

So far HPOlib contains a TPE -> SMAC and SMAC -> Spearmint converter.

• Do these scripts work correctly?
• We also need a Spearmint -> SMAC converter

One can use a converter:
HPOlib-convert --from TPE --to SMAC -f some/where/space.py [-s some/where/params.pcs]

Currently choosers can not return what they belief where the best configuration is. This is problematic for e.g. EntropySearch as belief and what's to be evaluated differ. It would be nice if one could transmit the belief as well.

via email from Manuel Watter

HPOlib-run --paramFile some/where/config.pb -o any/optimizer/version

or even make is possible to use a converter (#36) when the searchspace does not belong to the optimizer

Current runsolver (3.3.2) crashes for very short (<1sec) benchmark runs

via email from Manuel Watter.

Commit 9b46421 introduced the feature to override config parameters from the command line but is limited to the DEFAULT section. Generalize this to all sections.

Include to this repository all simple benchmarks, like

• More artificial testfunction that are commonly used
  • camelback
  • more testfunction
  • focus on low dimensional problems that can be visualized
• One benchmark not written in Python, e.g. Octave, C, Java, etc.
• SVM on grid, LDA on grid
• Everything that does not need any special libraries or GPU support

Right now it is not possible to have an experiment time limit, e.g. to reproduce the AutoWEKA experiment. We did the AutoWEKA experiments on a grid machine with a wall time limit of 30 hours.

Add documentation to both the plot scripts and the website. Make sure that all plot scripts have the same interface.

Example:

(hpolib)feurerm@aadpool4:~/HPOlib/Software/HPOlib/HPOlib/benchmarks/branin$ python ../../wrapping.py -o smac
[INFO] [11:26:04:HPOlib.check_before_start] Check config.cfg..
[INFO] [11:26:04:HPOlib.check_before_start] ..passed
[WARNING] [11:26:07:HPOlib.wrapping] You called -o smac, I am using optimizer defined in /home/feurerm/HPOlib/Software/HPOlib/HPOlib/optimizers/smac/smac_2_06_01-devDefault.cfg
[CRITICAL] [11:26:07:HPOlib.optimizers.smac.smac_2_06_01-dev_parser] Path to optimizer not found: /home/feurerm/HPOlib/Software/HPOlib/HPOlib/optimizers/smac/smac_2_06_01-dev_src
[INFO] [11:26:07:HPOlib.check_before_start] Check dependencies:
[INFO] [11:26:07:HPOlib.check_before_start] Runsolver..
[INFO] [11:26:07:HPOlib.check_before_start] ..passed
[INFO] [11:26:07:HPOlib.check_before_start] Check python_modules..
[WARNING] [11:26:16:HPOlib.check_before_start] CUDA not in $PATH
[INFO] [11:26:16:HPOlib.check_before_start] ..passed
[INFO] [11:26:16:HPOlib.smac_2_06_01-dev] ### INFORMATION ################################################################
[INFO] [11:26:16:HPOlib.smac_2_06_01-dev] # You're running /home/feurerm/HPOlib/Software/HPOlib/HPOlib/optimizers/smac/smac_2_06_01-dev_src                      #
[INFO] [11:26:16:HPOlib.smac_2_06_01-dev] #   Automatic Configurator Library ==> v2.06.01-development-643 (a1f71813a262) #
[INFO] [11:26:16:HPOlib.smac_2_06_01-dev] #             Random Forest Library ==> v1.05.01-development-95 (4a8077e95b21) #
[INFO] [11:26:16:HPOlib.smac_2_06_01-dev] #                             SMAC ==> v2.06.01-development-620 (9380d2c6bab9) #
[INFO] [11:26:16:HPOlib.smac_2_06_01-dev] # A newer version might be available, but not yet built in.                    #
[INFO] [11:26:16:HPOlib.smac_2_06_01-dev] # Please use this version only to reproduce our results on automl.org          #
[INFO] [11:26:16:HPOlib.smac_2_06_01-dev] ################################################################################
[INFO] [16:48:49:HPOlib.check_before_start] Check algorithm..
[INFO] [16:48:49:HPOlib.check_before_start] ..passed
[INFO] [16:48:49:HPOlib.wrapping] export PYTHONPATH=$PYTHONPATH:/home/feurerm/HPOlib/Software/HPOlib/HPOlib
/home/feurerm/HPOlib/Software/HPOlib/HPOlib/optimizers/smac/smac_2_06_01-dev_src/smac --numRun 1 --scenario-file /home/feurerm/HPOlib/Software/HPOlib/HPOlib/benchmarks/branin/smac_2_06_01-dev_1_2014-3-3--16-48-49-300838/scenario.txt --cutoffTime 3700 --intraInstanceObj MEAN --runObj QUALITY --algoExec "python /home/feurerm/HPOlib/Software/HPOlib/HPOlib/runsolver_wrapper.py" --execDir /home/feurerm/HPOlib/Software/HPOlib/HPOlib/benchmarks/branin/smac_2_06_01-dev_1_2014-3-3--16-48-49-300838 -p smac_2_06_01-dev/params.txt --numIterations 2147483647 --totalNumRunsLimit 200 --outputDirectory /home/feurerm/HPOlib/Software/HPOlib/HPOlib/benchmarks/branin/smac_2_06_01-dev_1_2014-3-3--16-48-49-300838 --numConcurrentAlgoExecs 1 --maxIncumbentRuns 2000 --retryTargetAlgorithmRunCount 0 --save-runs-every-iteration true --intensification-percentage 0 --rf-split-min 10 --deterministic true --instanceFile /home/feurerm/HPOlib/Software/HPOlib/HPOlib/benchmarks/branin/smac_2_06_01-dev_1_2014-3-3--16-48-49-300838/train.txt --testInstanceFile /home/feurerm/HPOlib/Software/HPOlib/HPOlib/benchmarks/branin/smac_2_06_01-dev_1_2014-3-3--16-48-49-300838/test.txt
[INFO] [16:48:49:HPOlib.wrapping] -----------------------RUNNING----------------------------------
[INFO] [16:48:49:HPOlib.wrapping] Optimizer runs with PID (+1): 3794
[ERR]:/bin/sh: 2: /home/feurerm/HPOlib/Software/HPOlib/HPOlib/optimizers/smac/smac_2_06_01-dev_src/smac: not found
[ERR]:[INFO] [16:48:49:HPOlib.wrapping] 
-----------------------END--------------------------------------
[INFO] [16:48:49:HPOlib.wrapping] Best result
Traceback (most recent call last):
  File "../../wrapping.py", line 394, in <module>
    main()
  File "../../wrapping.py", line 372, in main
    logger.info(trials.get_best())
  File "/home/feurerm/HPOlib/Software/HPOlib/HPOlib/Experiment.py", line 194, in get_best
    best_idx = self.get_arg_best()
  File "/home/feurerm/HPOlib/Software/HPOlib/HPOlib/Experiment.py", line 189, in get_arg_best
    raise ValueError("No best value found.")
ValueError: No best value found.

e.g.

• Hostname
• installed python packages
• date
• other information we can find out about the machine

Might be useful for reconstruction. We should have a look at OpenML for inspiration because they log information about the target machine.

• Allow multiple values to be returned and saved in the experiment pickle

These are:

• tpe.py
• tpecall.py
• smac.py
• spearmint.py
• randomtpe.py

So far, we try to import all the dependencies of all optimizers in check_before_start. This checks should be performed by a callback in the python script which builds the command line call for the respective optimizer.

We import numpy before checking whether it is installed.

Right now wrapping.py does not produce any output saying anything about the current state of the optimization process. The user needs to manually tail -f the optimizers output file and understand each optimizer outpout. As we do not want to pipe the optimizers output, because this is either too much or less information, we somehow need to produce and create our own output.

Suggestions:

• Print a dot for each iteration and incumbent changes
• Print commandline call and return string for each iteration (might become too messy)

Problem:

As we call the optimizer via CLI we do not have directly access to out process pipeline. This problem need to be somehow solved

We only use the Experiment.py in a sequential way. The Locker.py might slow down our workflow and is not necessary. We only need a replacement as soon as we switch to non-sequential optimization.

Start HPOlib like this:

HPOlib-run -o /home/feurerm/HPOlib/working_directory/hpolib/optimizers/smac/ --HPOLIB:memory_limit 1
HPOlib-run -o /home/feurerm/HPOlib/working_directory/hpolib/optimizers/spearmint/ --HPOLIB:memory_limit 1

=== Spearmint output ===

Running in wrapper mode for '/home/feurerm/HPOlib/Software/HPOlib/benchmarks/branin/spearmint_april2013_mod_1_2014-3-11--13-51-50-763114/jobs/00000000.pb'
Waiting to lock grid......acquired
Released lock on job grid.
Running python job.
Changed into dir /home/feurerm/HPOlib/Software/HPOlib/benchmarks/branin/spearmint_april2013_mod_1_2014-3-11--13-51-50-763114
[CRITICAL] [13:51:53:HPOlib.cv] args: (0L, {u'y': array([ 0.]), u'x': array([-5.])}) kwargs: {}
[INFO] [13:51:53:HPOlib.cv] Starting Cross validation
[INFO] [13:51:53:HPOlib.cv] {u'y': 0.0, u'x': -5.0}
[INFO] [13:51:53:HPOlib.cv] Calling command:
python /home/feurerm/thesis/virtualenvs/hpolib/lib/python2.7/site-packages/HPOlib-0.0.1-py2.7.egg/HPOlib/runsolver_wrapper.py 0 spearmint_april2013_mod 0 0 0 -y 0.0 -x -5.0
[INFO] [13:51:53:HPOlib.cv] --------------RUNNING RUNSOLVER_WRAPPER--------------
[ERROR] [13:51:53:HPOlib.cv] /home/feurerm/thesis/virtualenvs/hpolib/local/lib/python2.7/site-packages/numpy/lib/nanfunctions.py:514: FutureWarning: In Numpy 1.9 the sum along empty slices will be zero.  FutureWarning)
[INFO] [13:51:53:HPOlib.runsolver_wrapper] -----------------------RUNNING RUNSOLVER----------------------------
[CRITICAL] [13:51:53:HPOlib.runsolver_wrapper] We could not find anything matching our regexp. Last lines of your output:
None
Traceback (most recent call last):
  File "/home/feurerm/thesis/virtualenvs/hpolib/lib/python2.7/site-packages/HPOlib-0.0.1-py2.7.egg/HPOlib/runsolver_wrapper.py", line 387, in <module>
main()
  File "/home/feurerm/thesis/virtualenvs/hpolib/lib/python2.7/site-packages/HPOlib-0.0.1-py2.7.egg/HPOlib/runsolver_wrapper.py", line 357, in main
parse_output_files(cfg, run_instance_output, runsolver_output_file)
  File "/home/feurerm/thesis/virtualenvs/hpolib/lib/python2.7/site-packages/HPOlib-0.0.1-py2.7.egg/HPOlib/runsolver_wrapper.py", line 251, in parse_output_files
instance_wallclock_time = cfg.getfloat("HPOLIB", "runsolver_time_limit")
  File "/usr/lib/python2.7/ConfigParser.py", line 362, in getfloat
return self._get(section, float, option)
  File "/usr/lib/python2.7/ConfigParser.py", line 356, in _get
return conv(self.get(section, option))
ValueError: could not convert string to float: 

[ERROR] [13:51:53:HPOlib.cv] 
Traceback (most recent call last):
  File "/home/feurerm/thesis/virtualenvs/hpolib/local/lib/python2.7/site-packages/HPOlib-0.0.1-py2.7.egg/HPOlib/cv.py", line 107, in do_cv
    raise NotImplementedError("No result string available or result string doesn't contain SAT")
NotImplementedError: No result string available or result string doesn't contain SAT

Traceback (most recent call last):
    File "/home/feurerm/thesis/virtualenvs/hpolib/local/lib/python2.7/site-packages/HPOlib-0.0.1-py2.7.egg/HPOlib/cv.py", line 107, in do_cv
    NotImplementedError: No result string available or result string doesn't contain SAT

[ERROR] [13:51:53:HPOlib.cv] CV failed <type 'exceptions.NotImplementedError'> No result string available or result string doesn't contain SAT
[INFO] [13:51:53:HPOlib.cv] Finished CV
[INFO] [13:51:53:HPOlib.cv] Result: 1000.000000
/home/feurerm/thesis/virtualenvs/hpolib/local/lib/python2.7/site-packages/numpy/lib/nanfunctions.py:514: FutureWarning: In Numpy 1.9 the sum along empty slices will be zero.   FutureWarning)
Got result 1000.000000
Job file reloaded.
Completed successfully in 0.85 seconds. [1000.000000]
Waiting to lock grid......acquired
Released lock on job grid.

=== SMAC output ===

[INFO ] Initialization: Scheduling 1 run(s): [INFO ] Initialization: Scheduling run for config (0x000B) on instance 1 with seed -1 and captime 2.147483647E9 [WARN ] This version of SMAC hardcodes run length for calls to the target algorithm to 2147483647. [WARN ] [PROCESS-ERR] /home/feurerm/thesis/virtualenvs/hpolib/local/lib/python2.7/site-packages/numpy/lib/nanfunctions.py:514: FutureWarning: In Numpy 1.9 the sum along empty slices will be zero. [WARN ] [PROCESS-ERR] FutureWarning) [WARN ] [PROCESS-ERR] [INFO] [13:50:14:HPOlib.runsolver_wrapper] -----------------------RUNNING RUNSOLVER---------------------------- [WARN ] [PROCESS-ERR] [CRITICAL] [13:50:14:HPOlib.runsolver_wrapper] We could not find anything matching our regexp. Last lines of your output: [WARN ] [PROCESS-ERR] None [WARN ] [PROCESS-ERR] Traceback (most recent call last): [WARN ] [PROCESS-ERR] File "/home/feurerm/thesis/virtualenvs/hpolib/local/lib/python2.7/site-packages/HPOlib-0.0.1-py2.7.egg/HPOlib/runsolver_wrapper.py", line 387, in <module> [WARN ] [PROCESS-ERR] main() [WARN ] [PROCESS-ERR] File "/home/feurerm/thesis/virtualenvs/hpolib/local/lib/python2.7/site-packages/HPOlib-0.0.1-py2.7.egg/HPOlib/runsolver_wrapper.py", line 357, in main [WARN ] [PROCESS-ERR] parse_output_files(cfg, run_instance_output, runsolver_output_file) [WARN ] [PROCESS-ERR] File "/home/feurerm/thesis/virtualenvs/hpolib/local/lib/python2.7/site-packages/HPOlib-0.0.1-py2.7.egg/HPOlib/runsolver_wrapper.py", line 251, in parse_output_files [WARN ] [PROCESS-ERR] instance_wallclock_time = cfg.getfloat("HPOLIB", "runsolver_time_limit") [WARN ] [PROCESS-ERR] File "/usr/lib/python2.7/ConfigParser.py", line 362, in getfloat [WARN ] [PROCESS-ERR] return self._get(section, float, option) [WARN ] [PROCESS-ERR] File "/usr/lib/python2.7/ConfigParser.py", line 356, in _get [WARN ] [PROCESS-ERR] return conv(self.get(section, option)) [WARN ] [PROCESS-ERR] ValueError: could not convert string to float: [ERROR] Failed Run Detected Call: cd "/home/feurerm/HPOlib/Software/HPOlib/benchmarks/branin/smac_2_06_01-dev_1_2014-3-11--13-50-13-371410" ; python /home/feurerm/thesis/virtualenvs/hpolib/local/lib/python2.7/site-packages/HPOlib-0.0.1-py2.7.egg/HPOlib/runsolver_wrapper.py 0 0 2.147483647E9 2147483647 -1 -y '7.5' -x '2.5' [ERROR] Failed Run Detected output last 0 lines [ERROR] Output complete [INFO ] State saved for iteration 0 in /home/feurerm/HPOlib/Software/HPOlib/benchmarks/branin/smac_2_06_01-dev_1_2014-3-11--13-50-13-371410/scenario-SMAC-ac-false-cores1-cutoff2.147483647E9-2014-03-11/state-run1 [INFO ] Saving state took 34 ms [INFO ] Writing trajectory file to /home/feurerm/HPOlib/Software/HPOlib/benchmarks/branin/smac_2_06_01-dev_1_2014-3-11--13-50-13-371410/scenario-SMAC-ac-false-cores1-cutoff2.147483647E9-2014-03-11/traj-run-1 [INFO ] Total Reported Runtime: 0.0 (s), Total of Sum Max(runtime-cutoff,0): 0.0 (s) [INFO ] Total Walltime: 0.0 (s), Total of Sum Max(walltime - cutoff, 0): 0.0 (s) [INFO ] Total Difference between Walltime and Runtime (Sum of the amount of wallclock time - sum of the amount of reported CPU time) : 0.0 seconds [ERROR] Message: First Run Crashed : [<Instance:1, Seed:-1, Config:0x000B, Kappa:2.147483647E9> ==> <CRASHED, 2.147483647E9, 0.0, 0.0, -1,Wrapper did not output anything that matched our regex please see the manual for more information. Please try executing the wrapper directly and ensuring that some line starts with: "Results for ParamILS:" (case sensitive). In more gorey detail it needs to match the following Regular Expression: ^\s*(Final)?\s*[Rr]esult\s+(?:([Ff]or)|([oO]f))\s+(?:(HAL)|(ParamILS)|(SMAC)|([tT]his [wW]rapper)):> W:(0.638)] [ERROR] A serious problem occured during target algorithm execution and we are aborting execution ca.ubc.cs.beta.aclib.targetalgorithmevaluator.exceptions.TargetAlgorithmAbortException: First Run Crashed : [<Instance:1, Seed:-1, Config:0x000B, Kappa:2.147483647E9> ==> <CRASHED, 2.147483647E9, 0.0, 0.0, -1,Wrapper did not output anything that matched our regex please see the manual for more information. Please try executing the wrapper directly and ensuring that some line starts with: "Results for ParamILS:" (case sensitive). In more gorey detail it needs to match the following Regular Expression: ^\s*(Final)?\s*[Rr]esult\s+(?:([Ff]or)|([oO]f))\s+(?:(HAL)|(ParamILS)|(SMAC)|([tT]his [wW]rapper)):> W:(0.638)] at ca.ubc.cs.beta.aclib.targetalgorithmevaluator.decorators.safety.AbortOnFirstRunCrashTargetAlgorithmEvaluator.validate(AbortOnFirstRunCrashTargetAlgorithmEvaluator.java:55) ~[aclib.jar:na] at ca.ubc.cs.beta.aclib.targetalgorithmevaluator.decorators.safety.AbortOnFirstRunCrashTargetAlgorithmEvaluator.evaluateRun(AbortOnFirstRunCrashTargetAlgorithmEvaluator.java:34) ~[aclib.jar:na] at ca.ubc.cs.beta.aclib.targetalgorithmevaluator.decorators.AbstractForEachRunTargetAlgorithmEvaluatorDecorator.evaluateRun(AbstractForEachRunTargetAlgorithmEvaluatorDecorator.java:31) ~[aclib.jar:na] at ca.ubc.cs.beta.aclib.targetalgorithmevaluator.decorators.AbstractForEachRunTargetAlgorithmEvaluatorDecorator.evaluateRun(AbstractForEachRunTargetAlgorithmEvaluatorDecorator.java:31) ~[aclib.jar:na] at ca.ubc.cs.beta.aclib.targetalgorithmevaluator.decorators.AbstractForEachRunTargetAlgorithmEvaluatorDecorator.evaluateRun(AbstractForEachRunTargetAlgorithmEvaluatorDecorator.java:31) ~[aclib.jar:na] at ca.ubc.cs.beta.aclib.targetalgorithmevaluator.decorators.AbstractTargetAlgorithmEvaluatorDecorator.evaluateRun(AbstractTargetAlgorithmEvaluatorDecorator.java:46) ~[aclib.jar:na] at ca.ubc.cs.beta.aclib.targetalgorithmevaluator.decorators.functionality.OutstandingEvaluationsTargetAlgorithmEvaluatorDecorator.evaluateRun(OutstandingEvaluationsTargetAlgorithmEvaluatorDecorator.java:45) ~[aclib.jar:na] at ca.ubc.cs.beta.aclib.targetalgorithmevaluator.decorators.debug.CheckForDuplicateRunConfigDecorator.evaluateRun(CheckForDuplicateRunConfigDecorator.java:45) ~[aclib.jar:na] at ca.ubc.cs.beta.aclib.targetalgorithmevaluator.decorators.helpers.WalltimeAsRuntimeTargetAlgorithmEvaluatorDecorator.evaluateRun(WalltimeAsRuntimeTargetAlgorithmEvaluatorDecorator.java:52) ~[aclib.jar:na] at ca.ubc.cs.beta.aclib.targetalgorithmevaluator.decorators.helpers.KillCaptimeExceedingRunsRunsTargetAlgorithmEvaluatorDecorator.evaluateRun(KillCaptimeExceedingRunsRunsTargetAlgorithmEvaluatorDecorator.java:61) ~[aclib.jar:na] at ca.ubc.cs.beta.aclib.targetalgorithmevaluator.decorators.AbstractForEachRunTargetAlgorithmEvaluatorDecorator.evaluateRun(AbstractForEachRunTargetAlgorithmEvaluatorDecorator.java:31) ~[aclib.jar:na] at ca.ubc.cs.beta.aclib.targetalgorithmevaluator.decorators.AbstractTargetAlgorithmEvaluatorDecorator.evaluateRun(AbstractTargetAlgorithmEvaluatorDecorator.java:40) ~[aclib.jar:na] at ca.ubc.cs.beta.aclib.initialization.classic.ClassicInitializationProcedure.evaluateRun(ClassicInitializationProcedure.java:125) ~[aclib.jar:na] at ca.ubc.cs.beta.aclib.initialization.classic.ClassicInitializationProcedure.evaluateRun(ClassicInitializationProcedure.java:107) ~[aclib.jar:na] at ca.ubc.cs.beta.aclib.initialization.classic.ClassicInitializationProcedure.run(ClassicInitializationProcedure.java:86) ~[aclib.jar:na] at ca.ubc.cs.beta.smac.configurator.AbstractAlgorithmFramework.run(AbstractAlgorithmFramework.java:414) ~[smac.jar:na] at ca.ubc.cs.beta.smac.executors.SMACExecutor.oldMain(SMACExecutor.java:127) [smac.jar:na] at ca.ubc.cs.beta.smac.executors.SMACExecutor.main(SMACExecutor.java:91) [smac.jar:na] [ERROR] We tried to call the target algorithm wrapper, but this call failed. [ERROR] The problem is (most likely) somewhere in the wrapper. [ERROR] There is also possibly additional error information above (in this log) [ERROR] The easiest way to debug this problem is to manually execute the call we tried and see why it did not return the correct result [ERROR] The required syntax is something like "Final Result for ParamILS: x,x,x,x,x".); [ERROR] Specifically the regex we are matching is ^\s*(Final)?\s*[Rr]esult\s+(?:([Ff]or)|([oO]f))\s+(?:(HAL)|(ParamILS)|(SMAC)|([tT]his [wW]rapper)): [INFO ] Exiting SMAC with failure. Log: /home/feurerm/HPOlib/Software/HPOlib/benchmarks/branin/smac_2_06_01-dev_1_2014-3-11--13-50-13-371410/scenario-SMAC-ac-false-cores1-cutoff2.147483647E9-2014-03-11/log-run1.txt [INFO ] For a list of available commands use: --help [INFO ] Please see above for the available options. Further information is available in the following documents: [INFO ] - The FAQ (doc/faq.pdf) contains commonly asked questions regarding troubleshooting, and usage. [INFO ] - The Quickstart Guide (doc/quickstart.pdf) gives a simple example for getting up and running. [INFO ] - The Manual (doc/manual.pdf) contains detailed information on file format semantics. [INFO ] Returning with value: 255 Error occured running SMAC ( TargetAlgorithmAbortException : First Run Crashed : [<Instance:1, Seed:-1, Config:0x000B, Kappa:2.147483647E9> ==> <CRASHED, 2.147483647E9, 0.0, 0.0, -1,Wrapper did not output anything that matched our regex please see the manual for more information. Please try executing the wrapper directly and ensuring that some line starts with: "Results for ParamILS:" (case sensitive). In more gorey detail it needs to match the following Regular Expression: ^\s*(Final)?\s*[Rr]esult\s+(?:([Ff]or)|([oO]f))\s+(?:(HAL)|(ParamILS)|(SMAC)|([tT]his [wW]rapper)):> W:(0.638)]) Error Log: /home/feurerm/HPOlib/Software/HPOlib/benchmarks/branin/smac_2_06_01-dev_1_2014-3-11--13-50-13-371410/scenario-SMAC-ac-false-cores1-cutoff2.147483647E9-2014-03-11/log-run1.txt

Script to automatically convert the SMAC .pcs format to hyperopts pyll format.

Make it possible to call

HPOlib-run -o some/optimizer/version --OPTIMIZER:space=searchme.pcs --HPOLIB:number_of_jobs=20 --HPOlib:function="python ../branin.py" -s 23

For easy benchmarks there is no need to have a config.cfg. So we need to put some default values for number_of_jobs and result_on_terminate.

This field is not needed anymore as we deleted run_instance.py