tobegit3hub / advisor Goto Github PK

Hi,
does advisor server have any way to persist trails to a storage back-end?

If such an option isn't available atm could you point me to the source code best suited to extend with such capabilities?

System Information

OS: Ubuntu 18.04.1 running in an LXD container
LXD: 3.6 (snap)
conda: 4.5.11
Python: 3.7.0.final.0
advisor: 0.1.6

Issue

I have installed advisor through pip on a fresh Python 3.7 miniconda installation. Running advisor_admin server start results in the following error:

Traceback (most recent call last):
  File "/home/ubuntu/.conda/envs/ubuntu/bin/advisor_admin", line 7, in <module>
    from advisor_client.commandline.admin_command import main
  File "/home/ubuntu/.conda/envs/ubuntu/lib/python3.7/site-packages/advisor_client/commandline/admin_command.py", line 110
    except subprocess.CalledProcessError, e:
                                        ^
SyntaxError: invalid syntax

Fix

The error can be resolved by simply changing the following line in admin_command.py:

    except subprocess.CalledProcessError, e:

to

    except subprocess.CalledProcessError as e:

After that change, advisor_admin server start runs on my system without errors.

Note that as is not supported in Python versions below 2.5 (according to an answer in https://stackoverflow.com/questions/2535760/python-try-except-comma-vs-as-in-except).

Pull Request

Note that I have created a pull request for the proposed change.

When I ran the code

client.complete_trial_with_one_metric(trial, some_value)

I got the following error:

OperationalError: no such table: suggestion_trialmetric

BYW, when I open the webpage of server, there are a lot of pending studies and trials, is there any way to delete them?

(black_box_optimization) ~/code/advisor (master ✘)✹ ᐅ python examples/simple_function_example/min_function.py Id: 15, name: Study, study_configuration: {"maxTrials": 20, "randomInitTrials": 10, "params": [{"parameterName": "x", "minValue": -10, "type": "DOUBLE", "maxValue": 10, "scallingType": "LINEAR"}], "goal": "MINIMIZE", "maxParallelTrials": 1}, algorithm: BayesianOptimization Traceback (most recent call last): File "examples/simple_function_example/min_function.py", line 49, in <module> trial = client.get_suggestions(study.id, 1)[0] IndexError: list index out of range

Installing advisor through pip install advisor fails with the following error:

ERROR: More than one .egg-info directory found in /tmp/pip-pip-egg-info-<somehash>

As @phdru pointed out on SO, there are two setup() calls in setup.py

setup.py:22
setup.py:29

In the examples like tensorflow_mnist or scikit_learn in advisor_client, the config file has the goal MINIMIZE. In the scripts the metric used is accuracy. Am I missing something? Shouldn't the goal be maximize?

I have parameter defined like:

        {
            "parameterName": "l1024",
            "type": "INTEGER",
            "minValue": 0,
            "maxValue": 2,
            "feasiblePoints": "",
            "scallingType": "LINEAR"
        },

but I got suggestion l1024 = 16

System Information

OS: Ubuntu 18.04.1 running in an LXD container
LXD: 3.6 (snap)
conda: 4.5.11
Python: 3.7.0.final.0
advisor: 0.1.6
requests: 2.19.1

Issue

I am having trouble running the examples in the repo. Executing the following commands:

git clone --depth 1 https://github.com/tobegit3hub/advisor.git && cd ./advisor/
advisor run -f ./advisor_client/examples/python_function/config.json

raises a json.decoder.JSONDecodeError:

2018-10-23 01:27:13 INFO Run with config: {'name': 'demo', 'algorithm': 'BayesianOptimization', 'trialNumber': 10, 'path': './advisor_client/examples/python_function/', 'command': './min_function.py', 'search_space': {'goal': 'MINIMIZE', 'randomInitTrials': 3, 'params': [{'parameterName': 'x', 'type': 'DOUBLE', 'minValue': -10.0, 'maxValue': 10.0}]}}
Traceback (most recent call last):
  File "/home/ubuntu/.conda/envs/ubuntu/bin/advisor", line 11, in <module>
    sys.exit(main())
  File "/home/ubuntu/.conda/envs/ubuntu/lib/python3.7/site-packages/advisor_client/commandline/command.py", line 194, in main
    args.func(args)
  File "/home/ubuntu/.conda/envs/ubuntu/lib/python3.7/site-packages/advisor_client/commandline/command.py", line 131, in run_with_file
    launcher.run()
  File "/home/ubuntu/.conda/envs/ubuntu/lib/python3.7/site-packages/advisor_client/runner/runner_launcher.py", line 56, in run
    self.run_config_dict["algorithm"])
  File "/home/ubuntu/.conda/envs/ubuntu/lib/python3.7/site-packages/advisor_client/client.py", line 47, in get_or_create_study
    study_exist = response.json()["exist"]
  File "/home/ubuntu/.conda/envs/ubuntu/lib/python3.7/site-packages/requests/models.py", line 896, in json
    return complexjson.loads(self.text, **kwargs)
  File "/home/ubuntu/.conda/envs/ubuntu/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/home/ubuntu/.conda/envs/ubuntu/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/ubuntu/.conda/envs/ubuntu/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)

This error is not particular to the python_function example. Attempting to run the tensorflow_mnist example via advisor run -f ./advisor_client/examples/tensorflow_mnist/config.json raises the same error:

2018-10-23 01:29:18 INFO Run with config: {'name': 'simple1', 'algorithm': 'RandomSearch', 'trialNumber': 3, 'concurrency': 1, 'path': '~/code/advisor/advisor_client/examples/tensorflow_mnist/', 'command': './tensorflow_mnist.py', 'search_space': {'goal': 'MINIMIZE', 'randomInitTrials': 1, 'maxTrials': 5, 'maxParallelTrials': 1, 'params': [{'parameterName': 'batch_size', 'type': 'DISCRETE', 'minValue': 0, 'maxValue': 0, 'feasiblePoints': '16, 32, 64, 128, 256', 'scalingType': 'LINEAR'}, {'parameterName': 'batch_number', 'type': 'INTEGER', 'minValue': 500, 'maxValue': 3000, 'feasiblePoints': '', 'scalingType': 'LINEAR'}, {'parameterName': 'optimizer', 'type': 'CATEGORICAL', 'minValue': 0, 'maxValue': 0, 'feasiblePoints': 'sgd, adagrad, adam, ftrl', 'scalingType': 'LINEAR'}, {'parameterName': 'learning_rate', 'type': 'DOUBLE', 'minValue': 0.01, 'maxValue': 0.1, 'feasiblePoints': '', 'scalingType': 'LINEAR'}]}}
Traceback (most recent call last):
  File "/home/ubuntu/.conda/envs/ubuntu/bin/advisor", line 11, in <module>
    sys.exit(main())
  File "/home/ubuntu/.conda/envs/ubuntu/lib/python3.7/site-packages/advisor_client/commandline/command.py", line 194, in main
    args.func(args)
  File "/home/ubuntu/.conda/envs/ubuntu/lib/python3.7/site-packages/advisor_client/commandline/command.py", line 131, in run_with_file
    launcher.run()
  File "/home/ubuntu/.conda/envs/ubuntu/lib/python3.7/site-packages/advisor_client/runner/runner_launcher.py", line 56, in run
    self.run_config_dict["algorithm"])
  File "/home/ubuntu/.conda/envs/ubuntu/lib/python3.7/site-packages/advisor_client/client.py", line 47, in get_or_create_study
    study_exist = response.json()["exist"]
  File "/home/ubuntu/.conda/envs/ubuntu/lib/python3.7/site-packages/requests/models.py", line 896, in json
    return complexjson.loads(self.text, **kwargs)
  File "/home/ubuntu/.conda/envs/ubuntu/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/home/ubuntu/.conda/envs/ubuntu/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/ubuntu/.conda/envs/ubuntu/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)

Perhaps there's an error on my part? I'm afraid I'm quite new to advisor.

Installation of advisor with command "pip3 install advisor" failed, but with command "pip2 install advisor" succeeded. Whether the advisor is currently only compatible with python2 and not python3.

If I use pip install advisor, it warns ERROR: More than one .egg-info directory found in C:\Users\xxx\AppData\Local\Temp\pip-pip-egg-info-y3dtu1ju.
@tobegit3hub #45

Newb here.
A quick walkthrough of something like the Chocolate Chip cookie recipe optimization INCLUDING INSTALL OF everything would be really really really useful. This has been really painful to try to figure out how to get this going and I still don't feel like I'm even that close.

I attended a talk on this about 2 years ago at Google X was told this "should" be easy enough for even a dumb mechanical engineer.

same question

macbook:~ andrewssobral$ advisor_admin server start
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/bin/advisor_admin", line 7, in
from advisor_client.commandline.admin_command import main
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/advisor_client/commandline/admin_command.py", line 110
except subprocess.CalledProcessError, e:
^
SyntaxError: invalid syntax
macbook:~ andrewssobral$
Any status update for this? Thanks

I did a simple test to minimize x^2-3x+2 using BayesianOpt.

It seems incapable of minimizing. It always maximize the function.
I can maximize the inverse of the function.

examples.zip

This repo https://github.com/fmfn/BayesianOptimization is a good python implementation of BO, but unfortunately it only supports Double type inputs without support for Integer、Discrete and Categorical data types. So i came here for solutions, but unfortunately again, advisor currently can't assign specific acquisition function to use and can't assign parameters for GaussianProcessRegressor from sklearn either. Are you considering adding those features recently?

Now we need to install the advisor_client with source code and python ./setup.py develop.

/usr/bin/python3.5 /home/hadoop/PycharmProjects/advisor/advisor_server/manage.py runserver 0.0.0.0:8000
Performing system checks...

Unhandled exception in thread started by <function check_errors..wrapper at 0x7f1339688840>
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/django/utils/autoreload.py", line 225, in wrapper
fn(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/django/core/management/commands/runserver.py", line 121, in inner_run
self.check(display_num_errors=True)
File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 364, in check
include_deployment_checks=include_deployment_checks,
File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 351, in _run_checks
return checks.run_checks(**kwargs)
File "/usr/local/lib/python3.5/dist-packages/django/core/checks/registry.py", line 73, in run_checks
new_errors = check(app_configs=app_configs)
File "/usr/local/lib/python3.5/dist-packages/django/core/checks/urls.py", line 40, in check_url_namespaces_unique
all_namespaces = _load_all_namespaces(resolver)
File "/usr/local/lib/python3.5/dist-packages/django/core/checks/urls.py", line 57, in _load_all_namespaces
url_patterns = getattr(resolver, 'url_patterns', [])
File "/usr/local/lib/python3.5/dist-packages/django/utils/functional.py", line 36, in get
res = instance.dict[self.name] = self.func(instance)
File "/usr/local/lib/python3.5/dist-packages/django/urls/resolvers.py", line 536, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
File "/usr/local/lib/python3.5/dist-packages/django/utils/functional.py", line 36, in get
res = instance.dict[self.name] = self.func(instance)
File "/usr/local/lib/python3.5/dist-packages/django/urls/resolvers.py", line 529, in urlconf_module
return import_module(self.urlconf_name)
File "/usr/lib/python3.5/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 986, in _gcd_import
File "", line 969, in _find_and_load
File "", line 958, in _find_and_load_unlocked
File "", line 673, in _load_unlocked
File "", line 665, in exec_module
File "", line 222, in _call_with_frames_removed
File "/home/hadoop/PycharmProjects/advisor/advisor_server/advisor/urls.py", line 29, in
url(r'^suggestion/', include('suggestion.urls')),
File "/usr/local/lib/python3.5/dist-packages/django/urls/conf.py", line 34, in include
urlconf_module = import_module(urlconf_module)
File "/usr/lib/python3.5/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 986, in _gcd_import
File "", line 969, in _find_and_load
File "", line 958, in _find_and_load_unlocked
File "", line 673, in _load_unlocked
File "", line 665, in exec_module
File "", line 222, in _call_with_frames_removed
File "/home/hadoop/PycharmProjects/advisor/advisor_server/suggestion/urls.py", line 2, in
from . import views
File "/home/hadoop/PycharmProjects/advisor/advisor_server/suggestion/views.py", line 16, in
from suggestion.algorithm.random_search import RandomSearchAlgorithm
File "/home/hadoop/PycharmProjects/advisor/advisor_server/suggestion/algorithm/random_search.py", line 6, in
from base_algorithm import BaseSuggestionAlgorithm
ImportError: No module named 'base_algorithm'

I get lot of warnings from sklearn gaussian process:

/usr/local/lib/python3.6/dist-packages/sklearn/gaussian_process/gpr.py:457: UserWarning: fmin_l_bfgs_b terminated abnormally with the state: {'grad': array([ 0.20470272]), 'task': b'ABNORMAL_TERMINATION_IN_LNSRCH', 'funcalls': 59, 'nit': 7, 'warnflag': 2}
" state: %s" % convergence_dict)

I'm using scikit-learn 0.19.2

change goal to MAXIMIZE in advisor_client/examples/scikitlearn_mnist/config.json.

If the study name of the client is existed on the server, the client will output following error.

Traceback (most recent call last):
File "example.py", line 28, in <module>
  study = client.create_study(name, study_configuration)
File "/Users/stvreumi/dev/crypto/lib/python3.6/site-packages/advisor_client/client.py", line 36, in create_study
  study = Study.from_dict(response.json()["data"])
KeyError: 'data'

Because of the response only have following key and value
{'error': 'The study Study exists'}

Maybe client can show clear message to user that using same study name causes error.

I follow this link to start server from source and then get below error message:

Traceback (most recent call last):
  File "./manage.py", line 21, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.6/dist-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.6/dist-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.6/dist-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.6/dist-packages/django/core/management/base.py", line 361, in execute
    self.check()
  File "/usr/local/lib/python3.6/dist-packages/django/core/management/base.py", line 390, in check
    include_deployment_checks=include_deployment_checks,
  File "/usr/local/lib/python3.6/dist-packages/django/core/management/commands/migrate.py", line 65, in _run_checks
    issues.extend(super()._run_checks(**kwargs))
  File "/usr/local/lib/python3.6/dist-packages/django/core/management/base.py", line 377, in _run_checks
    return checks.run_checks(**kwargs)
  File "/usr/local/lib/python3.6/dist-packages/django/core/checks/registry.py", line 72, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/usr/local/lib/python3.6/dist-packages/django/core/checks/urls.py", line 13, in check_url_config
    return check_resolver(resolver)
  File "/usr/local/lib/python3.6/dist-packages/django/core/checks/urls.py", line 23, in check_resolver
    return check_method()
  File "/usr/local/lib/python3.6/dist-packages/django/urls/resolvers.py", line 398, in check
    for pattern in self.url_patterns:
  File "/usr/local/lib/python3.6/dist-packages/django/utils/functional.py", line 80, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.6/dist-packages/django/urls/resolvers.py", line 579, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/usr/local/lib/python3.6/dist-packages/django/utils/functional.py", line 80, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.6/dist-packages/django/urls/resolvers.py", line 572, in urlconf_module
    return import_module(self.urlconf_name)
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/root/advisor/advisor/advisor_server/advisor/urls.py", line 27, in <module>
    url(r'^login/$', auth_views.login, name='login'),
AttributeError: module 'django.contrib.auth.views' has no attribute 'login'

My environment is docker container tensorflow/tensorflow:1.14.0-gpu-py3-jupyter
BTW, I only find manage.py in the folder advisor_server
So I ran the command ./manage.py migrate in this folder, is it correct?

Hi, there.

Is anything wrong with the pip installation?
I tried different pip mirror source even with or without conda, python2 and python3
But I got similar error and it will install a package named unknown.
Anyone same as me?

(master)⚡ [130] % pip install advisor --user
Collecting advisor
  Downloading https://files.pythonhosted.org/packages/c1/fd/5cf49e6fbf34e3879ac12561e258827a5aa422469fca6faf3a17aaf3e7a9/advisor-0.1.6.tar.gz
  Running setup.py (path:/tmp/pip-install-v6LivQ/advisor/setup.py) egg_info for package advisor produced metadata for project name unknown. Fix your #egg=advisor fragments.
Building wheels for collected packages: unknown, unknown
  Running setup.py bdist_wheel for unknown ... done
  Stored in directory: /home/myname/.cache/pip/wheels/97/10/41/fd24dfce3d66fe01c1a4ee2fca1e33c9dc5e68eef0d6177d88
  Running setup.py bdist_wheel for unknown ... error
  Complete output from command /home/tools/anaconda2/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-v6LivQ/unknown/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-JAGH_G --python-tag cp27:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-install-v6LivQ/unknown/setup.py'
  
  ----------------------------------------
  Failed building wheel for unknown
  Running setup.py clean for unknown
  Complete output from command /home/tools/anaconda2/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-v6LivQ/unknown/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" clean --all:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-install-v6LivQ/unknown/setup.py'
  
  ----------------------------------------
  Failed cleaning build dir for unknown
Successfully built unknown
Failed to build unknown
Installing collected packages: unknown
Successfully installed unknown-0.0.0
You are using pip version 10.0.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

from: https://github.com/tobegit3hub/advisor/blob/master/advisor_server/suggestion/algorithm/bayesian_optimization.py#L149

    acquisition_fucntion_values = mean + acquisition_function_kappa * std

    if study_goal == "MAXIMIZE":
      x_max = x_tries[acquisition_fucntion_values.argmax()]
    elif study_goal == "MINIMIZE":
      x_max = x_tries[acquisition_fucntion_values.argmin()]

When the goal is MINIMIZE, should the acquisition_fucntion_values be mean - acquisition_function_kappa * std ?

I was curious advisor runs multiple trials in parallel or is it all sequential? I saw there was a kubernetes configuration so was wondering if it leverages scaling there for parallel optimization.

hello,
we setup adviser server in linux,
but how we can manage related security, i.e anyone can delete the list and trials,
and there is not any instruction that how we can create users and what is the default admin credentials.
please let us know how we can go ahead for users and security.
thanks

When i run advisor_admin server start, it has a SyntaxError an invalid syntax in the site-packages/advisor-0.1.6-py3.6.egg/advisor_client/commandline/admin_command.py

def is_server_running():

command = "docker ps |grep 'tobegit3hub/advisor'"
print("Run the command: {}".format(command))

try:
command_output = subprocess.check_output(command, shell=True)

if command_output != "":
  return True

except subprocess.CalledProcessError, e:
if e.output == "":
pass
else:
print("Get error: {}".format(e.output))
return False

The keras example mnist_cnn.py is not working, this is what happens:

renan@renan-cunha-laptop:~/Downloads/advisor/examples/keras$ python3 mnist_cnn.py
Using TensorFlow backend.
Traceback (most recent call last):
File "mnist_cnn.py", line 134, in
keras_util.main(train)
File "/home/renan/Downloads/advisor/examples/keras/keras_util.py", line 74, in main
"BayesianOptimization")
File "/home/renan/.local/lib/python3.6/site-packages/advisor_client/client.py", line 36, in create_study
study = Study.from_dict(response.json()["data"])
KeyError: 'data'

I follow your instruction to open http://127.0.0.1:8000 in the browser and sign in with my github. But it shows 404 error the page not found.
When I use the command "pip install advisor_clients", it shows "Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 503 Forwarding failure',))': /simple/advisor-clients/"

Thank you!!

The words "Bayesian" and "scaling" are misspelled in the readme and possibly also in the code.

Ubuntu 16.04.4 LTS
python 3.5.2
advisor 0.1.5.1

I installed advisor with pip3, but can't start any server.
There is no docker on my OS, and I think docker is not necessary according the doc.

% advisor_admin server start 
Are you sure to run server container(Y/N): Y
Try to start the server with container
Run the command: docker run -d -p 8000:8000 tobegit3hub/advisor
/bin/sh: 1: docker: not found
Fail to run server, exit code: 127

% advisor_admin server start
Are you sure to run server container(Y/N): N
Cancel operation

Hi, when using Advisor to get suggestion, i get wrong trial that the parameter was out of the range i set.
The study conf shows as below

{ maxTrails: 20, randomInitTrail: 5, params: [ { scallingType: "LINEAR", type: "INTEGER", maxValue: 20, minValue: 5, parameterName: "max_depth", feasiblePoints: "" }, { scallingType: "LINEAR", type: "DOUBLE", maxValue: 0.2, minValue: 0, parameterName: "eta", feasiblePoints: "" } ], goal: "MAXIMIZE" }

and i get a trial like this

{"eta": 19.999988873780502, "max_depth": 0}

then i found line 153 of bayesian_optimization.py
bound_dict = {}
it's not in order as you insert in python 2.7 while iterating, so it may get wrong order when construct bounds array at line 182-184

for key in bound_dict.keys():
        bounds.append(bound_dict[key])
      bounds = np.asarray(bounds)

I think it's better to use
bound_dict = OrderedDict()
to keep the order in bound_dict to get the right suggestion.