xingchenwan / casmopolitan Goto Github PK

Credits to Yihang Shen at Carnegie Mellon University for discovering these minor bugs

• The Adam optimizer is not updating the gradient information of the continuous variables in the mixed problems during acquisition optimization
• Line 80 of optimizer_mixed.py: currently the continuous dimension information is not passed properly to the calling function

These issues should only affect problems involving mixed categorical and continuous dimensions. The maintainer is aware of the issues and a patch will be released soon.

According to the code in "main.py", binary variables of a problem should be assigned to categorical variables.
In "bo/optimizer.py", the binary variables are all one-hot encoded, but usually it is redundant to do so. For example, when optimizing "MaxSAT60", we can simply round the variable to {0,1} with only 60 dimensions, rather than one-hot encode it to 120 dimensions.
Is there a way to avoid one-hot encoding for binary variables in Casmopolitan?

In current implementation, categorical dimensions must be in the beginning of the vector like [cat, cat, cat, cont, cont] but this can not support [cont, cat, cont, cat] with f.categorical_dims. Could you please fix that? Correct me if I am wrong. Thanks.

Casmopolitan-main/mixed_test_func/AdvAttack/tf_models/setup_cifar.py", line 16, in
ModuleNotFoundError: No module named 'tensorflow.contrib'

from tensorflow.contrib.keras.api.keras.models import Sequential
from tensorflow.contrib.keras.api.keras.layers import Dense, Dropout, Activation, Flatten
from tensorflow.contrib.keras.api.keras.layers import Conv2D, MaxPooling2D
from tensorflow.contrib.keras.api.keras.models import load_model

I use Python 3.10 on Ubuntu 22.04. This repository has some compatibility issues with my python version and currently installed packages.
I'm writing this just in case others may have same problems as I have.

In "bo/localbo_utils.py", change line 14 to:
from typing import Callable

Files under "mixed_test_func/AdvAttack" requires Tensorflow 1.XX, which is not available from pip3 (python 3.10). Delete this file if you don't need to test the adversarial attack problem. I also deleted "xgboost_hp.py" because I don't need it either. I'm not sure if it's compatible with xgboost 1.7.6, which is latest version from pip3.

Then, run "python3 main.py -p func2C --max_iters 20" to test the code. It finished without any problem on my system.

Hi

Your published study is very interesting to me. It's really cool to use your method for vision domain and blackbox model (deep learning model). You have one section for CIFAR-10 dataset and black-box function is DNN. I'm excited to see how it work efficiently and outperforms SparseRS. Can you give me some instructions or sample code to replicate this result? It could benefit my study and make it more progress.

Thanks

Hello together,
to save computational costs I'm trying to intialize Casmopolitan with previously evaluated random search trials.
However, I am not sure if all previous trials are used correctly in the BO search because I get the following warning:
The input matches the stored training data. Did you forget to call model.train()
However, when calling self.optim.restart() I assume that the model is trained with the stored data (previous trials). So why does this message appear?

My (working) code to initialize with previous trials is as follows:

self.optim.casmopolitan._X = np.empty((0, self.optim.true_dim))
self.optim.casmopolitan._fX = np.empty((0, self.batch_size))
for idx, x in enumerate(self.previous_trials):
    x = x.reshape((1, len(x)))
    self.optim.observe(x, self.previous_metrics[idx])
self.optim.restart()
# Then run BO loop of Casmopolitan 

And I use Cosmopolitan with n_init = 1 (lowest number possible). It seems to me that Cosmopolitan will always run >= 1 additional random search trial(s) on top. Because of the warning above I am not sure if the initialization is successful and Cosmopolitan is using all my previous trials for the BO search.

Is there a better way to initialize with the previous RS trials so that Casmopolitan will not run additional RS trials on top of my initialized trials?

I tried optimizing "func2C" with the model on GPU but got an error saying there're tensors on both CPU and GPU. I tried the same optimization experiment with CPU only and it completed without a problem.

Code
I changed line 88~97 in "main.py" to this:

    if problem_type == 'mixed':
        optim = MixedOptimizer(f.config, f.lb, f.ub, f.continuous_dims, f.categorical_dims,
                               n_init=args.n_init, use_ard=args.ard, acq=args.acq,
                               kernel_type=kernel_type,
                               noise_variance=noise_variance,
                               device = 'cuda',
                               min_cuda = 1,
                               **kwargs)
    else:
        optim = Optimizer(f.config, n_init=args.n_init, use_ard=args.ard, acq=args.acq,
                          kernel_type=kernel_type,
                          noise_variance=noise_variance,
                          device = 'cuda', 
                          min_cuda = 1,
                          **kwargs)

Terminal Output

idwwwoqq808@DESKTOP-35T3LEC:~/workspace/optimizer/Casmopolitan$ py main.py -p func2C --max_iters 4 --batch_size 8 --n_init 8 --n_trials 1
{'problem': 'func2C', 'max_iters': 4, 'lamda': 1e-06, 'batch_size': 8, 'n_trials': 1, 'n_init': 8, 'save_path': 'output/', 'ard': False, 'acq': 'ei', 'random_seed_objective': 20, 'debug': False, 'no_save': False, 'seed': None, 'kernel_type': None, 'infer_noise_var': False}
----- Starting trial 1 / 1 -----
Traceback (most recent call last):
	File "/home/idwwwoqq808/workspace/optimizer/Casmopolitan/main.py", line 106, in <module>
		x_next = optim.suggest(args.batch_size)
	File "/home/idwwwoqq808/workspace/optimizer/Casmopolitan/bo/optimizer_mixed.py", line 184, in suggest
		next = self.casmopolitan._create_and_select_candidates(X, fX, length=self.casmopolitan.length_discrete,
	File "/home/idwwwoqq808/workspace/optimizer/Casmopolitan/bo/localbo_mixed.py", line 228, in _create_and_select_candidates
		x_next, acq = interleaved_search(x_center[0], _ei,
	File "/home/idwwwoqq808/workspace/optimizer/Casmopolitan/bo/localbo_utils.py", line 476, in interleaved_search
		res = _interleaved_search(x0[i, :])
	File "/home/idwwwoqq808/workspace/optimizer/Casmopolitan/bo/localbo_utils.py", line 420, in _interleaved_search
		acq_x = f(x).detach().numpy()
	File "/home/idwwwoqq808/workspace/optimizer/Casmopolitan/bo/localbo_mixed.py", line 181, in _ei
		preds = gp(X)
	File "/home/idwwwoqq808/.local/lib/python3.10/site-packages/gpytorch/models/exact_gp.py", line 283, in __call__
		if all(torch.equal(train_input, input) for train_input, input in length_safe_zip(train_inputs, inputs)):
	File "/home/idwwwoqq808/.local/lib/python3.10/site-packages/gpytorch/models/exact_gp.py", line 283, in <genexpr>
		if all(torch.equal(train_input, input) for train_input, input in length_safe_zip(train_inputs, inputs)):
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument other in method wrapper_CUDA__equal) 

Version
Python: 3.10.6
PyTorch: 2.0.1
OS: WSL-Ubuntu 22.04 (WSL2)