Hyperparameter Optimization of Machine Learning Algorithms

This code provides a hyper-parameter optimization implementation for machine learning algorithms, as described in the paper:
L. Yang and A. Shami, “On hyperparameter optimization of machine learning algorithms: Theory and practice,” Neurocomputing, vol. 415, pp. 295–316, 2020, doi: https://doi.org/10.1016/j.neucom.2020.07.061.

To fit a machine learning model into different problems, its hyper-parameters must be tuned. Selecting the best hyper-parameter configuration for machine learning models has a direct impact on the model's performance. In this paper, optimizing the hyper-parameters of common machine learning models is studied. We introduce several state-of-the-art optimization techniques and discuss how to apply them to machine learning algorithms. Many available libraries and frameworks developed for hyper-parameter optimization problems are provided, and some open challenges of hyper-parameter optimization research are also discussed in this paper. Moreover, experiments are conducted on benchmark datasets to compare the performance of different optimization methods and provide practical examples of hyper-parameter optimization.

This paper and code will help industrial users, data analysts, and researchers to better develop machine learning models by identifying the proper hyper-parameter configurations effectively.

PS: A comprehensive Automated Machine Learning (AutoML) tutorial code can be found in: AutoML-Implementation-for-Static-and-Dynamic-Data-Analytics
- Including automated data pre-processing, automated feature engineering, automated model selection, hyperparameter optimization, and automated model updating (concept drift adaptation).

Paper

On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
One-column version: arXiv
Two-column version: Elsevier

Quick Navigation

Section 3: Important hyper-parameters of common machine learning algorithms
Section 4: Hyper-parameter optimization techniques introduction
Section 5: How to choose optimization techniques for different machine learning models
Section 6: Common Python libraries/tools for hyper-parameter optimization
Section 7: Experimental results (sample code in "HPO_Regression.ipynb" and "HPO_Classification.ipynb")
Section 8: Open challenges and future research directions
Summary table for Sections 3-6: Table 2: A comprehensive overview of common ML models, their hyper-parameters, suitable optimization techniques, and available Python libraries
Summary table for Sections 8: Table 10: The open challenges and future directions of HPO research

Implementation

Sample code for hyper-parameter optimization implementation for machine learning algorithms is provided in this repository.

Sample code for Regression problems

HPO_Regression.ipynb
Dataset used: Boston-Housing

Sample code for Classification problems

HPO_Classification.ipynb
Dataset used: MNIST

Machine Learning & Deep Learning Algorithms

Random forest (RF)
Support vector machine (SVM)
K-nearest neighbor (KNN)
Artificial Neural Networks (ANN)

Hyperparameter Configuration Space

ML Model	Hyper-parameter	Type	Search Space
RF Classifier	n_estimators	Discrete	[10,100]
	max_depth	Discrete	[5,50]
	min_samples_split	Discrete	[2,11]
	min_samples_leaf	Discrete	[1,11]
	criterion	Categorical	'gini', 'entropy'
	max_features	Discrete	[1,64]
SVM Classifier	C	Continuous	[0.1,50]
	kernel	Categorical	'linear', 'poly', 'rbf', 'sigmoid'
KNN Classifier	n_neighbors	Discrete	[1,20]
ANN Classifier	optimizer	Categorical	'adam', 'rmsprop', 'sgd'
	activation	Categorical	'relu', 'tanh'
	batch_size	Discrete	[16,64]
	neurons	Discrete	[10,100]
	epochs	Discrete	[20,50]
	patience	Discrete	[3,20]
RF Regressor	n_estimators	Discrete	[10,100]
	max_depth	Discrete	[5,50]
	min_samples_split	Discrete	[2,11]
	min_samples_leaf	Discrete	[1,11]
	criterion	Categorical	'mse', 'mae'
	max_features	Discrete	[1,13]
SVM Regressor	C	Continuous	[0.1,50]
	kernel	Categorical	'linear', 'poly', 'rbf', 'sigmoid'
	epsilon	Continuous	[0.001,1]
KNN Regressor	n_neighbors	Discrete	[1,20]
ANN Regressor	optimizer	Categorical	'adam', 'rmsprop'
	activation	Categorical	'relu', 'tanh'
	loss	Categorical	'mse', 'mae'
	batch_size	Discrete	[16,64]
	neurons	Discrete	[10,100]
	epochs	Discrete	[20,50]
	patience	Discrete	[3,20]

HPO Algorithms

Grid search
Random search
Hyperband
Bayesian Optimization with Gaussian Processes (BO-GP)
Bayesian Optimization with Tree-structured Parzen Estimator (BO-TPE)
Particle swarm optimization (PSO)
Genetic algorithm (GA)

Requirements

Contact-Info

Please feel free to contact me for any questions or cooperation opportunities. I'd be happy to help.

Email: [email protected]
GitHub: LiYangHart and Western OC2 Lab
LinkedIn: Li Yang
Google Scholar: Li Yang and OC2 Lab

Citation

If you find this repository useful in your research, please cite this article as:

L. Yang and A. Shami, “On hyperparameter optimization of machine learning algorithms: Theory and practice,” Neurocomputing, vol. 415, pp. 295–316, 2020, doi: https://doi.org/10.1016/j.neucom.2020.07.061.

@article{YANG2020295,
title = "On hyperparameter optimization of machine learning algorithms: Theory and practice",
author = "Li Yang and Abdallah Shami",
volume = "415",
pages = "295 - 316",
journal = "Neurocomputing",
year = "2020",
issn = "0925-2312",
doi = "https://doi.org/10.1016/j.neucom.2020.07.061",
url = "http://www.sciencedirect.com/science/article/pii/S0925231220311693"
}

LightGBMError: Cannot change bin_construct_sample_cnt after constructed Dataset handle.

Environment info

Operating System: windows 10
Python version: python 3.8
LightGBM version: 3.3.2

code

This error occurs when I execute this code. seek help
The code is in this file HPO_Classification.ipynb

params = {'class_weight': 'balanced',
'boosting_type': 'dart',
'num_leaves': 44,
'learning_rate': 0.05979047181117413,
'subsample_for_bin': 220000,
'min_child_samples': 390,
'reg_alpha': 0.3877551020408163,
'reg_lambda': 0.5102040816326531,
'colsample_bytree': 0.7333333333333333,
'subsample': 0.8535353535353536}

def random_objective(params, iteration, n_folds = N_FOLDS):
   """Random search objective function. Takes in hyperparameters
      and returns a list of results to be saved."""

   start = timer()
   
   # Perform n_folds cross validation
   cv_results = lgb.cv(params, train_set, num_boost_round = 10000, nfold = n_folds, 
                       early_stopping_rounds = 100, metrics = 'auc', seed = 50)
   end = timer()
   best_score = np.max(cv_results['auc-mean'])
   
   # Loss must be minimized
   loss = 1 - best_score
   
   # Boosting rounds that returned the highest cv score
   n_estimators = int(np.argmax(cv_results['auc-mean']) + 1)
   
   # Return list of results
   return [loss, params, iteration, n_estimators, end - start]

%%capture

random.seed(50)

for i in range(MAX_EVALS):
   
   # Randomly sample parameters for gbm
   params = {key: random.sample(value, 1)[0] for key, value in param_grid.items()}
   
   print(params)
   
   if params['boosting_type'] == 'goss':
       # Cannot subsample with goss
       params['subsample'] = 1.0
   else:
       # Subsample supported for gdbt and dart
       params['subsample'] = random.sample(subsample_dist, 1)[0]
       
       
   results_list = random_objective(params, i)
   
   # Add results to next row in dataframe
   random_results.loc[i, :] = results_list

Error message

---------------------------------------------------------------------------
LightGBMError                             Traceback (most recent call last)
<ipython-input-20-9bb1341efb38> in <module>
    17 
    18 
---> 19     results_list = random_objective(params, i)
    20 
    21     # Add results to next row in dataframe

<ipython-input-19-060262a0c5a6> in random_objective(params, iteration, n_folds)
     6 
     7     # Perform n_folds cross validation
----> 8     cv_results = lgb.cv(params, train_set, num_boost_round = 10000, nfold = n_folds, 
     9                         early_stopping_rounds = 100, metrics = 'auc', seed = 50)
    10     end = timer()

F:\anaconda\lib\site-packages\lightgbm\engine.py in cv(params, train_set, num_boost_round, folds, nfold, stratified, shuffle, metrics, fobj, feval, init_model, feature_name, categorical_feature, early_stopping_rounds, fpreproc, verbose_eval, show_stdv, seed, callbacks, eval_train_metric, return_cvbooster)
   597         params['metric'] = metrics
   598 
--> 599     train_set._update_params(params) \
   600              ._set_predictor(predictor) \
   601              .set_feature_name(feature_name) \

F:\anaconda\lib\site-packages\lightgbm\basic.py in _update_params(self, params)
  1932                     self._free_handle()
  1933                 else:
-> 1934                     raise LightGBMError(_LIB.LGBM_GetLastError().decode('utf-8'))
  1935         return self
  1936 

LightGBMError: Cannot change bin_construct_sample_cnt after constructed Dataset handle.

liyanghart / hyperparameter-optimization-of-machine-learning-algorithms Goto Github PK