wepe / tgboost Goto Github PK

Tiny Gradient Boosting Tree

License: MIT License

Python 6.09% Java 93.91%

boosted-trees gradient-boosting-machine machine-learning xgboost sliq

tgboost's Introduction

What is TGBoost

It is a Tiny implement of Gradient Boosting tree, based on XGBoost's scoring function and SLIQ's efficient tree building algorithm. TGBoost build the tree in a level-wise way as in SLIQ (by constructing Attribute list and Class list). Currently, TGBoost support parallel learning on single machine, the speed and memory consumption are comparable to XGBoost.

TGBoost supports most features as other library:

Built-in loss , Square error loss for regression task, Logistic loss for classification task
Early stopping , evaluate on validation set and conduct early stopping
Feature importance , output the feature importance after training
Regularization , lambda, gamma
Randomness, subsample，colsample
Weighted loss function , assign weight to each sample

Another two features are novel:

Handle missing value, XGBoost learn a direction for those with missing value, the direction is left or right. TGBoost take a different approach: it enumerate missing value go to left child, right child and missing value child, then choose the best one. So TGBoost use Ternary Tree.
Handle categorical feature, TGBoost order the categorical feature by their statistic (Gradient_sum / Hessian_sum) on each tree node, then conduct split finding as numeric feature.

Installation

The current version is implemented in pure Java, to use TGBoost you should first install JDK. For Python user, Python binding is also provided:

git clone [email protected]:wepe/tgboost.git
cd python-package
sudo python setup.py install

To Understand TGBoost

For those want to understand how TGBoost work, and dive into Gradient Boosting Machine, please refer to the Python implementation of TGBoost: tgboost-python, the python source code is relatively easy to follow.

Example

Here is an example, download the data here

import tgboost as tgb

# training phase
ftrain = "data/train.csv"
fval = "data/val.csv"
params = {'categorical_features': ["PRI_jet_num"],
          'early_stopping_rounds': 10,
          'maximize': True,
          'eval_metric': 'auc',
          'loss': 'logloss',
          'eta': 0.3,
          'num_boost_round': 20,
          'max_depth': 7,
          'scale_pos_weight':1.,
          'subsample': 0.8,
          'colsample': 0.8,
          'min_child_weight': 1.,
          'min_sample_split': 5,
          'reg_lambda': 1.,
          'gamma': 0.,
          'num_thread': -1
          }

model = tgb.train(ftrain, fval, params)

# testing phase
ftest = "data/test.csv"
foutput = "data/test_preds.csv"
model.predict(ftest, foutput)

# save the model
model.save('./tgb.model')

# load model and predict
model = tgb.load_model('./tgb.model')
model.predict(ftest, foutput)

Reference

tgboost's People

Contributors

Stargazers

Watchers

Forkers

gubobo lxj0276 mathlf2015 nackel eroszip awesome-archive chenyqk chdd chagge hczheng benjamesbabala gucasbrg wenmin-wu qgzang ichdream willsonn huxiaoman7 topspin26 luolaihu fangzheng354 wai7niu8 hadoop73 limitspro zhanglbjames hongwangli hanlos victor-gun 358463121 melody-xiaomi zq2501330612 jiongxiao kmustyxl wangyipengpeter wuzijie i-math liuxiaozeeee jintw chinesehuang wangdaiyin joneswan ghdeng1992 andysyan cyzlovedream mowayao implus lijiannuist pavantyagi dongfengchi zldeng leonall suiguoxin buptygz zerocurve wilsonsky18 wtx626 guanlongtianzi lihengtianxia 0x0all qiuqiang1985 zhabzhang trcoder bullio scarpdiem srepho lmsreborn iamlira eryueniaobp rosefun zhengkaifu liujxing chinaluoyanbin zhzou2020 zouzias lvjianwei123 batermj webber26232 jeremytian2019 xuke100 hwaking jason08 kamiz22 dwkan niyingqiu sky700 zorrock rumourcy priveyes andrewliujian askintution mldlx wutonghua zhouyaou quanjunzhang igithubj jsusu t-allen-sudo gaotian123 cjltctc eperezbal

tgboost's Issues

关于离散特征处理的问题

现在的代码逻辑，离散特征也会被装箱(bin) 然后就是按照连续特征来处理了。这样的处理，默认离散特征就是有序的了，所以m个特征值有m-1个箱子。但是如果离散特征是无序的，那么箱子数量应该是指数级的。这个地方是不是可以在优化下那？还是说xgboost就是按照无序来处理的？

参考Spark中决策树对于离散特征的处理会先判断离散特征是有序还是无序的，然后在进行装箱。

才疏学浅，很想把大佬的代码吃透了。。。

    private void initialize_cutting_inds_thresholds(){
        cutting_inds = new int[feature_dim][][];
        cutting_thresholds = new float[feature_dim][];

        for(int i=0;i<feature_dim;i++){
            //for this feature, get its cutting index
            ArrayList<Integer> list = new ArrayList<>();
            int last_index = 0;
            for(int j=0;j<attribute_list[i].length;j++){
                if(attribute_list[i][j][0]==attribute_list[i][last_index][0]){
                    last_index = j;
                }else {
                    list.add(last_index);
                    last_index = j;
                }
            }
            //for this feature,store its cutting threshold
            cutting_thresholds[i] = new float[list.size()+1];
            for(int t=0;t<cutting_thresholds[i].length-1;t++){
                cutting_thresholds[i][t] = attribute_list[i][list.get(t)][0];
            }
            cutting_thresholds[i][list.size()] = attribute_list[i][list.get(list.size()-1)+1][0];

            //for this feature,store inds of each interval
            cutting_inds[i] = new int[list.size()+1][]; //list.size()+1 interval

            list.add(0,-1);
            list.add(attribute_list[i].length-1);
            for(int k=0;k<cutting_inds[i].length;k++){
                int start_ind = list.get(k)+1;
                int end_ind = list.get(k+1);
                cutting_inds[i][k] = new int[end_ind-start_ind+1];
                for(int m=0;m<cutting_inds[i][k].length;m++){
                    cutting_inds[i][k][m] = (int) attribute_list[i][start_ind+m][1];
                }
            }

        }
    }

Edited code is:

      private void initialize_cutting_idx_thresholds() {
		cutting_idx = new int[feature_dim][][];
		cutting_thresholds = new double[feature_dim][];
		
		for (int i = 0; i < feature_dim; ++i) {
			List<Integer> list = new ArrayList<>();
			int last_index = -1;
			for (int j = 0; j < attribute_list[i].length; ++j) {
				if (last_index == -1 || attribute_list[i][j][0] == attribute_list[i][last_index][0]) {
					last_index = j;
				}
				else {
					list.add(last_index);
					last_index = j;
				}
			}
			
			cutting_thresholds[i] = new double[list.size()];
			for (int t = 0; t < cutting_thresholds[i].length; ++t) {
				cutting_thresholds[i][t] = attribute_list[i][list.get(t)][0];
			}
			
			cutting_idx[i] = new int[list.size()][];
			list.add(attribute_list[i].length);
			
			for (int k = 0; k < cutting_idx[i].length; ++k) {
				int s_idx = list.get(k);
				int e_idx = list.get(k + 1);
				cutting_idx[i][k] = new int[e_idx - s_idx];
				for (int m = 0; m < cutting_idx[i][k].length; ++m) {
					cutting_idx[i][k][m] = (int)attribute_list[i][s_idx + m][1];
				}
			}
		}
      }

是我哪里算错了吗，期待回答

能解释下ClassList和AttributeList命名的意思吗

这两个命名不是很理解，想不通为什么这样叫。能解释下具体的含义吗？非常感谢~ 最近在研究xgboost想把你的开源代码吃透，自己也写写。就是没什么文档，看不太懂。。