Metric: Logloss (lower is better). In the first brackets - std, in the second - the percentage difference from the tuned CatBoost.

You can find detailed information about experimental setup in comparison description

• Docker has everything necessary for running experiments with Python.
• Link: https://hub.docker.com/r/yandex/catboost_benchmarks/
• To launch:

docker run --workdir /root -v <path_to_local_folder>:/root/shared -p 80:8888 -it yandex/catboost_benchmarks sh -c "ipython notebook --ip=* --no-browser --allow-root"

• experiment.py:

  • The parent class that implements reading data, splitting data into subsets, calculating counter values, the cross-validation function, and the run function that starts the experiment.

• *_experiment.py:

  • Child classes that implement functions for converting data to the format of a specific algorithm, setting parameters to tune and the distributions they are selected from, as well as functions that start training.

• cat_counter.py:

  • Class for calculating counter values.

• run.py, run_default.py:

  • Files for starting experiments.

• h2o_preprocessing.py:

  • Class for h2o data preprocessing.

• run_grid.R:

  • File for starting h2o experiments in R (classification).

• run_grid_reg.R:

  • File for starting h2o experiments in R (regression).

• run_grid_enc.R:

  • File for starting h2o experiments in R (classification) using native h2o categorical_encoding options.

• run_grid_reg_enc.R:

  • File for starting h2o experiments in R (regression) using native h2o categorical_encoding options.

• run_install.R:

  • File for installing necessary R libraries.

• install/:

  • The folder with the Docker file and scripts for installing the necessary libraries.

• notebooks/:

  • The folder containing notebooks with examples.

• comparison_description.*:

  • PDF and TEX files describing the format of the comparison.

• prepare_*/

  • Scripts and notebooks for catboost train and test files preparation

• You can run it either from the command line or from the interpreter, importing the class you need.

• Parameters:

  Positional arguments:
    bst                            Algorithm name {cab, xgb, lgb}
    learning_task                  Type of task {classification, regression}
  
  Required arguments:
    --train                        Path to the train-file (str)
    --test                         Path to the test-file (str)
    --cd                           Path to the cd-file (str)
  
  Optional arguments:
    -t [ --n_estimators ]          Number of trees (int; by default 5000)
    -n [ --hyperopt_evals ]        Number of hyperopt iterations (int; by default 50)
    -o [ --output_folder_path ]    Path to the results folder (str; by default None
                                      and the result isn't stored)
    --holdout                      Size of the holdout section (float; by default 0)
    -h [ --help ]                  Help
  

• Usage:

  python run.py bst learning_task --train TRAIN_FILE_PATH --test TEST_FILE_PATH --cd CD_FILE_PATH 
        [-t N_ESTIMATORS] [-n HYPEROPT_EVALS] [-o OUTPUT_FOLDER_PATH] [--holdout HOLDOUT] [-h]
  

• Launch examples

  • From the command line:

    python run.py cab classification -n 2 -t 10 -i --train amazon/train.txt
        --test amazon/test.txt --cd amazon/cd
    

  • From the interpreter:

    from catboost_experiment import CABExperiment
    cab_exp = CABExperiment('classification', n_estimators=10, max_hyperopt_evals=2,
        train_path='amazon/train.txt', test_path='amazon/test.txt', cd_path='amazon/cd')
    cab_exp.run()
    

• You can run it from the command line from the directory with train and test files. The results will be saved in the working directory. You can also run R and run the commands from here.

• H2O works smoother with preprocessed input (all categorical features are transformed to numerical), see the preprocessing script below. But it also works with raw data using its own categorical encoding methods (see run_grid_enc.R and run_grid_reg_enc.R).

• Files:

  Input:
    parsed_train                           Preprocessed train
    parsed_test                            Preprocessed test
  
  Output:
    result_default.tsv                     Results with default parameters
    result_tuned.tsv                       Results with tuned parameters
    result_default_seeds.tsv               Results with default parameters using a range of seeds
    result_tuned_seeds.tsv                 Results with tuned parameters using a range of seeds
  

• Usage:

  Rscript ./run_install.R
  Rscript ./run_grid.R
  

• Preprocessing: The script creates two files in the -o directory using three input files.

python h2o_preprocessing.py <classification|regression> --train <path to train> --test <path to test> --cd <path to column desc> -o <output directory>

• Launch examples:

  > python h2o_preprocessing.py classification --train ./train_full3 --test ./test3 --cd ./train_full3.cd -o .
  > Rscript ../run_grid.R
  ...
  R is connected to the H2O cluster:
  H2O cluster version:        3.10.4.6
  H2O Connection ip:          localhost
  H2O Connection port:        23776
  R Version:                  R version 3.3.3 (2017-03-06)
  ...
  After starting H2O, you can use the Web UI at http://localhost:23776