This repository contain the code and supplementary documents used in the paper
An evaluation framework for personalization strategy experiment design
which is due to appear in AdKDD 2020 Workshop, in conjunction with SIGKDD'20 (San Diego, CA, now online due to the COVID-19 pandemic). Please cite the pre-print paper as (BibTeX):
@misc{liu2020evaluation,
title={An Evaluation Framework for Personalization Strategy Experiment Designs},
author={C. H. Bryan Liu and Emma J. McCoy},
year={2020},
eprint={2007.11638},
archivePrefix={arXiv},
primaryClass={stat.ME}
}
Quick navigation:
This file assumes you have access to a *nix-like machine (both MacOS or Linux would do).
The projects uses pyenv
and poetry
for package management.
Before you start, please ensure you have gcc
, make
, and pip
installed.
For Linux (together with other required libraries):
sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \
libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
xz-utils tk-dev libffi-dev liblzma-dev python3-openssl git
wget -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
chmod u+x pyenv-installer
./pyenv-installer
For OS X:
brew install pyenv
We then need to configure the PATHs:
export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
...and install the right Python version for our environment:
pyenv install 3.11.3
See https://python-poetry.org/docs/#installation for the installation instructions.
git clone https://github.com/liuchbryan/experiment_design_evaluation.git
cd experiment_design_evaluation
# Switch to Python 3.11.3 for pyenv
pyenv local 3.11.3
poetry env use ~/.pyenv/versions/3.11.3/bin/python
poetry install
To run the evaluations described in Section 4 of the paper, replace <script_name>
with the name of the
script found in the script
directory (e.g. run_eval_normal_intersectiononly_AE
), and run either
poetry run python -m scripts.<script_name>
# For example (Setup 1, actual effect)
# poetry run python -m scripts.run_eval_normal_intersectiononly_AE
or
poetry shell
python -m scripts.<script_name>
# For example (Setup 1, actual effect)
# python -m scripts.run_eval_normal_intersectiononly_AE
The outputs are stored in the output
directory (where we have pre-populated with the pickle files
containing evaluations we've run).
They can be analyzed using the Jupyter notebook 04_evaluate_theoretical_values.ipynb
, which can be opened
using the command
poetry run jupyter notebook
supplementary_document.pdf
contains the detailed mathematical derivations not featured in the paper for brevity. It can also be viewed on https://arxiv.org/pdf/2007.11638.pdf after the main text.98_exploration_binary_response.ipynb
and99_exploration_misc.ipynb
contain experimental/exploration code as part of the ongoing research.