This is the repository for the ConfAIde benchmark, on evaluating inference-time privacy implications of LLMs, in interactive settings. The benchmark has 4 tiers, and you can find the dataset/scenarios under the ./benchmark directory.

First, create the conda environment by running:

conda env create -f environment.yml

and then activate it:

conda activate confaide

We have also stored all the results and model responses in the ./eval_results folder.

./annotations: Results from the human evaluation study
./benchmark: Data files for all the 4 tiers of the dataset
./eval_results: Model outputs and statistics, when evaluated on confAIde