๐ Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
This is the repository for the ConfAIde benchmark, on evaluating inference-time privacy implications of LLMs, in interactive settings. The benchmark has 4 tiers, and you can find the dataset/scenarios under the ./benchmark
directory.
First, create the conda environment by running:
conda env create -f environment.yml
and then activate it:
conda activate confaide
We have also stored all the results and model responses in the ./eval_results
folder.
./annotations: Results from the human evaluation study
./benchmark: Data files for all the 4 tiers of the dataset
./eval_results: Model outputs and statistics, when evaluated on confAIde