By Parsa Hejabi, Elnaz Rahmati, Alireza Salkhordeh Ziabari, Preni Golazizian, Jesse Thomason, Morteza Dehghani - University of Southern California

Large Language Models (LLMs) have shown impressive capabilities in complex tasks and interactive environments, yet their creativity remains underexplored. This paper introduces a simulation framework utilizing the game Balderdash to evaluate both the creativity and logical reasoning of LLMs. In Balderdash, players generate fictitious definitions for obscure terms to deceive others while identifying correct definitions. Our framework enables multiple LLM agents to participate in this game, assessing their ability to produce plausible definitions and strategize based on game rules and history. We implemented a centralized game engine featuring various LLMs as participants and a judge LLM to evaluate semantic equivalence. Through a series of experiments, we analyzed the performance of different LLMs, examining metrics such as True Definition Ratio, Deception Ratio, and Correct Guess Ratio. The results provide insights into the creative and deceptive capabilities of LLMs, highlighting their strengths and areas for improvement. Specifically, the study reveals that infrequent vocabulary in LLMs' input leads to poor reasoning on game rules and historical context.

• python 3.10+
• Ubuntu GPU-enabled server with CUDA 12.1+
  • Check your GPUs with nvidia-smi
• python environment with packages installed as in requirements.txt
• A local MongoDB server or Atlas cluster, and a connection string
• An OpenAI account, with an API key and a project ID
• Hugging Face account, with an access token for the gated models

# Clone the repository
git clone [email protected]:ParsaHejabi/Simulation-Framework-for-Multi-Agent-Balderdash.git
# Change directory to the root of the repository
cd ROOT_OF_THE_REPO
# Create an empty directory for data
mkdir data
# Create a .env file and fill in the required fields
echo -e "OPENAI_API_KEY=\nOPENAI_PROJECT_ID=\nMONGODB_CONNECTION_STRING=\nMONGODB_USERNAME=\nMONGODB_PASSWORD=" > .env
# Create a virtual environment
python3 -m venv venv
# Activate the virtual environment
source venv/bin/activate
# Install the required packages
pip install -r requirements.txt

Please contact the corresponding author for access to the datasets used in this study.

• run_first_experiment.py: This script runs the leaderboard experiment on two datasets: "Basic Frequent English Words" and "All Balderdash," five different random seeds, four different players (LLMs) in each game, and three different settings based on history type (HT) with the ruling score of 1, 2, and 3 for receiving votes, correct votes, and correct definitions, respectively.
• run_second_experiment.py: This script runs the convergence experiment on the known words dataset for each LLM in all and mini history types with three players of the same LLM in each game, and the ruling score of 1, 2, and 3 for receiving votes, correct votes, and correct definitions, respectively.
• run_third_experiment.py: This script runs the game rules experiment on the known words dataset for each LLM in none history type with one player (LLM) in each game, and the ruling score of 1, 2, 0 in one setting and 1, 2, 50 in the other setting for receiving votes, correct votes, and correct definitions, respectively.

Here are the variables that you can set along with their descriptions:

All functions needed for retrieving game data from the database, analyzing, and visualizing model performance are provided in evaluation/eval_tools.py. Additionally, a Python script (evaluation/evaluation.py) is available to help run evaluations on LLMs using the previously mentioned tools. The type of evaluation can be determined using input arguments. Below are the variables you can set along with their descriptions:

All prompts are stored in the prompts directory. You can change the prompts by editing the files in this directory. The prompts are being rendered using jinja2 templates, so you can use variables in the prompts.

Table 1: Leaderboard experiment results on "Basic Frequent English Words," evaluating each LLM in three different settings based on history type (HT) using the average of LKR, TDR, DR, CGR, and AS metrics over all rounds and games. The highest value of each metric for different game settings is in bold. However, based on the standard deviation, this does not represent absolute superiority.

Table 2: Leaderboard experiment results on "All Balderdash," evaluating each LLM in three different settings based on history type (HT) using the average of LKR, TDR, DR, CGR, and AS metrics over all rounds and games. The highest value of each metric for different game settings is in bold. However, based on the standard deviation, this does not represent absolute superiority.