This is the official repository of our paper Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations, EMNLP 2023.

TL;DR: This work presents Self-ICL, a prompting framework bootstrapping LLMsʼ intrinsic task understanding ability to perform in-context learning via self-generated pseudo-demonstrations.

Given the [task_description] and a corresponding [test_input], Self-ICL consists of three steps:

1. Construction of pseudo-inputs.

   • Prompt template (the same prompt is use in both direct prompting and chain-of-thought prompting).
   • Prompt the LLM to generate [num_shot] pseudo-inputs, conditioned on the [task_description] and [test_input].

2. Construction of pseudo-labels.

   • Prompt template for direct prompting.
   • Prompt template for chain-of-thought prompting.
   • Collect the [num_shot] pseudo-inputs generated in step 1 and predict their pseudo-labels by zero-shot prompting the LLM.

3. In-context learning with pseudo-demonstrations.

   • Prompt template for direct prompting.
   • Prompt template for chain-of-thought prompting.
   • Collect the pseudo-labels generated in step 2, and construct [num_shot] pseudo-demonstrations (i.e., pseudo-input-label pairs).
   • Concatenate [task_description], pseudo-demonstrations, and [test_input] and prompt the LLM to perform ICL.

One may follow the steps below to reproduce the experiments. As an example, the following steps reproduce the text-bison-001 column in Table 3.

Set the corresponding environment variables to your API keys. For example, GOOGLE_API_KEY for PaLM-2, and OPENAI_API_KEY for GPT models.

export GOOGLE_API_KEY=<your_api_key>

Set the configuration file in ./configs to your experiment settings. For example, the ./configs/config_template_standard.yml file records the settings for running the ZS-Direct prompting method using the text-bison-001 API endpoint.

Run the following script to run different prompting methods and log output to log_path in the YAML file:

python experiment.py \
  --config_path ./configs/config_template_standard.yml \
  --label_type "class"

Note that one may need to configure other command line arguments in other experiments (please refer to experiment.py for more details).

After finishing the previous step, you may evaluate the performance by running:

python experiment.py \
  --config_path ./configs/config_template_standard.yml \
  --label_type "class" \
  --eval

The evaluation results would be stored in the log_path along with model outputs from the previous step.