🛠️ Awesome LMs with Tools

Language models (LMs) are powerful yet mostly for text-generation tasks. Tools have substantially enhanced their performance for tasks that require complex skills.

Based on our recent survey about LM-used tools, "What Are Tools Anyway? A Survey from the Language Model Perspective", we provide a structured list of literature relevant to tool-augmented LMs.

• Tool basics ($\S2$)
• Tool use paradigm ($\S3$)
• Scenarios ($\S4$)
• Advanced methods ($\S5$)
• Evaluation ($\S6$)

If you find our paper or code useful, please cite the paper:

@article{wang2022what,
  title={What Are Tools Anyway? A Survey from the Language Model Perspective},
  author={Zhiruo Wang, Zhoujun Cheng, Hao Zhu, Daniel Fried, Graham Neubig},
  journal={arXiv preprint arXiv:2403.15452},
  year={2024}
}

• Definition and discussion of animal-used tools

  Animal tool behavior: the use and manufacture of tools by animals Shumaker, Robert W., Kristina R. Walkup, and Benjamin B. Beck. 2011 [Book]

• Early discussions on LM-used tools

  ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs Qin, Yujia, et al. 2023.07 [Paper]

• A survey on augmented LMs, including tool augmentation

  Augmented Language Models: a Survey Mialon, Grégoire, et al. 2023.02 [Paper]

• Definition of agents

  Artificial intelligence a modern approach Russell, Stuart J., and Peter Norvig. 2016 [Book]

• Survey about agents that perceive and act in the environment

  The Rise and Potential of Large Language Model Based Agents: A Survey Xi, Zhiheng, et al. 2023.09 [Preprint]

• Survey about the cognitive architectures for language agents

  Cognitive Architectures for Language Agents Sumers, Theodore R., et al. 2023.09 [Paper]

• Early works that set up the commonly used tooling paradigm

  Toolformer: Language Models Can Teach Themselves to Use Tools Schick, Timo, et al. 2024 [Paper]

• Provide in-context examples for tool-using on visual programming problems

  Visual Programming: Compositional visual reasoning without training Gupta, Tanmay, and Aniruddha Kembhavi. 2023 [Paper]

• Tool learning via in-context examples on reasoning problems involving text or multi-modal inputs

  Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models Lu, Pan, et al. 2024 [Paper]

• In-context learning based tool using for reasoning problems in BigBench and MMLU

  ART: Automatic multi-step reasoning and tool-use for large language models Paranjape, Bhargavi, et al. 2023.03 [Preprint]

• Providing tool documentation for in-context tool learning

  Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models Hsieh, Cheng-Yu, et al. 2023.08 [Preprint]

• Training on human annotated examples of (NL input, tool-using solution output) pairs

  API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs Li, Minghao, et al. 2023.12 [Paper]

  Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems Kadlčík, Marek, et al. 2023 [Paper]

• Training on model-synthesized examples

  ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases Tang, Qiaoyu, et al. 2023.06 [Preprint]

  ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs Qin, Yujia, et al. 2023.07 [Paper]

  MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use Huang, Yue, et al. 2023.10 [Paper]

  Making Language Models Better Tool Learners with Execution Feedback Qiao, Shuofei, et al. 2023.05 [Preprint]

  LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error Wang, Boshi, et al. 2024.03 [Preprint]

• Self-training with bootstrapped examples

  Toolformer: Language Models Can Teach Themselves to Use Tools Schick, Timo, et al. 2024 Paper

• Collect data from structured knowledge sources, e.g., databases, knowledge graphs, etc.

  LaMDA: Language Models for Dialog Applications Thoppilan, Romal, et al. 2022.01 [Paper]

  TALM: Tool Augmented Language Models Parisi, Aaron, Yao Zhao, and Noah Fiedel. 2022.05 [Preprint]

  ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings Hao, Shibo, et al. 2024 [Paper]

  ToolQA: A Dataset for LLM Question Answering with External Tools Zhuang, Yuchen, et al. 2024 [Paper]

  Middleware for LLMs: Tools are Instrumental for Language Agents in Complex Environments Gu, Yu, et al. 2024 [Paper]

  GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Information Jin, Qiao, et al. 2024 [Paper]

• Search information from the web

  Internet-augmented language models through few-shot prompting for open-domain question answering Lazaridou, Angeliki, et al. 2022.03 [Paper]

  Internet-Augmented Dialogue Generation Komeili, Mojtaba, Kurt Shuster, and Jason Weston. 2022 [Paper]

• Viewing retrieval models as tools under the retrieval-augmented generation context

  Retrieval-based Language Models and Applications Asai, Akari, et al. 2023 [Tutorial]

  Augmented Language Models: a Survey Mialon, Grégoire, et al. 2023.02 [Paper]

• Using calculator for math calculations

  Toolformer: Language Models Can Teach Themselves to Use Tools Schick, Timo, et al. 2024 [Paper]

  Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems Kadlčík, Marek, et al. 2023 [Paper]

• Using programs/Python interpreter to perform more complex operations

  Pal: Program-aided language models Gao, Luyu, et al. 2023 [Paper]

  Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks Chen, Wenhu, et al. 2022.11 [Paper]

  Mint: Evaluating llms in multi-turn interaction with tools and language feedback Wang, Xingyao, et al. 2023.09 [Paper]

  MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning Das, Debrup, et al. 2024 [Paper]

  ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving Gou, Zhibin, et al. 2023.09 [Paper]

• Tools for more advanced business activities, e.g., financial, medical, education, etc.

  On the Tool Manipulation Capability of Open-source Large Language Models Xu, Qiantong, et al. 2023.05 [Paper]

  ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases Tang, Qiaoyu, et al. 2023.06 [Preprint]

  Mint: Evaluating llms in multi-turn interaction with tools and language feedback Wang, Xingyao, et al. 2023.09 [Paper]

  AgentMD: Empowering Language Agents for Risk Prediction with Large-Scale Clinical Tool Learning Jin, Qiao, et al. 2024.02 [Paper]

• Access real-time or real-world information such as weather, location, etc.

  On the Tool Manipulation Capability of Open-source Large Language Models Xu, Qiantong, et al. 2023.05 [Paper]

  ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases Tang, Qiaoyu, et al. 2023.06 [Preprint]

• Managing personal events such as calendar or emails

  Toolformer: Language Models Can Teach Themselves to Use Tools Schick, Timo, et al. 2024 [Paper]

• Tools in embodied environments, e.g., the Minecraft world

  Voyager: An Open-Ended Embodied Agent with Large Language Models Wang, Guanzhi, et al. 2023.05 [Paper]

• Tools interacting with the physical world

  ProgPrompt: Generating Situated Robot Task Plans using Large Language Models Singh, Ishika, et al. 2023 [Paper]

  Alfred: A benchmark for interpreting grounded instructions for everyday tasks Shridhar, Mohit, et al. 2020 [Paper]

  Autonomous chemical research with large language models Boiko, Daniil A., et al. 2023 [Paper]

• Tools providing access to information in non-textual modalities

  Vipergpt: Visual inference via python execution for reasoning Surís, Dídac, Sachit Menon, and Carl Vondrick. 2023 [Paper]

  MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action Yang, Zhengyuan, et al. 2023.03 [Preprint]

  AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn Gao, Difei, et al. 2023.06 [Preprint]

• Tools that can answer questions about data in other modalities

  Visual Programming: Compositional visual reasoning without training Gupta, Tanmay, and Aniruddha Kembhavi. 2023 [Paper]

• Text-generation models that can perform specific tasks, e.g., question answering, machine translation

  Toolformer: Language Models Can Teach Themselves to Use Tools Schick, Timo, et al. 2024 [Paper]

  ART: Automatic multi-step reasoning and tool-use for large language models Paranjape, Bhargavi, et al. 2023.03 [Preprint]

• Integration of available models on Huggingface, TorchHub, TensorHub, etc.

  HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face Shen, Yongliang, et al. 2024 [Paper]

  Gorilla: Large language model connected with massive apis Patil, Shishir G., et al. 2023.05 [Paper]

  Taskbench: Benchmarking large language models for task automation Shen, Yongliang, et al. 2023.11 [Paper]

• Train retrievers that map natural language instructions to tool documentation

  DocPrompting: Generating Code by Retrieving the Docs Zhou, Shuyan, et al. 2022.07 [Paper]

  ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs Qin, Yujia, et al. 2023.07 [Paper]

• Ask LMs to write hypothetical tool descriptions and search relevant tools

  CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets Yuan, Lifan, et al. 2023.09 [Paper]

• Complex tool usage, e.g., parallel calls

  Function Calling and Other API Updates Eleti, Atty, et al. 2023.06 [Blog]

  An LLM Compiler for Parallel Function Calling Kim, Sehoon, et al. 2023.12 [Paper]

• Domain-specific logical forms to query structured data

  Semantic parsing on freebase from question-answer pairs Berant, Jonathan, et al. 2013 [Paper]

  Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task Yu, Tao, et al. 2018.09 [Paper]

  Break It Down: A Question Understanding Benchmark Wolfson, Tomer, et al. 2020 [Paper]

• Domain-specific actions for agentic tasks such as web navigation

  Reinforcement Learning on Web Interfaces using Workflow-Guided Exploration Liu, Evan Zheran, et al. 2018.02 [Paper]

  WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents Yao, Shunyu, et al. 2022.07 [Paper]

  Webarena: A realistic web environment for building autonomous agents Zhou, Shuyan, et al. 2023.07 [Paper]

• Using external Python libraries as tools

  ToolCoder: Teach Code Generation Models to use API search tools Zhang, Kechi, et al. 2023.05 [Paper]

• Using expert designed functions as tools to answer questions about images

  Visual Programming: Compositional visual reasoning without training Gupta, Tanmay, and Aniruddha Kembhavi. 2023 [Paper]

  Vipergpt: Visual inference via python execution for reasoning Surís, Dídac, Sachit Menon, and Carl Vondrick. 2023 [Paper]

• Using GPT as a tool to query external Wikipedia knowledge for table-based question answering

  Binding Language Models in Symbolic Languages Cheng, Zhoujun, et al. 2022.10 [Paper]

• Incorporate QA API and operation APIs to assist table-based question answering

  API-Assisted Code Generation for Question Answering on Varied Table Structures Cao, Yihan, et al. 2023.12 [Paper]

• Approaches to abstract libraries for domain-specific logical forms from a large corpus

  DreamCoder: growing generalizable, interpretable knowledge with wake--sleep Bayesian program learning Ellis, Kevin, et al. 2020.06 [Paper]

  Leveraging Language to Learn Program Abstractions and Search Heuristics] Wong, Catherine, et al. 2021 [Paper]

  Top-Down Synthesis for Library Learning Bowers, Matthew, et al. 2023 [Paper]

  LILO: Learning Interpretable Libraries by Compressing and Documenting Code Grand, Gabriel, et al. 2023.10 [Paper]

• Make and learn skills (Java programs) in the embodied Minecraft world

  Voyager: An Open-Ended Embodied Agent with Large Language Models Wang, Guanzhi, et al. 2023.05 [Paper]

• Leverage LMs as tool makers on BigBench tasks

  Large Language Models as Tool Makers Cai, Tianle, et al. 2023.05 [Preprint]

• Create tools for math and table QA tasks by example-wise tool making

  CREATOR: Disentangling Abstract and Concrete Reasonings of Large Language Models through Tool Creation Qian, Cheng, et al. 2023.05 [Paper]

• Make tools via heuristic-based training and tool deduplication

  CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets Yuan, Lifan, et al. 2023.09 [Paper]

• Learning tools by refactoring a small amount of programs

  ReGAL: Refactoring Programs to Discover Generalizable Abstractions Stengel-Eskin, Elias, Archiki Prasad, and Mohit Bansal. 2024.01 [Preprint]

• A training-free approach to make tools via execution consistency

  🎁 TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Tasks Wang, Zhiruo, Daniel Fried, and Graham Neubig. 2024.01 [Preprint]

• Datasets that require reasoning over texts

  Measuring Mathematical Problem Solving With the MATH Dataset Hendrycks, Dan, et al. 2021.03 [Paper]

  Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models Srivastava, Aarohi, et al. 2022.06 [Paper]

• Datasets that require reasoning over structured data, e.g., tables

  Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning Lu, Pan, et al. 2022.09 [Paper]

  Compositional Semantic Parsing on Semi-Structured Tables Pasupat, Panupong, and Percy Liang. 2015 [Paper]

  HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation Cheng, Zhoujun, et al. 2022 [Paper]

• Datasets that require reasoning over other modalities, e.g., images and image pairs

  Gqa: A new dataset for real-world visual reasoning and compositional question answering Hudson, Drew A., and Christopher D. Manning. 2019.02 [Paper]

  A Corpus for Reasoning about Natural Language Grounded in Photographs Suhr, Alane, et al. 2019 [Paper]

• Example datasets that require retriever model (tool) to solve

  Natural Questions: A Benchmark for Question Answering Research Kwiatkowski, Tom, et al. 2019 [Paper]

  TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension Joshi, Mandar, et al. 2017 [Paper]

• Collect RapidAPIs and use models to synthesize examples for evaluation

  ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs Qin, Yujia, et al. 2023.07 [Paper]

• Collect APIs from PublicAPIs and use models to synthesize examples

  ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases Tang, Qiaoyu, et al. 2023.06 [Preprint]

• Collect APIs from PublicAPIs and manually annotate examples for evaluation

  API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs Li, Minghao, et al. 2023.12 [Paper]

• Collect APIs from OpenAI plugin list and use models to synthesize examples

  MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use Huang, Yue, et al. 2023.10 [Paper]

• Collect neural model tools from Huggingface hub, TorchHub, and TensorHub

  Gorilla: Large language model connected with massive apis Patil, Shishir G., et al. 2023.05 [Paper]

• Collect neural model tools from Huggingface

  HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face Shen, Yongliang, et al. 2024 [Paper]

• Collect tools from Huggingface and PublicAPIs

  Taskbench: Benchmarking large language models for task automation Shen, Yongliang, et al. 2023.11 [Paper]

• Collect Action Sequences in real-world macOS/iPadOS/iOS.

  ShortcutsBench: A Large-Scale Real-World Benchmark for API-Based Agents Shen, Haiyang, et al. 2024.07 [Paper]