Awesome LLM Eval

English | 中文

Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, learderboard, papers, docs and models, mainly for Evaluation on Large Language Models (like ChatGPT, LLaMA, GLM, Baichuan, etc).

Awesome-LLM-Eval: 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表，主要面向大型语言模型评测（例如ChatGPT、LLaMA、GLM、Baichuan等）。

• News
• Tools
• Datasets / Benchmark
  • 通用
  • 垂直领域
  • Agent能力
  • 多模态/跨模态
• Demos
• Leaderboards
• Papers
• LLM-List
  • Pre-trained LLM
  • Instruction finetuned LLM
  • Aligned LLM
  • Open LLM
  • Popular LLM
• LLMOps
• Frameworks for Training
• Courses
• Others
• Other-Awesome-Lists
• Licenses
• Citation

• [2023/11/15] We add Instruction_Following_Eval and LLMBar for the evaluation of Instruction Following ability of LLMs.

• [2023/10/20] We add SuperCLUE-Agent for LLM agent evaluation.

• [2023/09/25] We add ColossalEval from Colossal-AI.

• [2023/09/22] We add Leaderboards chapter.

• [2023/09/20] We add DeepEval, FinEval and SuperCLUE-Safety from CLUEbenchmark.

• [2023/09/18] We add OpenCompass from Shanghai AI Lab.

• [2023/08/03] We add new Chinese LLMs: Baichuan and Qwen.

• [2023/06/28] We add AlpacaEval and multiple tools.

• [2023/04/26] We released the V0.1 Eval list with multiple benchmarks, etc.

• Chat Arena: anonymous models side-by-side and vote for which one is better - 开源AI大模型“匿名”竞技场！你在这里可以成为一名裁判，给两个事先不知道名字的模型回答打分，评分后将给出他们的真实身份。目前已经“参赛”的选手包括Vicuna、Koala、OpenAssistant (oasst)、Dolly、ChatGLM、StableLM、Alpaca、LLaMA等。

• Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators,
  by Liang Chen, Yang Deng, Yatao Bian et al.
  
  
• A Closer Look into Automatic Evaluation Using Large Language Models,
  by Cheng-han Chiang, Hungyi Li
  
  
• A Survey on Evaluation of Large Language Models,
  by Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi et al.
  
  
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment,
  by Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, Chenguang Zhu
  
  
• A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity,
  by Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji et al.
  
  
• Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators,
  by Liang Chen, Yang Deng, Yatao Bian, Zeyu Qin, Bingzhe Wu, Tat-Seng Chua, Kam-Fai Wong
  
  
• Is ChatGPT a General-Purpose Natural Language Processing Task Solver?,
  by Qin, Chengwei, Zhang, Aston, Zhang, Zhuosheng, Chen, Jiaao, Yasunaga, Michihiro and Yang, Diyi
  
  
• ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbots,
  by Reham Omar, Omij Mangukiya, Panos Kalnis and Essam Mansour
  
  
• Mathematical Capabilities of ChatGPT,
  by Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier and Julius Berner
  
  
• Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization,
  by Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen and Wei Cheng
  
  
• On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective,
  by Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang et al.
  
  
• ChatGPT is not all you need. A State of the Art Review of large Generative AI models,
  by Roberto Gozalo-Brizuela and Eduardo C. Garrido-Merch'an
  
  
• Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT,
  by Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du and Dacheng Tao
  
  
• Evaluation of ChatGPT as a Question Answering System for Answering Complex Questions,
  by Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen and Guilin Qi
  
  
• ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models,
  by Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu and Ben He
  
  
• Holistic Evaluation of Language Models,
  by Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan et al.
  
  
• Evaluating the Text-to-SQL Capabilities of Large Language Models,
  by Nitarshan Rajkumar, Raymond Li and Dzmitry Bahdanau
  
  
• Are Visual-Linguistic Models Commonsense Knowledge Bases?,
  by Hsiu-Yu Yang and Carina Silberer
  
  
• Is GPT-3 a Psychopath? Evaluating Large Language Models from a Psychological Perspective,
  by Xingxuan Li, Yutong Li, Linlin Liu, Lidong Bing and Shafiq R. Joty
  
  
• GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models,
  by Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li and Kai-Wei Chang
  
  
• RobustLR: A Diagnostic Benchmark for Evaluating Logical Robustness of Deductive Reasoners,
  by Soumya Sanyal, Zeyi Liao and Xiang Ren
  
  
• A Systematic Evaluation of Large Language Models of Code,
  by Frank F. Xu, Uri Alon, Graham Neubig and Vincent J. Hellendoorn
  
  
• Evaluating Large Language Models Trained on Code,
  by Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond'e de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda et al.
  
  
• GLGE: A New General Language Generation Evaluation Benchmark,
  by Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu et al.
  
  
• Evaluating Pre-Trained Models for User Feedback Analysis in Software Engineering: A Study on Classification of App-Reviews,
  by Mohammad Abdul Hadi and Fatemeh H. Fard
  
  
• Do Language Models Perform Generalizable Commonsense Inference?,
  by Peifeng Wang, Filip Ilievski, Muhao Chen and Xiang Ren
  
  
• RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms,
  by Pei Zhou, Rahul Khanna, Seyeon Lee, Bill Yuchen Lin, Daniel Ho, Jay Pujara and Xiang Ren
  
  
• Evaluation of Text Generation: A Survey,
  by Asli Celikyilmaz, Elizabeth Clark and Jianfeng Gao
  
  
• Neural Language Generation: Formulation, Methods, and Evaluation,
  by Cristina Garbacea and Qiaozhu Mei
  
  
• BERTScore: Evaluating Text Generation with BERT,
  by Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger and Yoav Artzi
  
  

• LLaMA - A foundational, 65-billion-parameter large language model. LLaMA.cpp Lit-LLaMA

  • Alpaca - A model fine-tuned from the LLaMA 7B model on 52K instruction-following demonstrations. Alpaca.cpp Alpaca-LoRA
  • Flan-Alpaca - Instruction Tuning from Humans and Machines.
  • Baize - Baize is an open-source chat model trained with LoRA. It uses 100k dialogs generated by letting ChatGPT chat with itself.
  • Cabrita - A portuguese finetuned instruction LLaMA.
  • Vicuna - An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality.
  • Llama-X - Open Academic Research on Improving LLaMA to SOTA LLM.
  • Chinese-Vicuna - A Chinese Instruction-following LLaMA-based Model.
  • GPTQ-for-LLaMA - 4 bits quantization of LLaMA using GPTQ.
  • GPT4All - Demo, data, and code to train open-source assistant-style large language model based on GPT-J and LLaMa.
  • Koala - A Dialogue Model for Academic Research
  • BELLE - Be Everyone's Large Language model Engine
  • StackLLaMA - A hands-on guide to train LLaMA with RLHF.
  • RedPajama - An Open Source Recipe to Reproduce LLaMA training dataset.
  • Chimera - Latin Phoenix.

• BLOOM - BigScience Large Open-science Open-access Multilingual Language Model BLOOM-LoRA

  • BLOOMZ&mT0 - a family of models capable of following human instructions in dozens of languages zero-shot.
  • Phoenix

• T5 - Text-to-Text Transfer Transformer

  • T0 - Multitask Prompted Training Enables Zero-Shot Task Generalization

• OPT - Open Pre-trained Transformer Language Models.

• UL2 - a unified framework for pretraining models that are universally effective across datasets and setups.

• GLM- GLM is a General Language Model pretrained with an autoregressive blank-filling objective and can be finetuned on various natural language understanding and generation tasks.

  • ChatGLM-6B - ChatGLM-6B 是一个开源的、支持中英双语的对话语言模型，基于 General Language Model (GLM) 架构，具有 62 亿参数.
  • ChatGLM2-6B -开源中英双语对话模型 ChatGLM-6B 的第二代版本，在保留了初代模型对话流畅、部署门槛较低等众多优秀特性的基础之上，ChatGLM2-6B 引入了更长的上下文、更好的性能和更高效的推理.

• RWKV - Parallelizable RNN with Transformer-level LLM Performance.

  • ChatRWKV - ChatRWKV is like ChatGPT but powered by my RWKV (100% RNN) language model.

• StableLM - Stability AI Language Models.

• YaLM - a GPT-like neural network for generating and processing text. It can be used freely by developers and researchers from all over the world.

• GPT-Neo - An implementation of model & data parallel GPT3-like models using the mesh-tensorflow library.

• GPT-J - A 6 billion parameter, autoregressive text generation model trained on The Pile.

  • Dolly - a cheap-to-build LLM that exhibits a surprising degree of the instruction following capabilities exhibited by ChatGPT.

• Pythia - Interpreting Autoregressive Transformers Across Time and Scale

  • Dolly 2.0 - the first open source, instruction-following LLM, fine-tuned on a human-generated instruction dataset licensed for research and commercial use.

• OpenFlamingo - an open-source reproduction of DeepMind's Flamingo model.

• Cerebras-GPT - A Family of Open, Compute-efficient, Large Language Models.

• GALACTICA - The GALACTICA models are trained on a large-scale scientific corpus.

  • GALPACA - GALACTICA 30B fine-tuned on the Alpaca dataset.

• Palmyra - Palmyra Base was primarily pre-trained with English text.

• Camel - a state-of-the-art instruction-following large language model designed to deliver exceptional performance and versatility.

• h2oGPT

• PanGu-α - PanGu-α is a 200B parameter autoregressive pretrained Chinese language model develped by Huawei Noah's Ark Lab, MindSpore Team and Peng Cheng Laboratory.

• MOSS - MOSS是一个支持中英双语和多种插件的开源对话语言模型.

• Open-Assistant - a project meant to give everyone access to a great chat based large language model.

  • HuggingChat - Powered by Open Assistant's latest model – the best open source chat model right now and @huggingface Inference API.
  • Baichuan - An open-source, commercially available large-scale language model developed by Baichuan Intelligent Technology following Baichuan-7B, containing 13 billion parameters. (20230715)
  • • Qwen - Qwen-7B is the 7B-parameter version of the large language model series, Qwen (abbr. Tongyi Qianwen), proposed by Alibaba Cloud. Qwen-7B is a Transformer-based large language model, which is pretrained on a large volume of data, including web texts, books, codes, etc. (20230803)

• Accelerate - 🚀 A simple way to train and use PyTorch models with multi-GPU, TPU, mixed-precision.
• Apache MXNet - Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler.
• Caffe - A fast open framework for deep learning.
• ColossalAI - An integrated large-scale model training system with efficient parallelization techniques.
• DeepSpeed - DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
• Horovod - Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.
• Jax - Autograd and XLA for high-performance machine learning research.
• Kedro - Kedro is an open-source Python framework for creating reproducible, maintainable and modular data science code.
• Keras - Keras is a deep learning API written in Python, running on top of the machine learning platform TensorFlow.
• LightGBM - A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.
• MegEngine - MegEngine is a fast, scalable and easy-to-use deep learning framework, with auto-differentiation.
• metric-learn - Metric Learning Algorithms in Python.
• MindSpore - MindSpore is a new open source deep learning training/inference framework that could be used for mobile, edge and cloud scenarios.
• Oneflow - OneFlow is a performance-centered and open-source deep learning framework.
• PaddlePaddle - Machine Learning Framework from Industrial Practice.
• PyTorch - Tensors and Dynamic neural networks in Python with strong GPU acceleration.
• PyTorch Lightning - Deep learning framework to train, deploy, and ship AI products Lightning fast.
• XGBoost - Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library.
• scikit-learn - Machine Learning in Python.
• TensorFlow - An Open Source Machine Learning Framework for Everyone.
• VectorFlow - A minimalist neural network library optimized for sparse data and single machine environments.

名称	Stars 数	介绍
agenta		用于构建强大LLM应用的LLMOps平台。轻松尝试和评估不同提示、模型和工作流，以构建稳健的应用程序。
Arize-Phoenix		用于LLMs、视觉、语言和表格模型的ML可观测性。
BudgetML		在不到10行代码的情况下，以有限的预算部署ML推理服务。
CometLLM		100%开源的LLMOps平台，用于记录、管理和可视化LLM提示和链条。跟踪提示模板、提示变量、提示持续时间、令牌使用等其他元数据。评分提示输出并在单个UI中可视化聊天历史。
deeplake		流式传输大型多模态数据集，实现接近100%的GPU利用率。查询、可视化和版本控制数据。无需重新计算模型微调的嵌入即可访问数据。
Dify		开源框架旨在使开发人员（甚至非开发人员）能够快速构建基于大型语言模型的有用应用程序，确保它们具有可视性、可操作性和可改进性。
Dstack		在任何云（AWS、GCP、Azure、Lambda等）中进行经济高效的LLM开发。
Embedchain		用于在数据集上创建类似ChatGPT的机器人的框架。
GPTCache		创建语义缓存以存储LLM查询的响应。
Haystack		快速构建带有LLM代理、语义搜索、问答等功能的应用程序。
langchain		通过可组合性构建LLM应用程序。
LangFlow		通过拖放组件和聊天界面轻松实验和原型化LangChain流程的无忧方式。
LangKit		开箱即用的LLM遥测收集库，可提取有关LLM随时间性能如何的特征和提示、响应和元数据的配置文件，以规模找到问题。
LiteLLM 🚅		一个简单且轻量的100行包，用于跨OpenAI、Azure、Cohere、Anthropic、Replicate API端点标准化LLM API调用。
LlamaIndex		提供一个**接口，将您的LLMs与外部数据连接起来。
LLMApp		LLM App是一个Python库，帮助您以几行代码构建实时的LLM启用数据管道。
LLMFlows		LLMFlows是一个用于构建简单、明确和透明的LLM应用程序的框架，例如聊天机器人、问答系统和代理。
LLMonitor		用于AI应用程序和代理的可观测性和监控。通过强大的跟踪和日志记录来调试代理。使用分析工具深入研究请求历史。开发人员友好的模块，可以轻松集成到LangChain中。
magentic		无缝集成LLMs作为Python函数。使用类型注释来指定结构化输出。将LLM查询和函数调用与常规Python代码混合使用，创建复杂的LLM驱动功能。
Pezzo 🕹️		Pezzo是为开发人员和团队构建的开源LLMOps平台。仅需两行代码，您就可以轻松排除AI操作中的问题，协作和管理您的提示，在一个地方立即部署更改。
promptfoo		用于测试和评估提示质量的开源工具。创建测试用例，自动检查输出质量并捕获回归，降低评估成本。
prompttools		用于测试和尝试提示的开源工具。核心**是使开发人员能够使用熟悉的界面（如代码和笔记本）评估提示。只需几行代码，您就可以在不同模型之间测试提示和参数（无论您是使用OpenAI、Anthropic还是LLaMA模型）。您甚至可以评估向量数据库的检索准确性。
TrueFoundry	无Github链接	在自己的Kubernetes（EKS、AKS、GKE、On-prem）基础设施上部署LLMOps工具，包括Vector DBs、嵌入式服务器等，包括部署、微调、跟踪提示和提供完整的数据安全性和最佳GPU管理的开源LLM模型。使用最佳的软件工程实践在生产规模上训练和启动您的LLM应用程序。
ReliableGPT 💪		为您的生产LLM应用程序处理OpenAI错误（超载的OpenAI服务器、旋转的密钥或上下文窗口错误）。
Weights & Biases (Prompts)	无Github链接	开发者优先的W&B MLOps平台中的一套LLMOps工具。利用W&B Prompts来可视化和检查LLM执行流程，跟踪输入和输出，查看中间结果，安全管理提示和LLM链配置。
xTuring		使用快速和高效的微调来构建和控制您的个人LLMs。
ZenML		用于编排、实验和部署生产级ML解决方案的开源框架，具有内置的langchain和llama_index集成。

• 大语言模型课程notebooks集-Large Language Model Course - Course with a roadmap and notebooks to get into Large Language Models (LLMs).
• Full+Stack+LLM+Bootcamp - LLM相关学习/应用资源集.

• Evaluating Language Models by OpenAI, DeepMind, Google, Microsoft - Evaluating Language Models by OpenAI, DeepMind, Google, Microsoft.
• Efficient Finetuning of Quantized LLMs --- 低资源的大语言模型量化训练/部署方案 - 旨在构建和开源遵循指令的baichuan/LLaMA/Pythia/GLM中文大模型微调训练方法，该方法可以在单个 Nvidia RTX-2080TI上进行训练，多轮聊天机器人可以在单个 Nvidia RTX-3090上进行上下文长度 2048的模型训练。使用bitsandbytes进行量化，并与Huggingface的PEFT和transformers库集成.

• Awesome LLM - A curated list of papers about large language models.
• Awesome-Efficient-LLM - A curated list for Efficient Large Language Models.
• Awesome-production-machine-learning - A curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning.
• Awesome-marketing-datascience - Curated list of useful LLM / Analytics / Datascience resources.
• Awesome-llm-tools - Curated list of useful LLM tool.
• Awesome-LLM-Compression - A curated list for Efficient LLM Compression.
• Awesome-Multimodal-Large-Language-Models - A curated list of Multimodal Large Language Models.
• Awesome-LLMOps - An awesome & curated list of the best LLMOps tools for developers.
• Awesome-MLops - An awesome list of references for MLOps - Machine Learning Operations.
• Awesome ChatGPT Prompts - A collection of prompt examples to be used with the ChatGPT model.
• awesome-chatgpt-prompts-zh - A Chinese collection of prompt examples to be used with the ChatGPT model.
• Awesome ChatGPT - Curated list of resources for ChatGPT and GPT-3 from OpenAI.
• Chain-of-Thoughts Papers - A trend starts from "Chain of Thought Prompting Elicits Reasoning in Large Language Models.
• Instruction-Tuning-Papers - A trend starts from Natrural-Instruction (ACL 2022), FLAN (ICLR 2022) and T0 (ICLR 2022).
• LLM Reading List - A paper & resource list of large language models.
• Reasoning using Language Models - Collection of papers and resources on Reasoning using Language Models.
• Chain-of-Thought Hub - Measuring LLMs' Reasoning Performance
• Awesome GPT - A curated list of awesome projects and resources related to GPT, ChatGPT, OpenAI, LLM, and more.
• Awesome GPT-3 - a collection of demos and articles about the OpenAI GPT-3 API.

本项目遵循 MIT License.

本项目遵循 Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

如果我们的项目对您有帮助，请引用我们的项目。

@misc{junwang2023,
  author = {Jun Wang, Changyu Hou, Xiaorui Wang, Pengyong Li, Jingjing Gong, Chen Song, Peng Gao, Guotong Xie},
  title = {Awesome-LLM-Eval: a curated list of tools, benchmarks, demos, papers for Large Language Models Evaluation},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/onejune2018/Awesome-LLM-Eval}},
}

作者简介：

• SFE平台算法研发负责
• 医药发现大模型MPG(Molecular Pretraining GraphTransformer)作者
• 国际竞赛SemEval2022习语判别任务第一、MIT AI-Cure第一、VQA2021第一、TREC2021第一、EAD2019第一
• 主页：https://onejune2018.github.io/homepage/
• Google Scholar: https://scholar.google.com/citations?user=0Be01PgAAAAJ&hl=en