To view as a website, see: http://zhangyushao.site/awesome-database-testing/.

This is a curitated list of resources on database testing.

What covered: papers, blogs, notes, tools and etc. for whoever wants to test a database mangement system (DBMS).

What is not: using SQLs, learning database basic knowledge, implementing a database system.

Note: this is NOT an exhaustive list of materials, you may want to find more accroding to references of each item.

Here we put materials with general discussions.

NOTE: We only include peer-reviewed and published papers here, the same also applies to other entries. You may want to find the authors' free-version of the papers on their personal page.

• Understanding the query optimization Query Optimization in Database Systems [1984]

• Understanding the query optimization - a more recent view An Overview of Query Optimization [1998]

• What are the problems from the indutrial view Testing SQL Server's Query Optimizer : Challenges , Techniques and Experiences

• Optimizer Evaluation How good are query optimizers, really?

• What are the problems from the academic view Is Query Optimization a “Solved” Problem?

• Query generation with template substitution Generating Thousand Benchmark Queries in Seconds
• Language for customized data generator Flexible Database Generators
• RAGS Microsoft SQL Server Massive Stochastic Testing of SQL
• Generate data regarding the query constraints QAGen: Generating Query-Aware Test Databases
• How could symbolic execution help Qex: Symbolic SQL Query Explorer
• Data generation as a search problem Search-based test data generation for SQL queries
• For more valid inputs! SQUIRREL: Testing Database Management Systems with Language Validity and Coverage Feedback [CCS 2021]

• Randgen MySQL version (not maintained) doc, MariaDB version, PingCAP version
• pquery - stress testing MySQL
• Sysbench - MySQL benchmarks
• SQLsmith - fuzzing style tool
• Squirrel - tool for CCS 2021 paper
• mtr_to_sql.sh@Percona-QA - a script to extract mysql-test cases as a sql file, with engine replacement and shuffles according to your need

Here we mainly refer to the case where the engine returns wrong query result.

• RAGS with system-differential testing Massive Stochastic Testing of SQL
• Data generation with contraints sovling Query-Aware Test Generation Using a Relational Constraint Solver
• Differential testing for the optimizing rules A framework for testing query transformation rules
• Criteria on integrtity testing The Effectiveness of Test Coverage Criteria for Relational Database Schema Integrity Constraints

• Parallelism on the old machines Quickly generating billion-record synthetic databases
• Verified database system Toward a Verified Relational Database Management System
• Reply the transactions Debugging transactions and tracking their provenance with reenactment

• Search-based test data generation for SQL queries [2018]
• Automated verification of query equivalence using satisfiability modulo theories [2019]
• Detecting Optimization Bugs in Database Engines via Non-Optimizing Reference Engine Construction [2020]
• Testing query execution engines with mutations [2020]

• SQLancer@ETH_ZURICH
• go-sqlancer@PingCAP
• pquery@Percona-QA - a tool providing the ability to combine correctness and crash recovery testing on both single-node and clustered instances

Here we focus on detecting and debugging performance issues.

• Very first system Efficient testing of high performance transaction processing systems
• Is the exact estimation possible Exact Cardinality Query Optimization for Optimizer Testing
• How to identify issues on plan generation On the stability of plan costs and the costs of plan stability
• What's wrong with the models Testing cardinality estimation models in SQL server

• Generate and compare the plans Testing the accuracy of query optimizers

• APOLLO: Automatic Detection and Diagnosis of Performance Regressions in Database Systems [2020]

• TPC Benchmarking
• APOLLO@gatech

We list idea and tools which are adopted in the industry.

• Chaos Mesh from PingCAP
  • Article: Building an Automated Testing Framework Based on Chaos Mesh® and Argo
  • Talk: Testing TiDB Using Chaos Mesh® with TiPocket

• zhangysh1995
• mkx22

What you can contribute:

• papers (only peer-reviewed papers will be accepted to this repo)
• talks/slides/articles
• tools

Please also add yourself to Contributors in the PR.