This repository contains the test suite data for evaluating database queries as described in the paper:

Searching for Better Database Queries in the Outputs of Semantic Parsers

The test suites can be downloaded from this link.

The test suites were created for five datasets: Spider, GeoQuery, IMDB, Yelp, and Academic.

The names of the database files for each dataset are in the following format:

{dataset_name}_{test_database_count}_{gold_query_index}_ts.sqlite

The gold_query_index is the index of the corresponding gold query. You can find the mapping between the gold queries and their indexes in the file {dataset_name}_query_to_index.json, which is located in each dataset folder.

For each gold query, there is a separate test suite with several test databases. The test_database_count parameter enumerates all the databases corresponding to a given gold query. For instance, if a gold query in the Yelp dataset has an index of gold_query_index=5, there would be three test databases in the test suite, named yelp_0_5_ts.sqlit, yelp_1_5_ts.sqlite, and yelp_2_5_ts.sqlite.

Originally, the test suites were introduced in the following work: https://github.com/ruiqi-zhong/TestSuiteEval.

Our modifications are outlined in our paper, specifically in Section 4 and Appendix A which detail the construction of the test suite.

Below is the comparison of the test suites’ statistics.