In recent years, a significant number of question answering (QA) systems that retrieve answers to natural language questions from knowledge graphs (KG) have been introduced. However, finding a benchmark to appropriately measure the quality of a question answering system is a difficult task because of (1) the high degree of variation with respect to the fine-grained properties among the available benchmarks; (2) the available benchmarks are static by nature while the KG evolving over time; (3) the existing benchmarks in the literature target only a handful of KGs. We introduce SmartBench, an on-the-fly benchmarking system for QA over KGs systems that can be used to build a QA benchmark over any KG. The benchmark generated by SmartBench is guaranteed to cover all the properties of the natural language questions and queries that were encountered in the literature as long as the targeted KG’s topology demonstrates these properties. Moreover, SmartBench generates benchmarks that follow the same standards and complexity distribution to solve the variation of the fine-grained properties among generated benchmarks.

• The web verion is available here
• Talk: https://www.youtube.com/toPublish
• Demo talk: https://www.youtube.com/toPublish

• Standardized: We propose a list of standards that need to be met in a benchmark to accurately evaluate a QA system to solve the Variations problem.
• On-The-Fly: We make the benchmark generation process to be on-the-fly to avoid the Staleness issue. The system builds a new benchmark from the KG just before the evaluation process to guarantee adaptability to the evolving KG.
• Systematic: Generate complex questions from simple facts in a systematic way (for generalization and scalability).
• Richness: Generated questions vary in word choice (lexicon) and morphology and syntax (grammar) to evaluate the system’s ability to understand the same question written in different ways.
• Coverability: We defined the question complexity to use it in tuning SmartBench to cover all possible complexities in real-world deployment.

Paper: ToPublish

@Article{Orogat2022SmartBench,
  Title                    = {},
  Author                   = {},
  Journal                  = {},
  Year                     = {},
  volume                   = {},
  number                    = {}
}

• Run CBench
  • Getting Started
  • Lexicon Generation
  • Benchmark Generation
    • DBpedia
    • Wikidata
    • GeoLinkedData
    • MAKG
    • Nobel Prize
  • Benchmark Fine-Grained Analysis
  • QA Evaluatioe

SmartBench requires the following development kits and liberaries. You can download the liberaries with the system.

• for Java [JDK 8, Download liberaries from Lib folder]
• for Python [Python 3, nltk, inflect, Spacy]

• Download CBench.jar: Download the CBench.jar file and other folders. The project structure must be as follow

projectFolder
│   
└─── SmartBench.jar
│
|─── data
│   |─── benchmark.json
│   |─── src
│       │─── 
│       └─── 
│
└─── lib
    └─── ... All .jar files

• Run CBench.jar: Using the command java -jar "PATH/TO/projectFolder/SmartBench.jar", run the project.
• Configure the System: While the system is running, it asks you about some parameters. Theses parameters are
  • KG's name: KG name as you prefer.
  • KG's URL: The URL of SPARQL endpoint of the desired knowledge graph.

Please raise potential bugs on github. If you have a research related question, please send it to this email([email protected])