Giter Club home page Giter Club logo

checktestdata's Introduction

Checktestdata

Build Status Coverage Status Coverity Scan Status

Checktestdata is a tool to verify the syntactical integrity of test cases in programming contests like the ACM ICPC.

It allows you to specify a simple grammar for your testdata input files, according to which the testdata is checked. In the examples directory you find two sample scripts hello.ctd and fltcmp.ctd from the DOMjudge sample problems hello and fltcmp, and under examples/nwerc2008 the scripts used for the NWERC 2008 problemset.

Grammar specification

Checktestdata language specification

We also have a Haskell implementation of the checktestdata program, which supports this specification as well as a Haskell embedded domain-specific language.

Installation and use

Requirements:

If you don't have flexc++ and/or bisonc++ available, you may use the release branch where we've pre-generated the scanner/parser files.

Command line for installing the build dependencies on Debian or Ubuntu, when using the release branch:

apt-get install make g++ libboost-dev libgmp-dev autotools-dev automake

For Redhat-like distributions try:

yum install make g++ boost-devel gmp-devel automake

To compile checktestdata, run:

./bootstrap
make dist
make

Leave out the make dist step if you use the pre-generated scanner/parser files on the release branch.

Finally, you can run

make check

to run a number of tests. Note that a few of the tests are non-deterministic and may sometimes return an error. This is ok, but the make check command should run successfully more than 90% of the time.

For usage instructions run:

checktestdata --help

Library

The commandline program is built upon the separate library libchecktestdata.hpp (see checktestdata.cc as an example for how to use this library) that can be used to write the syntax checking part of special compare scripts. It can easily handle the tedious task of verifying that a team's submission output is syntactically valid, leaving just the task of semantic validation to another program.

When you want to support presentation error as a verdict, also in variable output problems, the option whitespace-ok can be useful. This allows any non-empty sequence of whitespace (no newlines though) where the SPACE command is used, as well as leading and trailing whitespace on lines (when using the NEWLINE command). Please note that with this option enabled, whitespace matching is greedy, so the script code

INT(1,2) SPACE SPACE INT(1,2)

does not match 1__2 (where the two underscores represent spaces), because the first SPACE command already matches both, so the second cannot match anything.

Copyright & Licencing

Checktestdata is Copyright © 2008 - 2019 by the checktestdata developers and all respective contributors. The current checktestdata developers are Jeroen Bransen, Jaap Eldering, Jan Kuipers, and Tobias Werth; see the git commits for a complete list of contributors.

Checktestdata, including its documentation, is free software; you can redistribute it and/or modify it under the terms of the two-clause BSD license. See the file COPYING.

The M4 autoconf macros are licenced under all-permissive and GPL3+ licences; see the respective files under m4/ for details.

Contact

The developers can be reached through the mailinglist [email protected]. You need to be subscribed before you can post. Information, subscription and archives are available at: https://www.domjudge.org/mailman/listinfo/domjudge-devel

DOMjudge has a Slack workspace where a number of developers and users of DOMjudge & checktestdata linger. More information can be found at https://www.domjudge.org/chat

checktestdata's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

checktestdata's Issues

Improve performance

For some checktestdata programs performance is terrible. Are there low hanging fruits that improve the general performance of checktestdata?

leading zeros in front of real numbers not rejected

Consider the following script:

INT(0, 10, n) SPACE FLOATP(0, 10, 0, 5, x) NEWLINE

It rejects the following input:

05 5.23

But accepts the following input:

5 00000000000000000000000005.23

(using FLOAT instead of FLOATP gives the same results).

Is there a rationale for this, or just an oversight?

Specifying precision of reals

Feature request: for the FLOAT command, it would be very good if it was possible to specify the number of allowed digits after the decimal point, since input specifications are not really complete without such bounds. It should then be possible both to specify an upper bound on the number of decimals, and an exact bound (probably it is easiest to think of it as specifying a range).

Unusually poor performance in current release version

The Oct 21 release version of problemtools seems to have severe performance degradation from the April 22 release, on slow-running scripts.

Concrete examples include the ascendingphoto problem from NWERC 2017 (e.g. case 07-big.in went from 4.7 to 6.0 seconds) even though b281d3f, added between the two versions, claims to improve performance on that specific set.

As an extreme example, I found one script that took ~5 minutes to run on the April 22 release, and more than 3 hours (after which I decided to stop waiting) to run on the Oct 21 release.

Compile Error in current master

Hi,

I'm building checktestdata on WSL. When compiling on master, I get:

g++ -g -O2 -Wall -fstack-protector -fPIE -D_FORTIFY_SOURCE=2 -std=c++14 -DVERSION="\"20200702\"" -I/usr/include  -c -o parse.o parse.cc
In file included from parse.cc:6:0:
parser.ih: In member function ‘int Parser::lex()’:
parser.ih:23:2: error: ‘d_val_’ was not declared in this scope
  d_val_ = STYPE_(d_scanner.matched());
  ^~~~~~
parser.ih:23:2: note: suggested alternative: ‘d_val__’
  d_val_ = STYPE_(d_scanner.matched());
  ^~~~~~
  d_val__
parser.ih:23:11: error: ‘STYPE_’ was not declared in this scope
  d_val_ = STYPE_(d_scanner.matched());
           ^~~~~~
parser.ih:23:11: note: suggested alternative: ‘STYPE__’
  d_val_ = STYPE_(d_scanner.matched());
           ^~~~~~
           STYPE__
parse.cc: At global scope:
parse.cc:2363:30: error: no ‘void Parser::errorRecovery__()’ member function declared in class ‘Parser’
 void Parser::errorRecovery__()
                              ^
parse.cc:2395:44: error: no ‘void Parser::executeAction__(int)’ member function declared in class ‘Parser’
 void Parser::executeAction__(int production)
                                            ^
checktestdata.y:154:1: error: expected unqualified-id before ‘catch’
checktestdata.y:160:26: error: no ‘void Parser::nextCycle__()’ member function declared in class ‘Parser’
checktestdata.y:192:1: error: expected unqualified-id before ‘catch’
checktestdata.y:206:26: error: no ‘void Parser::nextToken__()’ member function declared in class ‘Parser’
checktestdata.y:233:22: error: no ‘void Parser::print__()’ member function declared in class ‘Parser’
checktestdata.y: In member function ‘int Parser::parse()’:
checktestdata.y:269:9: error: ‘nextCycle__’ was not declared in this scope
checktestdata.y:269:9: note: suggested alternative: ‘nextCycle_’
<builtin>: recipe for target 'parse.o' failed
make: *** [parse.o] Error 1

The release branch builds successfully. I'm a C++ noob and I've no idea what you might need to help :-) Feel free to ask! Some versions:

  • g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
  • bisonc++ V6.01.00
  • flexc++ V2.06.02

Add a flag to pass a deterministic seed in combination with `--generate`

Currently --generate output is non-reproducible because the random seed is not deterministic.

It would be nice to add a --seed {seed}/-s {seed} flag to make this deterministic. That way, .ctd files can be used to deterministically generate testdata from generators/generators.yaml files:

Create (or symlink) generators/ctd/input_format.ctd.
Create generators/ctd/run shell script with contents checktestdata --generate --seed $1 ./input_format.ctd.
Call it from generators.yaml using ctd {seed}.

`FLOAT` never matches `0`

For example, running FLOAT(-1, 1) against 0 gives:

0
 ^


ERROR: line 1 character 2 of testdata doesn't match FLOAT(-(1),1): prefix zero(s)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.