New test file format - using testbook? about otter-grader HOT 5 CLOSED

So it's been a long time since this issue was opened but I think I've finally figured out a good schematic for a new test case format that relies on more traditional unit test-like formatting.

I'm thinking of a format that looks something like this:

from otter.test_files import test_case

name = "q1"
points = None

@test_case()
def test_sieve_1(env):
    assert env["sieve"](1) == set()

@test_case(name="q1b")
def test_sieve_2(sieve):
    assert sieve(2) == {2}

@test_case(name="q1c")
def test_sieve_3(sieve, env):
    assert sieve(3) == {2, 3}
    assert env["a"] == 1

@test_case(name="q1d", hidden=True)
def test_sieve_4(env):
    assert env["sieve"](20) == {2, 3, 5, 7, 11, 13, 17, 19}

@test_case(name="q1e", points=1, hidden=True)
def test_sieve_5(env):
    assert env["sieve"](100) == {2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 
        61, 67, 71, 73, 79, 83, 89, 97}

The idea is similar to pytest, where a test fails if an exception is raised, and passes otherwise. The decorator is an object provided by Otter to store the test case metadata, and the student's global environment (or specific elements in it) are accessed through arguments to the test case functions.

from otter-grader.

Wow, this is really interesting, thanks! There have been some conversations recently about the limitations of the current testing format/framework for OK autograders (like what came up at the workshop) and so changing the testing format is something we might consider for a future release (v2?) as OkPy becomes more of a legacy solution and more classes get ported away from it.

Let's keep this thread open as a forum for considering this and other unit-testing frameworks as a method for writing Otter's tests!

from otter-grader.

👍 I bet they'd be excited to see testbook used in grading as well!

from otter-grader.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

from otter-grader.

Implemeted in #376. This feature will be included in Otter v4.

from otter-grader.