To reproduce:

Save sol.jl with:

test(x, f) = f(x)

Then save grader.jl with:

using ComparativeAutograder
test_cases = []
f(x) = x
for _ in 1:10
    push!(test_cases, TestCase((rand(), f)))
end

runtestsuite(TestSuite("test", test_cases))

Now run

julia grader.jl sol.jl sol.jl

The submission fails, and in the output you will find:

UndefVarError: #f not defined

This can be fixed by replacing push!(test_cases, TestCase((rand(), f))) with push!(test_cases, TestCase((rand(), x -> x))), but inserting the function like this is impractical when working with more sophisticated functions.

We'd be able to attach some semantic meaning to test cases. This is useful for identifying types of failure since we'd be able to differeniate between things that are wrong (fail all test cases) and, for example, things that just don't have correct edge cases handling.

TestCase((randn(), randn()); tolerance=1e-9, tag="random")
TestCase((0, 0); tolerance=1e-9, tag="zero")

The tag (if any) should be included in the JSON output.