gradescope / gradescope-utils Goto Github PK

It would be awesome if y'all would be willing to put a license on this code. Preferably MIT or Apache, but any clarity would be helpful.

One of the issues with Python testing, is that it's easy for students to read directories and figure out what files to potentially read. Although one could argue that a student who has figured this out probably should deserve to pass the tests. However, this kind of information then spreads.

There's many ways we mitigate this but all of them involve a lot of jumping through hoops (dynamically renaming the tests, removing the test files after they are loaded but before the students code is loaded), etc (which only work if in a sandboxed environment).

A simple and effective way to prevent this would be to allow the initialization of JSONTestRunner to take a parameter(s) to limit how much stdout and stderr is written out in the results. That's the most likely way for students to get information back. If both stdout and stderr are truncated to a specified maximum amount, then it allows both the framework to print relevant messages, but also keeps students from reading the test files.

Another option would be allow tests cases to write to a logfile that is then sent back to stdout first and anything else truncated.

https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests

Makes it easier to script parametrized tests.

I am getting this error when I use partial_credit decorator

@partial_credit(10)\nNameError: name 'partial_credit' is not defined\n\n"

I use it like this:
@partial_credit(10)
def test_exercise_1(self, set_score=None):
"""Exercise 1 answers"""
flag = 0
...
if(flag == 0):
set_score((8-flag)*/8)
return 1

Where am I going wrong?

Is your feature request related to a problem? Please describe.

Currently there is no way to have a test without an assigned score. A weight of 0 is assigned by default.

Describe the solution you'd like

No score should be assigned if the weight is not provided.

This occurs when it is used alongside the weight decorator.

When a student submits code that breaks a test but it's not due to an assertion error, the feedback is either confusing or misleading.

For example, if a student has a simple KeyError mistake

def my_code(dataframe):
   return dataframe['SalesPerson']  # assume SalesPerson is not a valid key

The test will fail but with the message:

"Test Failed: 'SalesPerson'

That really offers no help

Describe the solution you'd like
An easy solution is to provide the class of the exception or error:

"Test Failed: <class 'KeyError'> 'SalesPerson'\n"

The fix is simply in json_test_runner.py (buildResult) line 77

if err:
    if err[0] is AssertionError:
        output += "Test Failed: {0}\n".format(err[1])
    else:
        output += "Test Failed: {0} {1}\n".format(err[0], err[1])
# note that isinstance will NOT work here

Describe alternatives you've considered
A similar pull request
#18

However with that solution you get the following:

"Test Failed: 'SalesPerson'\nhashtable_class_helper.pxi:pandas._libs.hashtable.PyObjectHashTable.get_item line 1627: \n"

In this context, the backtrace is not helpful to the student. Perhaps a config option?

Also, in that pull request it should not use isinstance() to test the error class.
(tested in python3.6).

Additional context

Hi, I tried to run python setup.sh and I got this :
Could not open requirements file: [Errno 2] No such file or directory: '/autograder/source/requirements.txt'

any ideas?

I realize this could be submitted as a pull request.

FILE: gradescope-utils/gradescope_utils/autograder_utils/files.py:
ISSUE: SUBMISSION_BASE should be passed in as an default argument to allow local testing.

SUBMISSION_BASE = '/autograder/submission'
def check_submitted_files(paths, base=SUBMISSION_BASE):
...
    target_path = os.path.join(base, path)
...

Describe the bug
When using the JSONTestRunner class to obtain the test results in a JSON format, it throws an error when the input parameter buffer=False if any of the test cases fails.

To Reproduce
We create a new testcase that fails.

import unittest
from gradescope_utils.autograder_utils.decorators import weight, visibility

class TestAutograder(unittest.TestCase):
    @weight(1)
    @visibility("visible")
    def test_auto(self):
        """
        A test that throws an error
        """
        self.assertTrue(False)

Running the above with the JSONTestRunner provided in gradescope_utils produces the following error:

TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str'

Expected behavior
As mentioned in line 132 of the file json_test_runner.py, we Set buffer to True to include test output in JSON. So ideally, when it is False, there should be an empty output for the test.

Potential causes/fix
When buffer is set to False, the getOutput method in JSONTestResult returns None. As a result, in the method buildResult, the variable output gets set to None as well. In line 77, if there is an error, the method tries to append the message Test Failed to the output, but since output is None, it is unable to execute the line output += "Test Failed: {0}\n".format(err[1]) and hence throws an error. To fix this, a check needs to be put on output (to see if it is not None) before appending the Test failed message onto it.

I added the leaderboard decorator to a function and used your sample code. But it always sets in to none

The code:

@leaderboard("high score1")
def test_leaderboard(self):
    """Sets a leaderboard value"""
    set_leaderboard_value(random.randint(0, 10))

and the output is

"leaderboard": [
    {
        "name": "time",
        "value": null,
        "order": "asc"
    },
    {
        "name": "high score1",
        "value": null,
        "order": "desc"
    },

Description
Naming test cases with a new-line character on the first line causes the name of the function to be placed in the results.json file instead of the human-readable name in the docstring of the testcase.

To Reproduce
Create a testcase and name it with a docstring that has a new-line character on the first line. This file is called tests.py.

import unittest
from gradescope_utils.autograder_utils.decorators import weight, number

class TestAutograder(unittest.TestCase):
    @weight(1.0)
    @number("1")
    def test_prob1(self):
        """
        A human-readable name.
        """
        pass

Running the above with the JSONTestRunner provided in gradescope_utils produces the following results.json:

{
    "tests": [
        {
            "name": "test_prob1 (tests.TestAutograder)",
            "score": 1.0,
            "max_score": 1.0,
            "number": "1"
        },
    ],
    "leaderboard": [],
    "execution_time": "0.00",
    "score": 10.0
}

Expected behavior
Should have produced the following results.json file:

{
    "tests": [
        {
            "name": "A human-readable name.",
            "score": 1.0,
            "max_score": 1.0,
            "number": "1"
        },
    ],
    "leaderboard": [],
    "execution_time": "0.00",
    "score": 10.0
}

Additional context
Removing the new-line character seems to fix the issue. I.e. the following produces the expected result.

import unittest
from gradescope_utils.autograder_utils.decorators import weight, number

class TestAutograder(unittest.TestCase):
    @weight(1.0)
    @number("1")
    def test_prob1(self):
        """A human-readable name."""
        pass

This also seems to only be the case inside of Gradescope's docker containers used for actually grading student submissions. This did not happen when I ran the code on my macbook.

Would there be a way to set a default visibility other than visible for all tests?

Right now I didn't find any, and adding all the @visibility("after_published") decorators is quite annoying.

Thank you very much !

Describe the bug
Raising an Assertion Error in unittest's setUpClass method causes exception in jsontestrunner's getLeaderboardData

To Reproduce

import unittest
from gradescope_utils.autograder_utils.decorators import (
    weight,
    number,
    leaderboard,
)

class TestAutograder(unittest.TestCase):
    @classmethod
    def setUpClass(cls) -> None:
        raise AssertionError("compile for all tests failed")
        return super().setUpClass()
    
    @number(1.1)
    @weight(1)
    def test_01(self):
        "Test 01"
        pass

Expected behavior
Raised error would have populated results.json for all test cases of that class.

Instead, received following error:

The autograder produced the following output:

Traceback (most recent call last):
  File "/usr/lib/python3.10/unittest/suite.py", line 166, in _handleClassSetUp
    setUpClass()
  File "/autograder/source/tests/test.py", line 14, in setUpClass
    raise AssertionError("compile for all tests failed")
AssertionError: compile for all tests failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/autograder/source/run_tests.py", line 8, in <module>
    JSONTestRunner(visibility='visible',stdout_visibility='visible').run(suite)
  File "/usr/local/lib/python3.10/dist-packages/gradescope_utils/autograder_utils/json_test_runner.py", line 196, in run
    test(result)
  File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.10/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.10/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.10/unittest/suite.py", line 114, in run
    self._handleClassSetUp(test, result)
  File "/usr/lib/python3.10/unittest/suite.py", line 176, in _handleClassSetUp
    self._createClassOrModuleLevelException(result, e,
  File "/usr/lib/python3.10/unittest/suite.py", line 236, in _createClassOrModuleLevelException
    self._addClassOrModuleLevelException(result, exc, errorName, info)
  File "/usr/lib/python3.10/unittest/suite.py", line 246, in _addClassOrModuleLevelException
    result.addError(error, sys.exc_info())
  File "/usr/local/lib/python3.10/dist-packages/gradescope_utils/autograder_utils/json_test_runner.py", line 136, in addError
    self.processResult(test, err)
  File "/usr/local/lib/python3.10/dist-packages/gradescope_utils/autograder_utils/json_test_runner.py", line 123, in processResult
    if self.getLeaderboardData(test)[0]:
  File "/usr/local/lib/python3.10/dist-packages/gradescope_utils/autograder_utils/json_test_runner.py", line 51, in getLeaderboardData
    column_name = getattr(getattr(test, test._testMethodName), '__leaderboard_column__', None)
AttributeError: '_ErrorHolder' object has no attribute '_testMethodName'

Is your feature request related to a problem? Please describe.
gradescope-utils does not produce output for subtests
(for definition of subtest, see https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests)

Describe the solution you'd like
Output a separate json object for each subtest, so that each subtest appears in the json output just like each test.

Describe alternatives you've considered

• I could avoid using subtests -- I lose some of the power of the unittest framework
• Construct my json output by hand instead of using gradescope-utils

Additional context
gradescope-utils is a nice, useful package, thanks for developing it :)

In json_test_runner.py, the buffer argument is not being passed to the TestResult class because it is not an argument in JSONTestResult and therefore not passed to the super().__init__. This means that the buffer value for JSONTestRunner is not respected.

The correct change is to add buffer to the arguments of JSONTestResult.__init__ and pass it to TestResult in the super().__init__ function