Comments (2)
Hi @lrperlmu, thanks for the detailed request in #22! If you'd like, you could elaborate on your expected behavior here as well.
Since filing that issue we have added a partial credit feature, which might be a possible workaround if you wanted to merge multiple subtests into one test with partial credit: https://github.com/gradescope/gradescope-utils/blob/master/gradescope_utils/autograder_utils/decorators.py#L103
I haven't dug into this much but I'm not sure how the unittest framework represents subtests internally, whether we get separate passing or failing test results for each subtest. From the documentation it does look like it would do so, and hopefully it would be reasonable to add such a feature.
It looks like we'd need to add an addSubTest method to the JSONTestResult that would do the right thing and produce multiple outputs for each subtest, or at least store the appropriate data on the TestResult object so that the JSONTestRunner can produce the desired output at the end.
This isn't something that is on our current roadmap, but if anyone is able to submit a PR then we can consider adding this feature!
from gradescope-utils.
@ibrahima thanks for your reply! Pasting the relevant info from the dupicate #22 here:
Is your feature request related to a problem? Please describe.
gradescope-utils does not produce output for subtests
(for definition of subtest, see https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests)Describe the solution you'd like
Output a separate json object for each subtest, so that each subtest appears in the json output just like each test.
To elaborate, I have some test code that needs to run sequentially, i.e. it would not work to make separate tests for it. But I want to make several assertions in the middle of that code block, and I want the rest of the test code to continue executing regardless of the outcome of each assertion. Regular tests exit the first time an assertion fails and the code after the first failed assertion does not run. Subtests would support what I'm asking for because each assertion can be in its own subtest and the test keeps running if one subtest fails.
I don't have the capacity right now to make a PR, but it's on my radar in case my availability opens up. Good to know this project is open to outside contributions :)
from gradescope-utils.
Related Issues (15)
- Visibility decorator fails to add the visibility attribute in results.json HOT 2
- Error, not able to implement @partial_credit decorator HOT 3
- Error no such file or directory HOT 1
- buffer argument not respected in JSONTestRunner HOT 3
- Help prevent reverse engineering of tests HOT 3
- Better Error Messages (easy version)
- Human-readable name of test case does not appear in results.json HOT 1
- Support for `unittest` subtests HOT 1
- Setting buffer to False in JSONTestRunner throws an error HOT 1
- Add a License HOT 2
- Setting a default visibility HOT 1
- Leaderboard value in in results.json
- Score is displayed even if weight is None HOT 1
- Allow for local testing HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gradescope-utils.