comp1511unsw / autotest Goto Github PK

A well-defined style guide should be created and enforced to help in the maintenance and/or improvement of autotest.

bundle_autotests.sh generates a single autotest executable which can contain embedded tests for a set of exercises

This could convenient be delivered to a students as a single file to run autotests on their own hardware

But portability to platforms other than Linux has not been checked - and may require further work

• Create a test suite to ensure that autotest functions as expected
• Set up a CI action to run tests on push

The --generate_expected_output option given a model solution generates expected stdout.

It could be generalized to also generate expected_stderr and expected_files

Autotest attempts to run tests when no files were supplied, e.g.:

$ mkdir tmp
$ cat >tmp/tests.txt <<eof
files=a.sh
program=a.sh

1 args=1
2 args=2
3 args=3
eof 
$ autotest.py -a tmp
Test 1 (a.sh 1) - could not be run because these files are missing: a.sh
Test 2 (a.sh 2) - could not be run because these files are missing: a.sh
Test 3 (a.sh 3) - could not be run because these files are missing: a.sh
Test 4 (a.sh 4) - could not be run because these files are missing: a.sh
0 tests passed 0 tests failed  4 tests could not be run

better would be

$ autotest.py -a tmp
autotest.py: no files found to test,  expected these files: a.sh

When a test defines macros as part of compiler_args (e.g to replace a main function) and --generate_expected_output is used, it surrounds the -DFOO with single quotes. This is treated as a syntax error in autotest.

❯ cat autotest/tests.txt
files=test.c
command=./test

compiler_args=-DFOO=1 test.c -o test

1
❯ autotest -g -a autotest
files=test.c
command=./test

compiler_args='-DFOO=1' test.c -o test

1
### generated by: autotest --generate_expected_output - see https://github.com/COMP1511UNSW/autotest
1 expected_stdout='Hello, world!\n'
❯ autotest -g update -a autotest
❯ autotest -a autotest
autotest: /tmp/tmp.ihUpssD2nV/autotest/tests.txt:4: syntax error in assignment

Test specification documentation & source at: https://github.com/COMP1511UNSW/autotest - issues welcome

Typehints serve as a form of documentation, making the code easier to read and maintain. They also enable static type-checking.

​@asyncio.coroutine​ is deprecated and will be removed in python3.11
The new syntax is ​async def​, but this doesn't support yield from.
So run_coroutine will need to be rewritten.

https://docs.python.org/3.11/whatsnew/3.11.html#removed

Proposed approach - add these variables:

test_threads_max default value of number of cores
test_threads_n default value 1
tests_threads_files_copied default value list of supplied files

tests would be run in min(test_threads_max, test_threads_n) threads

if tests_threads_files_copied is non-empty run each test in a new directory,
after copying the specified files to the directory.

https://github.com/COMP1511UNSW/autotest/blob/main/explain_output_differences.py#L379

# ...
        if len(line) > max_line_length_shown:
            line = line[0:max_line_length_shown] + " ..."
        line = line.encode("unicode_escape").decode("ascii")
        if leave_colorization:
            line = line.replace(r"\x1b", "\x1b")
        if leave_tabs:
            line = line.replace(r"\t", "\t")
            line = line.replace(r"\\", "\\")
# ...

I would like to see the line line = line.encode("unicode_escape").decode("ascii") removed ideally,
but @hexDoor has pointed out that this may be necessary to escape things away such as ANSI colour codes.

Unfortunately it escapes normal characters that are of multi-byte width (eg. other language codepoints, emojis, etc.)

Can we make it less restrictive, i.e. normally render displayable characters, and just escape dangerous/non-displayable bytes away?

auto --generate uses repr to output expected output which is robust
but can be difficult to read

multi-line strings would be readable with triple-quotes

also probably more readable if non-ASCII characters replaced with code-points:

e.g. '\u597d' instead of '好'

Non unicode-output produces an error currently.

It is not possible to specify non-unicode output is expected

Non-unicode input also can not be specified directly.

There are workarounds, e.g: test_program | xxd

But it would be preferable to be able to specify non-unicode input/output, e.g

stdin=b'\xff'

a custom clear message should be produced when output is
on the wrong stream

e.g. output is correct but written to stdout instead of stderr or vice-versa

There is no way to specify that for example,
if clang can be found in $PATH compile with it,
and if not and gcc can be found in $PATH compile with it.
and if not ...

This would be allow more flexible use of tests

autotest includes files supplied as part of the autotest in uploads

They would be better omitted.

Standard tests have been included in the pytest test suite but it is currently lacking in real world examples.

Suggestion:
Perhaps extract some autotests from courses such as COMP2041 and COMP1521 for added assurance

• Requirements are currently presumed to be installed
• termcolor has literally been included via copy-pasta
• requirements.txt could be added