testing about c3c HOT 7 CLOSED

Let me address this one pieces at a time. It will be simple to generate a macro that adds all functions marked with the attribute @test (or similar) to an array at compile time. Using this it’s extremely easy to create a unittest framework.

from c3c.

I see problems with @test annotation:

1. It is not minimal. Too many characters to type. Also not visually distinctive enough.
2. You would need to invent a name for this function. This is pain, and useless. I just want to make a test here and now, without any fuss. Think of project with tens of thousand tests. Someone may call a test function. IDE may show them. Name clashes.

I am quite obsessed with testing, and how to remove noise around it.

• Zig almost fits my ideal, except for requiring useless string parameter for every test. Also no way to use my own test runner, so far.
• D is almost ideal, except that you need to run all tests all the time (one could reimplement test runner with breathtakingly complicated metaprogramming tricks).
• Nim allows to do everything with its metaprogramming, but this also means compiler is not aware of tests, which (probably, I am not expert here) limits possible advanced checks.
• The worst language in this regard is Pony. They had show an example of test, and I counted 1 assert, 17 helper lines with a need to create new class. It also required manual registration.

from c3c.

Using compile time reflection, you could also say "run everything starting with test as test". Or "run everything in this directory as test".

Even simpler would actually be a weird idea I call a "run include"

@run_include("foo.sh", $some_param, "-x", $another_param);

Basically you include the stdout from a program that is run during compilation.

from c3c.

Reply to #11 (comment):

Even if everything testable starts with test, you still need to invent the remaining part of the name. Plus the need to type more than absolutely necessary.

Having separate directory for tests is bad idea. Test should be as close to tested code as only possible. Right below the function (or if they get too big in test companion file, with forward reference in the main file).

I once tried to place them all at the end of source file. It was disaster, I couldn't orient among them, had no idea if some feature is tested or not.

I do not understand the "run include" idea.

from c3c.

I used to think that tests should be together with the source code, but have since changed my opinion:

1. It increases the length of reach file.
2. While functions are constantly refactored, tests should not be and might actually be rather ugly as long as they correctly test things. The discrepancy between the two styles is a problem.
3. I find it harder to distinguish working code from its tests. Often tests many times longer than the code they're testing. Having those thrown in between rows code makes code reading harder.
4. It should be easy to compile just the code without the tests.
5. To move something from a library into the language should in my opinion only be done if it's not possible to express it in the language itself. I don't feel that is the case here.

from c3c.

Reply to #11 (comment):

ad 1) Right, heavy testing expands source file size considerably. For this reason I wish to have the "test companion file" feature, dedicated test file sitting just next to the "main" source file. This proximity (in filesystem location and in file name) gives some chance to discover relevant tests. Moving the things into some test/ directory would make them virtually disappear.

Hypothetical, very advanced IDE would be able to hide the tests.

ad 2) Ugliness of tests: test companion files are intended to keep such tests out of sight.

ad 3) Hard to distinguish between the code and tests: that's why I picked syntax:

TEST()
{
   ...
}

I found this to be good visual clue to spot the tests. That's also why I am against tests being "similar" to ordinary functions.

ad 3a) Too big tests should be placed into test companion file.

ad 4) Ability to switch tests/no-tests: that's the task of TESTS define. One change in project configuration file, and tests are enabled/disabled.

ad 5) Yes. Most of the functionality could be implemented by a library. However, for following features compiler support is needed:

• Tests should be able to inspect everything, even private parts of data structures.
• Support for "this-must-not-compile" tests.
• Compiler should wrap tests into implicit #ifdef TESTS ... #endif brackets (or some equivalent). Typing these by hand is nuisance.
• Ability to pass free format attributes to the tests. Without attributes testing subsystem would be quite limited. Using not-a-free format (e.g. key=value pairs) is possible, but clumsy.
• Doing leak detection may or may not require help by the compiler. In C it was not needed, because the language allows to hijack allocations completely.
• The big feature, ability to replace (to mock) specified functions during test duration requires compiler support. The implementation is easy, but cannot be done by library code.
• Nested tests (feature inspired by Zig) seem to require compiler support.
• Ability to place test inside structures or within some code (as in D) would require compiler support.
• Test companion files would need compiler support: full access to "parent" source, ensuring that the companion file doesn't export anything, verifying that all forward declarations of tests are also implemented.

Only a language with extremely advanced metaprogramming capabilities could implement some or all of these features by a library. (For example, I once got confirmation that in Nim one could implement tests inside function body.)

from c3c.

Please file this on c3docs

from c3c.