Tasks are flaky because they start before duts are up

I just copied enochecker_test test_getflag[0] from the readme and it ran all tests.
It was supposed to run only the first testflag. In particular enochecker_test test_getflag[0] and enochecker_test rant the same amount of tests.

Can we do that @ldruschk ? Certain services and checkers still have no .env file that would set a COMPOSE_PROJECT_NAME

See enowars/enowars-service-example#12

Can we add -vvv to the pytest.main call (honest question, I have never used pytest.main before)?

Right now I can't see both sides' values of an assertion, and since the sample service tests pass locally I have no clue what could be going wrong in github actions.

I do think the assert rewriting worked, because it clearly shows the offending line:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

round_id = 19, havoc_id = 3, service_address = '10.1.0.216'
checker_url = 'http://localhost:8000', unique_variant_index = 3
expected_result = <CheckerTaskResult.CHECKER_TASK_RESULT_INTERNAL_ERROR: 'INTERNAL_ERROR'>

    def _test_havoc(
        round_id,
        havoc_id,
        service_address,
        checker_url,
        unique_variant_index=None,
        expected_result=CheckerTaskResult.CHECKER_TASK_RESULT_OK,
    ):
        if unique_variant_index is None:
            unique_variant_index = havoc_id
        request_message = _create_request_message(
            "havoc",
            round_id,
            havoc_id,
            service_address,
            unique_variant_index=unique_variant_index,
        )
        msg = _jsonify_request_message(request_message)
        r = requests.post(
            f"{checker_url}", data=msg, headers={"content-type": "application/json"}
        )
        assert r.status_code == 200
        result_message: CheckerResultMessage = jsons.loads(
            r.content, CheckerResultMessage, key_transformer=jsons.KEY_TRANSFORMER_SNAKECASE
        )
>       assert CheckerTaskResult(result_message.result) == expected_result
E       AssertionError

/opt/hostedtoolcache/Python/3.9.4/x64/lib/python3.9/site-packages/enochecker_test/base.py:302: AssertionError

add tests to the package itself. Should be pretty easy, since all we need is mocking up some HTTP requests/responses

debugging (as in "with a debugger") is hard with all tests firing

this is NOT according to spec, and is missing bugs

Basically putflag A -> putflag B -> getflag A where A and B only differ in the flag and task_chain_id

at least include a random prefix to allow rerunning without needing to clean up the database

https://docs.pytest.org/en/stable/reference.html#pytest-register-assert-rewrite

by default pytest only rewrites asserts for tests discovered through the built-in test discovery mechanisms

It catches all errors, also if e.g. the Request times out:

It shouldn't

to allow a quick assessment of which code path is wrong