datadog / system-tests Goto Github PK

The FlushTraceStats function for the Go implementation should be able to call Flush(), but this function doesn't actually flush the trace stats. This causes problems for the some of the tests which are then forced to call Stop().

#576 will include some stopgaps, but we need to investigate if the Go Tracer should actually be flushing stats in Flush()

/cc @Kyle-Verhoog

Following #642, remove asynchronous mechanism https://github.com/DataDog/system-tests/blob/main/docs/internals/revamp-asynchronous-model.md

As now, its' run.sh + docker-compose

Move it to pytest, and define scenario in python

According to the spec, the remote config AppSec 1 click activation feature should not be active when appsec has been locally configured. We need to remove any DD_APPSEC_ENABLED for those test to work as intended.

... When #628 is merged

useless and flaky

Same thing in the remote config scenario static json generator: https://github.com/DataDog/rc-tracer-client-test-generator/issues/3

[ASM] IP blocking scenario #525

and remove cgroup dirty hack

out-of-the-box, VScode should provide python autocompletion and pylint/black utils

As now, it's done only for python/flask. Look for USE_NEW_CGROUP_GETTER in code

If this env var is set : SYSTEM_TESTS_FORCE_EXECUTE=feature-id-1,feature-id-2,feature-id-3

Then feature-id-1, feature-id-2 and feature-id-3 must be executed, even if they are skipped.

Issue list

TODO

Build time, Exec time

• build time is too slow
  • #562
    • #1331

Hard to write tests

• #1722
• Hard to know what are the scenario useful for a given feature
  • #1088
• Sugars
  • #1094

Debuggability

• #1696
• #1723

CI ecosystem

• #975
• #1475

Done during Q2-Q3 2023

Build time, Exec time

• #1278
• try to understand why nodejs tracer take 15mn to build (and improve).
• Exec time is too slow
  • #1093 and #1169
  • option to not restart containers ? (need to know what is the time saved with that, and be sure we do not introduce flakyness)

Hard to write tests

• Hard to debug test locally // Hard to write a new test from scratch
  • #958
  • #1084
  • #1123 and #1124
  • #1126 and #1128
• Hard to know what are the scenario useful for a given feature
  • #1086
  • #1087
  • ./run.sh SCENARIO --help. -> print the documentation of scenario

PR process in system tests

• Egg-chicken issue between system tests PR and distant PR
  • #1054
  • #913
  • Communicate about the current recipe
• Decorators merge conflicts hell // Maintenance
  • #1089
• PR review bottleneck -> #1481
• PR without description
• Improve system tests CI
  • #1120

Maintain our CI

• maintenance of scenarios in each CI in each repo
  • #1085 and #1133
• #1210

Other

• Execution en local des N variants

  • N variants. Indeed. Hard to tackle, as one run require mostly all your laptop ressources. Maybe a script that run all variant for a given tracer, but it'll be very slow.

• Importance to follow feature matrix

Build step is very slow...

• get data on each image build time
• pre-build them, and use pre-build images

WIP

• cpp
• golang
• java
• nodejs
• php
• python: #1331
• ruby

They are all the same, only path_filter changes

This test is mainly schema, so put it in schemas

Starting cassandra and mongodb etc... is cumbersome when you know for sure your library or your tests do not care about it.
Loic suggested to put those tests in a different scenario.

See #612

Enable pylint on the project, and make it happy with the code

Following #642, revamp all setup as described here https://github.com/DataDog/system-tests/blob/main/docs/internals/revamp-asynchronous-model.md

Lot of entropy in that part ...

As now, test are asynchronous :

class TestCase():
    def test_case(self):
        # some setup code that should not fails
        r = get_weblog()
        # real test logic asyncronously
        interface.weblog.assert_waf_attack(r)

It's painful because

• we have to hack pytest to rewrite failure (test_case never fails)
• if test_case fails, we have to hack pytest to give the output
• if test_case fails, but is flagged as xfail, the error is silent
• and there are lot of hack to decorate asynchronous validation

It may be possible to change this to stick to a pure pytest code style :

class TestCase():
    def setup_case(self):
        self.r = get_weblog()

    def test_case(self):
        interface.weblog.assert_waf_attack(self.r)  # now synchronous

The test executor will look like :

collect
execute all setup_XXXX (need to be writted)
wait interfaces
execute test

1. no more asynchronous test
2. no more dirty hack to collect metadata

The only draw back is that object set in setup is no more sticked to the test logic. But I think it's reasonnable.

Actually, test set is declared in run.sh :

elif [ $SYSTEMTESTS_SCENARIO = "REMOTE_CONFIG_MOCKED_BACKEND_ASM_DD" ]; then
    export RUNNER_ARGS="scenarios/remote_config/test_remote_configuration.py::Test_RemoteConfigurationUpdateSequenceASMDD"

The idea is to move this set inside python files :

@scenarios("REMOTE_CONFIG_MOCKED_BACKEND_ASM_DD")
class Test_RemoteConfigurationUpdateSequenceASMDD():
    ...

if a test class does not have a @scenarios decorator, then it's included in default scenario.

The decorator can be added on both class on test method.

Needs to discuss with agent team

Plateform: Linux Ubuntu 22.04 (datadog preseed image), x64, docker compose compatibility.

If I execute ./build.sh twice, it fails with a access denied on pycache files. I checked and the files are all owned by root. (I did not run the tests outside of docker)
Workaround is to run py3clean . between every run.

See #622

The CI taking 30 minutes to complete does not spark joy. I've seen that the scenarios are executed 1 by 1 where they could be parallelized. In a local dev env it wouldn't make sense but in CI I'm guessing it does.

• build stats for how often the build step fails
• Add a numerical parameter to build.sh, it will retries to build as many time (default 1)

https://github.com/DataDog/system-tests-dashboard/actions/runs/3425508634/jobs/5706373742