coreruleset / go-ftw Goto Github PK

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

• chore(deps): update github/codeql-action action to v3.25.11

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Instead of using one config per web server, move under the logname the corresponding logfile, so we can use one config file for everything.

httpbin usually returns the request data in the response exactly as sent. At least in case of Unicode that is not true, unfortunately. Example:

curl -X POST "https://httpbin.org/anything" --json '{"über": "bort"}'
{
  "args": {},
  "data": "{\"\u00fcber\": \"bort\"}",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "application/json",
    "Content-Length": "17",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "curl/8.1.2",
    "X-Amzn-Trace-Id": "Root=1-6548859d-09da901d70e79647362f7b83"
  },
  "json": {
    "\u00fcber": "bort"
  },
  "method": "POST",
  "origin": "89.133.9.17",
  "url": "https://httpbin.org/anything"
}

The Unicode sequence is returned as a backslash escape sequence instead. That is semantically correct but bad for testing. We need to find a way around this.

There was a request enhancement form airween in the coreruleset channel for:

fzipitria is there any opportunity in ftw-go that we could add an expression in the request value (eg, a payload, header value), eg:

    data: eval(python -c "print('foo=%3d' + '+' * 34560)")

While evaluating any language would be awesome, for the sake of simplicity a quick win will be to add go templating.

So for example, we could write something like:

    data: foo=%3d{{ range seq 1 34560 }}+{{ end }}

With similar results.

Good thing: Go text templates are extremely useful, and have lots of available functions on top of it.

go-ftw and ftw before it both seem to assume the server being tested is at localhost:80.

The python ftw seems to have options to change that, e.g. --destaddr=domain.com --port 443 --protocol https, at least when run with test/test_default.py (I guess that was its equivalent of cloud mode).

So does go-ftw -- see applyInputOverride and config_test.go -- but the feature appears to be poorly documented,
with no example in README.md.

This makes it hard to test WAFs running on hosting providers that rely on the Host header.

Overriding Host in applyInputOverride where it overrides DestAddr seems to fix this, draft PR coming.

Currently, go-ftw only provides a way to send requests. Because httpbin reflects the request body in the response body, we can use a request specification to specify the expected response. However, from the perspective of a test author that is not the correct way to describe the behaviour. Instead, there should be a way to specify both the request and the expected response.

We sometimes need to duplicate tests in order to run the same test against multiple URIs (especially in rule exclusion tests). It would be great if we could simply specify a list of URIs and reuse the test data for each of them. I think we could add a new field called uris that takes a list, so we'd have both uri (single URI, backwards compatibility) and uris (multiple URIs).

Tests for rules in phase 5 are subject to a rare race condition, as exhibited in the following log for tests 980170-2 and 980170-3 (line breaks between rules added for readability):

[Sat Mar 18 16:47:21.474075 2023] [security2:error] [pid 193:tid 140523746522880] [client 172.18.0.1:39150] [client 172.18.0.1] ModSecurity: Warning. Pattern match "^.*$" at REQUEST_HEADERS:X-CRS-Test. [file "/etc/modsecurity.d/owasp-crs/crs-setup.conf"] [line "737"] [id "999999"] [msg "04709bb3-6b2b-477d-aa14-76a89363b2dd"] [tag "modsecurity"] [hostname "localhost"] [uri "/status/200"] [unique_id "ZBXrGVXqtKqlnATxdUEg7QAAANg"]
[Sat Mar 18 16:47:21.476378 2023] [security2:error] [pid 193:tid 140524082149120] [client 172.18.0.1:39164] [client 172.18.0.1] ModSecurity: Warning. Pattern match "(?:^([\\\\d.]+|\\\\[[\\\\da-f:]+\\\\]|[\\\\da-f:]+)(:[\\\\d]+)?$)" at REQUEST_HEADERS:Host. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "761"] [id "920350"] [msg "Host header is a numeric IP address"] [data "127.0.0.1"] [severity "WARNING"] [ver "OWASP_CRS/4.0.0-rc1"] [tag "modsecurity"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/210/272"] [tag "PCI/6.5.10"] [hostname "127.0.0.1"] [uri "/"] [unique_id "ZBXrGVXqtKqlnATxdUEg7gAAAMQ"]
[Sat Mar 18 16:47:21.480771 2023] [security2:error] [pid 193:tid 140524098930432] [client 172.18.0.1:39172] [client 172.18.0.1] ModSecurity: Warning. Pattern match "^.*$" at REQUEST_HEADERS:X-CRS-Test. [file "/etc/modsecurity.d/owasp-crs/crs-setup.conf"] [line "737"] [id "999999"] [msg "04709bb3-6b2b-477d-aa14-76a89363b2dd"] [tag "modsecurity"] [hostname "localhost"] [uri "/status/200"] [unique_id "ZBXrGVXqtKqlnATxdUEg7wAAAMM"]


[Sat Mar 18 16:47:21.483110 2023] [security2:error] [pid 35:tid 140523931117312] [client 172.18.0.1:39180] [client 172.18.0.1] ModSecurity: Warning. Pattern match "^.*$" at REQUEST_HEADERS:X-CRS-Test. [file "/etc/modsecurity.d/owasp-crs/crs-setup.conf"] [line "737"] [id "999999"] [msg "c50b1e8e-e8a5-46d1-afc8-bf82368390f4"] [tag "modsecurity"] [hostname "localhost"] [uri "/status/200"] [unique_id "ZBXrGcs3WOXFKxhOjQ9D-AAAAE0"]
[Sat Mar 18 16:47:21.483333 2023] [security2:error] [pid 193:tid 140524082149120] [client 172.18.0.1:39164] [client 172.18.0.1] ModSecurity: Warning. Unconditional match in SecAction. [file "/etc/modsecurity.d/owasp-crs/rules/RESPONSE-980-CORRELATION.conf"] [line "96"] [id "980170"] [msg "Anomaly Scores: (Inbound Scores: blocking=3, detection=3, per_pl=3-0-0-0, threshold=5) - (Outbound Scores: blocking=0, detection=0, per_pl=0-0-0-0, threshold=4) - (SQLI=0, XSS=0, RFI=0, LFI=0, RCE=0, PHPI=0, HTTP=0, SESS=0, COMBINED_SCORE=3)"] [ver "OWASP_CRS/4.0.0-rc1"] [tag "modsecurity"] [tag "reporting"] [hostname "127.0.0.1"] [uri "/"] [unique_id "ZBXrGVXqtKqlnATxdUEg7gAAAMQ"]
[Sat Mar 18 16:47:21.489998 2023] [security2:error] [pid 193:tid 140523830429440] [client 172.18.0.1:39206] [client 172.18.0.1] ModSecurity: Warning. Pattern match "^.*$" at REQUEST_HEADERS:X-CRS-Test. [file "/etc/modsecurity.d/owasp-crs/crs-setup.conf"] [line "737"] [id "999999"] [msg "c50b1e8e-e8a5-46d1-afc8-bf82368390f4"] [tag "modsecurity"] [hostname "localhost"] [uri "/status/200"] [unique_id "ZBXrGVXqtKqlnATxdUEg8AAAANM"]

Rule 920350 is triggered as expected for the first test but rule 980170 is not. However, looking at the unique_id field it becomes clear that rule 980170 did in fact trigger but was reported as part of the second rule (the msg field of rule 999999 denotes start and stop ID of a rule).

The logic for finding triggered rules in the log is:

1. send test request to mark start
2. run test
3. send test request to mark end
4. while end marker not in log
   1. send test request to flush log
5. check test against lines between start and end markers

Apparently, the log entry for rule 980170 may under some circumstances be written to the log much later and, which is worse, independently of other log entries of the same phase (both 999999 and 980170 run in phase 5), otherwise it would have shown up before the start marker for test 980170-3.

Currently, results are only printed to the screen but it would help in post-processing to have access to them in JSON. There is a TestStats which could be outputed, though for my use case it would need some extension as I'd like not only the status of non-successful items as current, but all tests and include the time taken for each test. The success/failure would be useful in verification tasks, while the timing (and not the verification) would be useful to compare runs with a WAF enabled vs disabled to check its overhead with FTW's real-world test cases (WAF disabled would fail most test cases but isn't relevant for that performance test so no problem).

Hi,

Currently, such behavior is implemented for the actual tests using the config field testoverride.input.headers of type map. Per my tests, these custom request headers are not used in the logmarker requests.

Considering the same configurability for the logmarker requests and extending its config API with this feature, makes it more flexible and avoids potential unnecessary logic customization or configuration on the custom WAF servers side.

If I'm missing any point here, I'd appreciate your clarifying it.

Thanks.

.ftw.yaml

logfile: 
logmarkerheadername: 
mode: "cloud"
testoverride:
  input:
    dest_addr: "testphp.vulnweb.com"
    port: 80
    protocol: "http"

go-ftw run --cloud --debug

🛠️  Starting tests!
🚀 Running go-ftw!
👉 executing tests in file 911100.yaml
        running 911100-1: 8:45AM DBG ftw/http: sending data:
GET / HTTP/1.1
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Connection: close
Host: localhost
User-Agent: OWASP CRS test agent



8:45AM DBG ftw/run: error receiving response: unexpected EOF

Error: failed sending request to destination &{DestAddr:testphp.vulnweb.com Port:80 Protocol:http}: unexpected EOF

Hi go-ftw Team,

I am working on updating some documentation, and I found that some of the docs are missing in the go-ftw. (e.g., the format of the YAML test file)

And I found it in the ftw repo: https://github.com/coreruleset/ftw/tree/main/docs

Should we migrate the documentation to go-ftw?

We want to show as little as possible when running in quiet mode.

The runner.Run method keeps changing the interface and it makes it harder to track.

Switching to a functional options pattern might make it easier to read and extend.

This package should be easy to use as a library.

We should document all the steps needed to use it in a standard and easy way.

To have better accuracy while performing tests that read logs, instead of relying on timestamps we could use the same concepts that were introduced into ftw with marks.

This is a placeholder for documenting what needs to be done.

As a security rules tester, I would like to have tests results coverage based on the tests and on the rules I am testing.

Requirements

• add an optional --coverage parameter, with a required additional --rules directory parameter that will contain rules.
• rules will be read from *.conf files in that directory
• a map will contain the rule identifiers as key
• for every test that starts with the ID of the rule, we add a counter or similar to the map
• after running all tests, we can display a summary of the map based on keys empty/with content.

Output

• summary of the tests run plus the coverage percentage based on the map

Hi!
I'm debugging CRS tests 920280-1 and 920290-1 that are failing against Coraza. It seems that there is a conflict between these specific tests (They are trying to send a request without a Host header or with an empty one) and the feature added in feat: set "Host" header from dest_addr override.
When a destination address override is provided, the feature replaces the empty host field with the destination address making the request a valid one.

It is not happening against Apache-Modsec (and CRS CI) because I think that destination overriding is not used there. A small ftw config like the following reproduces the issue:

logfile: '/Users/Repo/coreruleset/tests/logs/modsec2-apache/error.log'
testoverride:
  input:
    dest_addr: localhost

I may be missing something about #63, but I'm wondering why the host header is replaced only when == ""? I was expecting that this feature was mainly intended for replacing all the Host: "localhost" spread across the rules

I gave it a little go (just replacing == with !=, see this branch): 920280-1 and 920290-1 would be fixed, but ["920270-4" "920350-1" "920350-3" "920350-4" "920350-5" "920350-6" "931130-9" "931130-11" "931130-15" "980170-2"] start to fail.
It happens because, for example in test 920270-4, a manipulated host is replaced by a legit one (localhost%00 -> localhost).

Thanks for any input about it!

Sometimes users run the tool and fails because comparing logs is using a bad TZ and those will fail.

• provide a test to see if everything works
• check at the difference between logs in the docker vs. base host
• print the difference and warn users

Right now we can override the ignore list for the tests via .ftw.yaml file.

I would like to have the same feature for including tests. Right now we can only do this in the command line with the argument -i or --include.
Example: ftw run -d tests/ -i "920290-4$"

Sometimes we have tests that work only for some specific platform (like apache). But we want to run all tests in many platform, to see that everything keeps working.

Adding an excludelist at the config would allow us to do exactly that.

Don't know if this is feasible, but let's see. We found out that we might want to skip (or don't run tests) based on attributes of the rule we are testing.

An example is that we don't run tests that match rules that have paranoia level bigger than PL1.

There is no simple solution now, as we are not parsing rules files now. Once we have a parser it might be easy to implement.

Initial implementation specifically went around standard interfaces like Client or RoundTripper.

Now that the project is working properly, we should revisit using standard interfaces to enable extensions from other users.

• Use net/http#Client in http/client.go
• Review RoundTripper and move round trip times there

In the process to get the status page working, we need to overwrite the test result to only check for the http status code returned.

👉 That way, we don't need to check for logs that might not be accessible in cloud vendors, or maybe the information there is partial or different.

Relying only on http status code will give us an approximate result. Results of the test should be:

• If the test has log_contains, and we received 403 as response, test is Passed.
• If the test has log_contains, and we received 200 as response, test is Failed.
• If the test has no_log_contains, and we received 403 as response, test is Failed.
• If the test has no_log_contains, and we received 200 as response, test is Passed.
• If the test has status, and the status in the response is in that list, test is Passed. Otherwise, test has failed.

Also, we should be able to override test dest_addr and port, as this is going to hit remote resources in most cases (a tunnel can easily be setup using netcat, but it is better if we just override those settings).

In other tools we started using testify/suite that simplifies handling tests, setup and teardowns.

Requirements

• create suites for every test package
• move and simplify test creation

Output

• all tests changed to testify/suite
• all tests are passing

We found a bug while checking results in nginx.

Looks like it is caused by the date format while reading logs.

Example:

❯ ./dist/ftw_linux_amd64/ftw run --cfg .ftw-nginx.yaml -d tests --id 944210-6 --debug
11:58AM INF Using config file: .ftw-nginx.yaml

🛠️  Starting tests!
🚀 Running!👉 executing tests in file 944210.yaml
	running 944210-6: 11:58AM DBG ftw/run: sending request
11:58AM TRC ftw/http: this is data: "<?xml version=\"1.0\"?><xml><element rO0ABQ=\"attribute_value\">element_value</element></xml>", of len 89
11:58AM DBG ftw/http: adding standard headers
11:58AM DBG ftw/http: sending data:
POST / HTTP/1.0
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Encoding: gzip,deflate
Accept-Language: en-us,en;q=0.5
Connection: close
Content-Length: 89
Content-Type: application/xml
Host: localhost
User-Agent: ModSecurity CRS 3 Tests

<?xml version="1.0"?><xml><element rO0ABQ="attribute_value">element_value</element></xml>
11:58AM DBG ftw/http: sending data
11:58AM TRC ftw/run: send took 102839
11:58AM DBG ftw/check: getting response
11:58AM DBG ftw/http: receiving data
11:58AM TRC ftw/http: received data - "HTTP/1.1 405 Not Allowed\r\nServer: nginx/1.17.9\r\nDate: Tue, 16 Mar 2021 11:58:47 GMT\r\nContent-Type: text/html\r\nContent-Length: 157\r\nConnection: close\r\n\r\n<html>\r\n<head><title>405 Not Allowed</title></head>\r\n<body>\r\n<center><h1>405 Not Allowed</h1></center>\r\n<hr><center>nginx/1.17.9</center>\r\n</body>\r\n</html>\r\n"
11:58AM TRC ftw/run: response took 5925927
11:58AM DBG ftw/check: expected error? -> false, and error is nil
11:58AM DBG ftw/check: status 405, expected []
11:58AM DBG ftw/check: log contains? -> 
11:58AM DBG ftw/waflog: Looking at file ../coreruleset/tests/logs/modsec3-nginx/nginx/error.log, between 2021-03-16 08:58:47.81739945 -0300 -03 m=+0.122610188 and 2021-03-16 08:58:47.823293082 -0300 -03 m=+0.128503823
11:58AM DBG ftw/waflog: got 0 lines
11:58AM DBG ftw/waflog: Looking at file ../coreruleset/tests/logs/modsec3-nginx/nginx/error.log, between 2021-03-16 08:58:47.81739945 -0300 -03 m=+0.122610188 and 2021-03-16 08:58:47.823293082 -0300 -03 m=+0.128503823
11:58AM DBG ftw/waflog: got 0 lines
11:58AM DBG ftw/check: checking if log does not contains
✔ passed in 5.893635ms
➕ run 1 total tests in 5.893635ms
⏭  skept 2360 tests
🎉 All tests successful!

This test should fail.

While running the latest rc, I've stumbled on this:

        running 932150-4: 12:18PM INF ftw/http: cannot encode data to: "932150-2=dont match commands that are not at start;tar -xzf /var/www/exfiltrate.tar.gz /etc"
12:18PM FTL ftw/http: fatal error building request: invalid semicolon separator in query

Looking at the test:

   - test_title: 932150-4
     stages:
       - stage:
           input:
             dest_addr: "127.0.0.1"
             method: "POST"
             port: 80
             headers:
               User-Agent: OWASP ModSecurity Core Rule Set
               Host: "localhost"
               Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;   q=0.5
             data: '932150-2=dont match commands that are not at start;tar -xzf /var/www/exfiltrate.tar.gz /etc'
             protocol: "http"
           output:
             no_log_contains: "id \"932150\""

Seems like this has changed semantics now? Need to dive a bit because previous 0.3 version (go 1.16 works).

Sometimes when running in a pipeline, tests are a massive output that barely makes sense for the ones not failing.

Maybe extend the quiet flag to show failures only? On adding a new flag.

That would make results more succinct.

The function GetTestsFromFiles inf files.go takes several seconds to read all YAML test files into memory. While the number of tests has grown in recent weeks, the slowdown still feels significant.

Hello,
I was wondering if it could be helpful to provide to the testoverride lists (ignore, forcefail and forcepass) a regex syntax like the -i/-e commands. The use case comes from a willingness to exclude all the tests of some specific rules (or even a specific file?) and, currently, the only way I see is listing individually all the tests in a quite verbose way.

Before:

...
testoverride:
  ignore:
  '932140-1': 'Currently excluding rule 932140'
  '932140-2': 'Currently excluding rule 932140'
  '932140-4': 'Currently excluding rule 932140'
  '932140-5': 'Currently excluding rule 932140'
  '932140-6': 'Currently excluding rule 932140'
  '932140-7': 'Currently excluding rule 932140'
  '932140-8': 'Currently excluding rule 932140'
  '932140-9': 'Currently excluding rule 932140'
  '932140-10': 'Currently excluding rule 932140'
  '932140-11': 'Currently excluding rule 932140'
  ...

After:

...
testoverride:
  ignore:
  '932140-*': 'Currently excluding rule 932140'

My concern is about the implementation: the check would become not just looking for the a value inside the map (e.g. ignore), but iterating over a list of regex looking for a match. Do you think it's worth the overhead?

Thanks for the feedbacks!

Viper has been a pain for some of the config loads that I've been trying to do.

Let's give koanf a try and see if we can do better.

We just have just moved our test suite from the original python ftw to this version, but one of the issues we have is that the response_contains key in the output section of a stage is not checked if the response status matches. This worked in the python based ftw.

Example:

    test_title: example
    stages:
      -
        stage:
          input:
            method: POST
            uri: /wp-admin/admin-ajax.php?action=<script>alert(1)</script>
            headers:
              Host: example.com
            data: '...'
          output:
            status: [403]
            response_contains: "rule 1234"

Observed behaviour:

The test passes, even if it's blocked by a different rule than the one specified.

Expected behaviour:

The test should fail.

Github supports adding a summary by using echo "{markdown content}" >> $GITHUB_STEP_SUMMARY

It is the perfect place for printing the summary.

For values that need a default other than the zero value (e.g., 20), the configuration currently isn't initialized properly in tests, as the defaults would be set by cobra. Instead of adding the default values to each configuration that is being created in the tests, we should have a means to use a default configuration, e.g., by having a function NewConfig.

If /status/200 is not existing and if apache is set to loglevel info, we get the following error message(s) in the log:

[2022-11-12 23:08:18.012572] [-:error] 127.0.0.1:36126 Y3AZUo3Gja4gB-tPE9uasgAAAA4 [client 127.0.0.1] ModSecurity: Warning. Unconditional match in SecAction. [file "/apache/conf/httpd.conf_pod_2022-11-12_22:23"] [line "265"] [id "999999"] [msg "x-crs-test 6c7427f7-6638-46fd-8ad5-aa66961d3ffe"] [hostname "localhost"] [uri "/status/200"] [unique_id "Y3AZUo3Gja4gB-tPE9uasgAAAA4"]
[2022-11-12 23:08:18.013007] [core:info] 127.0.0.1:36126 Y3AZUo3Gja4gB-tPE9uasgAAAA4 AH00128: File does not exist: /apache/htdocs/status/200

The 404 message prevents go-ftw from identifying and parsing the marker line correctly.

The following body represents a chunk in a Transfer-Encoding: chunked request (content length, content, remaining chunks):

data: |
  7
  foo=bar
  0

While the contents of the lines is readable, the line breaks are not, but are relevant to the request, as the specification requires each line to be terminated with \r\n. In this particular context it is also not possible to properly encode the line breaks.

One idea to get around this limitation is to provide a "line specification", like so:

raw_lines:
  - content: 7
    ending: "\r\n"
  - content: "foo=bar"
    ending: "\r\n"
  - content: 0
    ending: "\r\n"

This format would give us more control and better readability, in addition to sending malformed request bodies.

Right now go-ftw executes requests one by one and then pull logs expecting the output to so the assertion.

While this is good, sometimes you want to profile and/or measure performance with close to real test data and go-ftw is a good option for this.

I wish go-ftw includes an option where N requests can be sent concurrently. This would allow to collect profiling data to debug latencies when evaluating and triggering rules.

CRS 942101-* tests received "411 Length Required" data. go-ftw sends Content-Length header if request data is not 0 byte for now.
https://github.com/coreruleset/go-ftw/blob/main/ftwhttp/header.go#L78-L80

I think the Content-Length header needed if request header's method is "POST".

Like this.
04ec420#diff-2c39b9ef80e191a7385eb8d54e378f12a1ee08e30e6682274902b68deb46bbbdR112-R114

If possible, Could we create a PR for this?

.ftw.yaml

testoverride:
  input:
    protocol: "https"
    dest_addr: "oursite.com"
    port: 443
mode: "cloud"

Also, Following is our command log.

go-ftw  run -t -d coreruleset/tests/regression/tests/REQUEST-942-APPLICATION-ATTACK-SQLI/ -i 942101 --connect-timeout 10s --trace

running 942101-6: 11:01AM DBG ftw/http: sending data:
POST /[email protected]"%20sleep(10.to_i)%20" HTTP/1.0
Connection: close
Host: oursite.com



11:01AM TRC ftw/http: sending data
11:01AM TRC ftw/http: receiving data
11:01AM TRC ftw/http: received data - "HTTP/1.0 411 Length Required\r\nContent-Type: text/html; charset=UTF-8\r\nReferrer-Policy: no-referrer\r\nContent-Length: 286\r\nDate: Sun, 26 Feb 2023 02:01:21 GMT\r\n\r\n\n<html><head>\n<meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\">\n<title>411 Length Required</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Length Required</h1>\n<h2>POST requests require a <code>Content-length</code> header.</h2>\n<h2></h2>\n</body></html>\n"
💥 failed in 146.461167ms (RTT 158.922959ms)

Hello,
currently, ftw is looking for the IDs of the triggered rules after sending a request. What we are facing running Coraza on Envoy is that the phase when the rule is triggered may differ from the phase when the disruptive action is executed. While it may be intended (enabling or disabling the CRS early blocking feature), it could still be useful to check that an interruption caused by a rule has been raised before a certain phase (therefore avoiding WAF bypasses).
For more context: corazawaf/coraza-proxy-wasm#129 (phase 1 rule with enough anomaly score triggered, but interruption raised only during phase 3. In this case the expected interruption phase would be phase 1 if early blocking, otherwise phase 2).

I'm aware that it is more a check of the expected behaviour of the proxy/server, but what do you think in terms of both usefulness and feasibility? Could there be a way to test when a triggered rule takes action?
I see it as a kind of a Cloud mode test, but checking inside the logs if the interruption has happened during the expected phase.

Just an idea, thanks for any feedback and advice!

I noticed that when looking for a log marker, it also checks for the marker name

https://github.com/fzipi/go-ftw/blob/f5f64a16b3d2bebea600ea070ffd5baa7f36213c/waflog/read.go#L114

The example rule for logging the marker doesn't output the name though

https://github.com/fzipi/go-ftw#how-log-parsing-works

Presumably there is some assumption that the log message automatically includes header names - but would the rule be more generic by including the name in it, e.g.

# Write the value from the X-CRS-Test header as a marker to the log
SecRule REQUEST_HEADERS:X-CRS-Test "@rx ^.*$" \
  "id:999999,\
  phase:1,\
  log,\
  msg:'X-CRS-Test %{MATCHED_VAR}',\
  pass,\
  t:none"

Error: failed to find start marker: can't find log marker. Am I reading the correct log? Log file: /etc/envoy/logs/ftw.log

I was able to find a log in ftw.log

Here's an example

[2023-09-10 15:41:22.182][23][critical][golang] [contrib/golang/common/log/cgo.cc:27] [client "172.23.0.5"] Coraza: Warning. Outbound Anomaly Score Exceeded (Total Score: 0) [file "/etc/envoy/rules/crs/RESPONSE-959-BLOCKING-EVALUATION.conf"] [line "12745"] [id "959100"] [rev ""] [msg "Outbound Anomaly Score Exceeded (Total Score: 0)"] [data ""] [severity "emergency"] [ver "OWASP_CRS/4.0.0-rc1"] [maturity "0"] [accuracy "0"] [tag "anomaly-evaluation"] [hostname "172.23.0.4"] [uri "/status/200"] [unique_id "NeZENfQNNRvvpfrnkqt"]
[2023-09-10 15:41:22.184][22][critical][golang] [contrib/golang/common/log/cgo.cc:27] [client "172.23.0.5"] Coraza: Warning. Outbound Anomaly Score Exceeded (Total Score: 0) [file "/etc/envoy/rules/crs/RESPONSE-959-BLOCKING-EVALUATION.conf"] [line "12745"] [id "959100"] [rev ""] [msg "Outbound Anomaly Score Exceeded (Total Score: 0)"] [data ""] [severity "emergency"] [ver "OWASP_CRS/4.0.0-rc1"] [maturity "0"] [accuracy "0"] [tag "anomaly-evaluation"] [hostname "172.23.0.4"] [uri "/status/200"] [unique_id "xXhtjpXeFQDyPJmWyaf"]

In many cases, more so when in a proxy, we need to wait and block until the proxy (using the WAF) is ready to accept requests.

The way we solved this in coraza-proxy-wasm is that we have a dedicated instruction running a curl and waiting for the host to be available. This is handy but not ideal as we have another piece of complexity for such a simple problem. Also in corporate envs, pulling an image isn't trivial and that is a friction.

I wish go-ftw allows to pass in a URL and it would wait until that URL returns 200.

I'm following README. This is what happens:

$ go install github.com/fzipi/go-ftw@latest
can't load package: package github.com/fzipi/go-ftw@latest: can only use path@version syntax with 'go get'

When I do

$ go install github.com/fzipi/go-ftw
...
build github.com/fzipi/go-ftw: cannot load time/tzdata: malformed module path "time/tzdata": missing dot in first path element

Go version:

$ go version
go version go1.13.8 linux/amd64

Hey there!

I just wanted to point out a few things I ran across when going through the documentation to setup my own local CRS testbed. They are all very small, but my help make it easier for others in the future.

Ftw YAML Placement

The logfile paths give the impression that the yaml placement is in a separate folder outside of the CRS. I later found the .ftw.yaml could be placed within the CRS folder, per the gitignore file, though I'm entirely sure if this is the most common placement.

Apache Logfile Path

Documentation shows

logfile: '../coreruleset/tests/logs/modsec2-apache/apache2/error.log'

but it seems the latest docker image (September 23rd) uses:

logfile: '../coreruleset/tests/logs/modsec2-apache/error.log'
# or if using CRS folder
logfile: 'tests/logs/modsec2-apache/error.log'

Nginx Support

I really don't know if there are any plans to continue Nginx support, but seems only Apache ModSec2 is supported, per the Testing Docs: Starting the Container. I'm not sure if this warrants a mention or not.

-> The supported platform is modsecurity 2 with Apache

Anyways, I know it's not much, but I hope it's helpful.

When running this test suit against proxies we usually hold on waiting for a health endpoint to return 200 before we run the test suite. That requires to introduce extra deps (e.g. curl) or run scripts or pull docker images to achieve that (see https://github.com/corazawaf/coraza-proxy-wasm/blob/main/ftw/tests.sh#L19).

I think ftw should take care of that and add an option --wait-for=http://localhost:8000 or --health-url.

Does that make sense?

% go version          
go version go1.22.0 darwin/amd64
% go install github.com/coreruleset/go-ftw@latest
go: github.com/coreruleset/go-ftw@latest (in github.com/coreruleset/[email protected]):
	The go.mod file for the module providing named packages contains one or
	more replace directives. It must not contain directives that would cause
	it to be interpreted differently than if it were the main module.

Maybe see also linuxkit/linuxkit#3873 for a similar issue.

Your README says this should work.

The ftw YAML format spec says:

response_contains

Description: Checks the entire response against the regular expression provided.

However, go-ftw only matches against the body of the response. This causes problems for tests that want to check that a a specific header is set by the firewall, or want to check responses that have no body.