Hello, I have a question for Mod_security v3 and Ruby
[ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method /' for nil:NilClass [ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method /' for nil:NilClass
[ERROR][logstash.filters.ruby ] Ruby exception occurred: no implicit conversion from nil to integer

Is it because logstash-modsecurity is not compatible with modsecurity v3.0?

my environment is modsecurity3.0+nginx1.15+ELK5

If section H, auditLogTrailer has more than one Message line, only one is retained.

This needs to be reworked so that auditLogTrailer.messages = array from auditLogMessages instead

If a audit log entry has the following (no headers trailing the 1st line of section F), the parse routine fails and this is not captured

...

--49253972-F--
HTTP/1.1 403 Forbidden

--49253972-H--

I'm catching all kinds of Ruby exception errors running in Logstash 5.4
-Direct Event References
-Undefined method '/' for nil.NilClass
-No implicit conversation from nil to integer

Do you have any advice/examples for running under 5.0 after the direct references breaking change?

Hi, I moved the file logstash-modsecurity-example.cfg to logstash-modsecurity.conf in /etc/logstash/conf.d/ and then logstash no more start. The error is the following:

Error: Expected one of #, input, filter, output at line 499, column 1 (byte 15172) after {:level=>:error}

If I disable or move this file it starts. I want to monitor the audit log of nginx, not of apache - this is not the problem why it can't start but it would be great if you tell me that this will not work. Why it can't start?

Also, in the documentation you say "Set the path in logstash-modsecurity.conf to path => "/var/log/httpd/modsec_audit.log"", but I nowhere I found path or where to put this path?

Thank you in advance!

Hi all,
I'm feeding the following log snippet to logstash:

--c7036611-A--
[09/Jan/2008:12:27:56 +0000] OSD4l1BEUOkAAHZ8Y3QAAAAH 123.45.67.89 64995 98.76.54.321 80

Which results in the following errors:

logstash_1      | {:timestamp=>"2015-12-07T12:14:34.546000+0000", :message=>"Ruby exception occurred: undefined method `/' for nil:NilClass", :level=>:error}
logstash_1      | {:timestamp=>"2015-12-07T12:14:34.547000+0000", :message=>"Ruby exception occurred: undefined method `/' for nil:NilClass", :level=>:error}
logstash_1      | {:timestamp=>"2015-12-07T12:14:34.555000+0000", :message=>"Ruby exception occurred: no implicit conversion from nil to integer", :level=>:error}
logstash_1      | {:timestamp=>"2015-12-07T12:14:34.577000+0000", :message=>"Ruby exception occurred: undefined method `/' for nil:NilClass", :level=>:error}
logstash_1      | {:timestamp=>"2015-12-07T12:14:34.582000+0000", :message=>"Ruby exception occurred: undefined method `/' for nil:NilClass", :level=>:error}
logstash_1      | {:timestamp=>"2015-12-07T12:14:34.589000+0000", :message=>"Ruby exception occurred: no implicit conversion from nil to integer", :level=>:error}

When I try to look up the entry in Kibana I'm seeing that rawSectionA is null.

I've also tried a complete audit entry, with the other sections included as well. But that just results in more of the same ruby exceptions.

I haven't touched the configuration file except for the input and output sections.

Update: after further investigation it seems that the tcp input plugin is the problem, since it seems to work with the file plugin.

For your reference, these are my inputs:

input {

  file {
    # IMPORTANT! set this correctly to the charset
    # that your server writes these log files in
    path => "/etc/logstash/conf.d/test_input.txt"
    start_position => "beginning"
    type => "mod_security"

    #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    # merge all modsec events for a given entity into the same event.
    # so essentially the modsec -Z marker is used as the splitter
    # which is the end of each modsec logical event in the logfile
    #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    codec => multiline {
      charset => "US-ASCII"
      pattern => "^--[a-fA-F0-9]{8}-Z--$"
      negate => true
      what => previous
    }
  }

  tcp {
    port => 5000

    type => "mod_security"

    #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    # merge all modsec events for a given entity into the same event.
    # so essentially the modsec -Z marker is used as the splitter
    # which is the end of each modsec logical event in the logfile
    #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    codec => multiline {
      pattern => "^--[a-fA-F0-9]{8}-Z--$"
      negate => true
      what => previous
    }
  }
}

Hi

I work on the exploitation of data "Audit Log Trailer.message" but "Section H" goes back a Array like this:

"auditLogTrailer" => {
               "Apache-Handler" => "application/x-httpd-php",
                    "Stopwatch" => "1457343928118215 7722 (- - -)",
                   "Stopwatch2" => "1457343928118215 7722; combined=2988, p1=333, p2=2515, p3=1, p4=49, p5=89, sr=18, sw=1, l=0, gc=0",
    "Response-Body-Transformed" => "Dechunked",
                     "Producer" => "ModSecurity for Apache/2.7.3 (http://www.modsecurity.org/); OWASP_CRS/2.2.6.",
                       "Server" => "Apache/2.4.6 (CentOS) PHP/5.4.16",
                  "Engine-Mode" => "\"DETECTION_ONLY\"",
                     "messages" => [
        [0] {
                "info" => "Warning. Pattern match \"^[\\\\d.:]+$\" at REQUEST_HEADERS:Host.",
                "file" => "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_21_protocol_anomalies.conf",
                "line" => "98",
                  "id" => "960017",
                 "msg" => "Host header is a numeric IP address",
            "severity" => "WARNING",
                "data" => "123.123.123.123",
                 "tag" => "OWASP_CRS/PROTOCOL_VIOLATION/IP_HOST"
        },
        [1] {
                "info" => "Warning. Pattern match \"(/\\\\*!?|\\\\*/|[';]--|--[\\\\s\\\\r\\\\n\\\\v\\\\f]|(?:--[^-]*?-)|([^\\\\-&])#.*?[\\\\s\\\\r\\\\n\\\\v\\\\f]|;?\\\\x00)\" at ARGS:username.",
                "file" => "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf",
                "line" => "49",
                  "id" => "981231",
                 "msg" => "SQL Comment Sequence Detected.",
            "severity" => "CRITICAL",
                "data" => "Matched Data: --  found within ARGS:username: ' or true -- ",
                 "tag" => "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"
        },
        [2] {
                "info" => "Warning. Pattern match \"(/\\\\*!?|\\\\*/|[';]--|--[\\\\s\\\\r\\\\n\\\\v\\\\f]|(?:--[^-]*?-)|([^\\\\-&])#.*?[\\\\s\\\\r\\\\n\\\\v\\\\f]|;?\\\\x00)\" at ARGS:passwd.",
                "file" => "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf",
                "line" => "49",
                  "id" => "981231",
                 "msg" => "SQL Comment Sequence Detected.",
            "severity" => "CRITICAL",
                "data" => "Matched Data: --  found within ARGS:passwd: ' or true -- ",
                 "tag" => "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"
        },
        [3] {
                "info" => "Warning. Pattern match \"(^[\\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98;]+|[\\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98;]+$)\" at ARGS:username.",
                "file" => "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf",
                "line" => "64",
                  "id" => "981318",
                 "msg" => "SQL Injection Attack: Common Injection Testing Detected",
            "severity" => "CRITICAL",
                "data" => "Matched Data: ' found within ARGS:username: ' or true -- ",
                 "tag" => "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"
        },
        [4] {
                "info" => "Warning. Pattern match \"(^[\\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98;]+|[\\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98;]+$)\" at ARGS:passwd.",
                "file" => "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf",
                "line" => "64",
                  "id" => "981318",
                 "msg" => "SQL Injection Attack: Common Injection Testing Detected",
            "severity" => "CRITICAL",
                "data" => "Matched Data: ' found within ARGS:passwd: ' or true -- ",
                 "tag" => "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"
        },
        [5] {
                "info" => "Warning. Pattern match \"(?i:\\\\bor\\\\b ?(?:\\\\d{1,10}|[\\\\'\\\"][^=]{1,10}[\\\\'\\\"]) ?[=<>]+|(?i:'\\\\s+x?or\\\\s+.{1,20}[+\\\\-!<>=])|\\\\b(?i:x?or)\\\\b\\\\s+(\\\\d{1,10}|'[^=]{1,10}')|\\\\b(?i:x?or)\\\\b\\\\s+(\\\\d{1,10}|'[^=]{1,10}')\\\\s*?[=<>])\" at ARGS:username.",
                "file" => "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf",
                "line" => "133",
                  "id" => "959071",
                 "msg" => "SQL Injection Attack",
            "severity" => "CRITICAL",
                "data" => "Matched Data: ' or true -- found within ARGS:username: ' or true -- ",
                 "tag" => "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"
        },
        [6] {
                "info" => "Warning. Pattern match \"(?i:\\\\bor\\\\b ?(?:\\\\d{1,10}|[\\\\'\\\"][^=]{1,10}[\\\\'\\\"]) ?[=<>]+|(?i:'\\\\s+x?or\\\\s+.{1,20}[+\\\\-!<>=])|\\\\b(?i:x?or)\\\\b\\\\s+(\\\\d{1,10}|'[^=]{1,10}')|\\\\b(?i:x?or)\\\\b\\\\s+(\\\\d{1,10}|'[^=]{1,10}')\\\\s*?[=<>])\" at ARGS:passwd.",
                "file" => "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf",
                "line" => "133",
                  "id" => "959071",
                 "msg" => "SQL Injection Attack",
            "severity" => "CRITICAL",
                "data" => "Matched Data: ' or true -- found within ARGS:passwd: ' or true -- ",
                 "tag" => "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"
        },
        [7] {
            "info" => "Warning. Pattern match \"\\\\W{4,}\" at ARGS:username.",
            "file" => "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf",
            "line" => "154",
              "id" => "960024",
             "msg" => "SQL Character Anomaly Detection Alert - Repetative Non-Word Characters",
            "data" => "Matched Data:  --  found within ARGS:username: ' or true -- "
        },
        [8] {
            "info" => "Warning. Pattern match \"\\\\W{4,}\" at ARGS:passwd.",
            "file" => "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf",
            "line" => "154",
              "id" => "960024",
             "msg" => "SQL Character Anomaly Detection Alert - Repetative Non-Word Characters",
            "data" => "Matched Data:  --  found within ARGS:passwd: ' or true -- "
        },
        [9] {
            "info" => "Warning. Operator GE matched 5 at TX:inbound_anomaly_score.",
            "file" => "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_60_correlation.conf",
            "line" => "37",
              "id" => "981204",
             "msg" => "Inbound Anomaly Score Exceeded (Total Inbound Score: 39, SQLi=24, XSS=): SQL Character Anomaly Detection Alert - Repetative Non-Word Characters"
        }
  ]
},
"event_date_microseconds" => 1457343928118215.0,
"event_date_milliseconds" => 1457343928118.215,
     "event_date_seconds" => 1457343928.118215,
        "event_timestamp" => "2016-03-07T09:45:28.118Z"

}

The problem is that Kibana can not use the IDs, files etc ... and I can make statistics of ones
Kibana error message : "Objects in arrays are not well suported"

Well I guess the problem lies in this part:

  if [rawSectionH] =~ /.+/ {
    kv {
      source => "rawSectionH"
      field_split => "\n"
      value_split => ":"
      target => "auditLogTrailer"
    }

    # trim leading/trailing hack  @see https://logstash.jira.com/browse/LOGSTASH-1369
    ruby {
 code => "
            auditLogTrailer = event.to_hash['auditLogTrailer']
            auditLogTrailerMessages = event.to_hash['auditLogTrailerMessages']
            auditLogTrailer.each { |k, v|
              if !v.nil? and v.is_a? String
                auditLogTrailer[k] = v.strip
              end
            }
            auditLogTrailer.delete('Message')
            auditLogTrailer['messages'] = auditLogTrailerMessages
            event.to_hash.delete('auditLogTrailerMessages')
          "
      }
  }

Do you have an idea ?
Thanks

Hi.. I try to process libmodsecurity audit file.. but it seem like nothing was process..

section H, audit log trailer "producer" can fail on split of : w/ literal ":" in the producer value

Hi Guys,

I am following the project for a while, congrats!

Yesterday we have released ModSecurity version 2.9.1. There was a modification in the logs that might affect you. The issue owasp-modsecurity/ModSecurity#840 contains more details about it.

Also, in v2.9.1 there is this possibility to save the audit logs in JSON format, not sure if the format is structured in the shape that you need for elasticsearch, but may be useful.

Hi,
I have this as log being send by log stash:

2015-04-24T11:43:30.000+0000 hostname.eu %{message}
2015-04-24T11:43:55.000+0000 hostname.eu %{message}

Please add a License to the repository.

@see https://discuss.elastic.co/t/logstash-1-5-x-grok-regex-multiline-behavior-change/34347

When performing a config test on a fresh checkout of the master branch, I'm getting the following error:

./logstash -t -f modsec_conf/0000_header.conf 
Error: Expected one of #, input, filter, output at line 40, column 1 (byte 1515) after ##########################################################

I'm only parsing the header now since I was trying to determine the location of the error and it turns out to be the header...

Am I the only one with this issue?

Using logstash 1.5.5

We did some tests with your modsecurity configuration for logstash and we really like it. Thank you!
Now we want to move a step forward and use this configuration in production. One problem we are facing is the following:

We have to apply some changes to the provided configuration, for example we want to remove the optional parts like "X-Forwarded-For" and "Cookie" and we plan to add some additional filters. At the same time we would love to stay compatible to the upstream configuration provided in this git repository. Therefore I would like to suggest a modularization of the provided configuration.

Before putting actual work into this proposal, I would like to know what you are thinking about this idea.

My suggestion is the following:

Logstash does allow to load the configuration from a directory, where the files are concatenated in lexicographical order.
If the logstash-modsecurity configuration is modularized into several files, it would become easy to select those parts, we want to use and omit the others, like cherry-picking.
This allows to have the logstash-modsecurity repository for example in /etc/logstash/logstash/modsecurty and then simply make symlinks to /etc/logstash/config.d where our logstash installation will look for the actual configuration.
To preserve the correct order, the modularized config parts and the respective symlinks could be prefixed with a number.

E. g.

• 001_input_file.conf
• 100_filter_split_sections.conf
• 110_filter_section_a.conf
• 120_filter_section_b.conf

Every part of the filter should be placed in a separate file an be surrounded by filter { ... }.

This way we are free select the filters we want to use from logstash-modsecurity and to insert own filters between and after the existing ones.

When matching against Audit Log Header (Part A), as it is implemented at the moment, the last audit log entry will not be processed until the next log entry is written to the log file, because logstash is still waiting for the next Audit Log Header (Part A) to match. If using SecAuditLogType Concurrent no modsecurity log entry will ever by processed, as every file does include only one modsecurity log entry and therefore never a second Audit Log Header (Part A) will show up.

The documentation of the multiline filter plugin states that this plugin does not work with multiple filter threads. As the logic to parse the ModSecurity audit log is quite cpu demanding, it would be nice to use multiple threads for the filtering. This may be achieved by using the multiline code on the file input (which is always single threaded per file), instead of the multiline filter plugin.

Hello,

From times to times, i got a ruby exception

Ruby exception occurred: undefined method `/' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `/' for nil:NilClass {:level=>:error}
Ruby exception occurred: no implicit conversion from nil to integer {:level=>:error}
Ruby exception occurred: undefined method `/' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `/' for nil:NilClass {:level=>:error}
Ruby exception occurred: no implicit conversion from nil to integer {:level=>:error}
Ruby exception occurred: undefined method `/' for nil:NilClass {:level=>:error}
....

I'm using logstash 2.4 with master branch

Hi,
I upgraded my logstash server to 2.3.4 using this filter.

After about a week the java process died with "out of heap space". Increasing the heap didn't help, it just took longer until the java process died.

After long hours of debugging I found a solution to this:

In the file 2030_filter_section_c_parse.conf I replaced the lines:

  grok {
    match => {
      "rawSectionC" => "(?<requestBody>.+)"
    }

with

  mutate {
    add_field => { "requestBody" => "%{rawSectionC}" }
  }

It basically just copies the field rawSectionC to requestBody. That's also what the grok is doing. For me the result is the same.

My logstash 2.3.4 server is running now a couple of weeks with this config without any issues.

For anyone using this filter with logstash 2.x you should also replace the line in the inputs from

  charset => "US-ASCII"

to

 codec => plain { charset => "US-ASCII" }

Other than that the filter is working fine for me with the new logstash version.

Hi,

Thanks for making this available. I'm quite new to logstash so struggled a bit when setting this up.
Not sure if everyone will have the same problem but I had to carry out the steps below.
The version I have is logstash-1.4.2-1_2c0f5a1.noarch on Fedora 18.

On the agent host, copy logstash-modsecurity.conf to /etc/logstash/conf.d and logstash_modsecurity_patterns to /etc/logstash/patterns.

In logstash-modsecurity.conf, change

patterns_dir => "./patterns/logstash_modsecurity_patterns"
to:
patterns_dir => "/etc/logstash/patterns/logstash_modsecurity_patterns"

And add the line below to specify the log type as mod_security

input {
file {
path => "/var/log/httpd/modsec_audit.log"
type => "mod_security"
}
}

Depending on the permissions on modsec_audit.log, you may need to run this also:

setfacl -m u:logstash:r /var/log/httpd/modsec_audit.log

After doing that, restart the logstash agent

Hope this is helpful!
Ed

Just a comment to try out this new "config.support_escapes" feature; this is supposedly fixed in PR elastic/logstash#7442

add support for section E (response body)

If any header is repeated in request/response, the last value wins. Bug.

This parses OK

[16/Jul/2014:10:38:28 --0400] ejnuZAoO04QAACVQG2oAAAAG 90.14.211.240 58564 50.44.29.132 80

This does not

[16/Jul/2014:19:27:34 +0200] 1vCybQpzE8QAABxlNBsAAAAF 19.222.19.19 61552 20.222.29.196 80

This pattern change fixes it

%{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} [-\+]{1,2}%{INT}

Not sure where the issue, agent consumption or filtering code. But this no longer works in Logstash 1.4

Hey there,

first of all, thank you very much for providing this repo! This helped me a lot.

I'm currently having an issue with some log data in sectionPartH not getting properly parsed into separate messages inside of the auditLogTrailer.Message.

Example sectionPartH:
Message: Found 1 byte(s) in REQUEST_URI outside range: 1-255. [file "/etc/httpd/owasp-modsecurity-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "574"] [id "920270"] [rev "2"] [msg "Invalid character in request (null character)"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "9"] [accuracy "9"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/EVASION"]
Message: Found 1 byte(s) in ARGS:t outside range: 1-255. [file "/etc/httpd/owasp-modsecurity-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "574"] [id "920270"] [rev "2"] [msg "Invalid character in request (null character)"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "9"] [accuracy "9"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/EVASION"]
Message: Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file "/etc/httpd/owasp-modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "57"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 8)"] [severity "CRITICAL"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"]
Message: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/etc/httpd/owasp-modsecurity-crs/rules/RESPONSE-980-CORRELATION.conf"] [line "73"] [id "980130"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 8 - SQLI=0,XSS=0,RFI=0,LFI=0,RCE=0,PHPI=0,HTTP=0,SESS=0): Invalid character in request (null character)"] [tag "event-correlation"]

Action: Intercepted (phase 2)
Apache-Handler: proxy-server
Stopwatch: 1520438720252919 3263 (- - -)
Stopwatch2: 1520438720252919 3263; combined=2245, p1=1078, p2=1048, p3=0, p4=0, p5=119, sr=453, sw=0, l=0, gc=0
Producer: ModSecurity for Apache/2.9.2 (http://www.modsecurity.org/); OWASP_CRS/3.0.2.
Server: Apache/2.4.6 (CentOS)
Engine-Mode: "ENABLED"

The auditLogTrailer.Message contains all the matched rules but instead of an array of messages as a single String.

logstash log:
[2018-03-07T18:57:15,892][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method `[]' for nil:NilClass

I'll examine the regExpressions in the meantime, Looking forward for your answer!

Best regards
Alex