Backend: Powershell about sigma HOT 7 CLOSED

Well... there we are :) I was able to implement (after some reversing about the inner workings of sigma...) an initial version of the PowerShell backend. Feel free to use it from my fork (referenced).

@thomaspatzke Hope it's ok for you that I made the PR for the initial version, so it's easier to collaborate on the backend. If not, please advice accordingly.

There is of course some further testing needed and some cleanup of the code, but I'm not able to test everything due to missing log sources. I tested a lot with sysmon and windows security events logs. Some todos: add backend options if needed (e.g. out-gridview) and add the timeframe condition.

Use "set-clipboard" to copy the output of the command without the hassle of copying the shell output:
python .\tools\sigmac -t powershell -c .\tools\config\powershell-windows-all.yml .\rules\windows\sysmon\sysmon_susp_certutil_command.yml | Set-Clipboard

Example how it looks like (stripped all irrelevant lines from the output):

PS> Get-WinEvent -LogName "Microsoft-Windows-Sysmon/Operational" | where { ($_.ID -eq "1" -and $_.message -match "Image.*.*\\attrib.exe" -and $_.message -match
"CommandLine.*.* \+h .*") -and  -not ($_.LogName -eq "Microsoft-Windows-Sysmon/Operational" -and ($_.message -match "CommandLine.*.*\\desktop.ini .*") -or ($_.m
essage -match "ParentImage.*.*\\cmd.exe" -and $_.message -match "CommandLine.*\+R \+H \+S \+A \\.*.cui" -and $_.message -match "ParentCommandLine.*C:\\WINDOWS\\
system32\\\\.*.bat")) } | select TimeCreated,Id,RecordId,ProcessId,MachineName,Message

TimeCreated : 23.09.2018 19:53:11
Id          : 1
RecordId    : 4862220
Message     : Process Create:
              UtcTime: 2018-09-23 17:53:11.642
              Image: C:\Windows\SysWOW64\attrib.exe
              CommandLine: attrib  x +h x
              ParentImage: C:\Windows\System32\cmd.exe
              ParentCommandLine: "C:\Windows\system32\cmd.exe"

I will put some notes here for discussion and some implementation issues I ran into (many I guess are because of my limited experience and knowledge of the inner workings of sigma).

First of all, Get-WinEvent is very slow for large event logs like sysmon. Normal sigma output doesn't differentiate enough for our PowerShell cmdlet. The cmdlet has very basic pre-filtering capapbilities (only log source, time and event id). So most of the filtering will be made with the "where" clause AFTER reading all of the events... the best with which I came up is to at least use the log source ("-LogName") parameter directly in Get-WinEvent for basic prefiltering (with a special regex hack)...we can't use the same prefiltering for the event id because there are rules with multiple event ids.

Change the default query from

PS> Get-WinEvent | where { $_.LogName -eq "Security" -and ($_.ID -eq "4624" -and $_.message -match "LogonType.*5" -and $_.message -match "AuthenticationPackageName.*Negotiate" -and $_.message -match "TargetUserName.*SYSTEM") } | ft -auto TimeCreated,Id,RecordId,ProcessId,MachineName,Message

to

PS> Get-WinEvent -LogName "Security"  | where { ($_.ID -eq "4624" -and $_.message -match "LogonType.*5" -and $_.message -match "AuthenticationPackageName.*Negotiate" -and $_.message -match "TargetUser Name.*SYSTEM") } | ft -auto TimeCreated,Id,RecordId,ProcessId,MachineName,Message

I tested one of the rules and the following output shows the time usage...

PS> Measure-Command -Expression {Get-WinEvent -LogName "Microsoft-Windows-Sysmon/Operational" | where { ($_.ID -eq "1" -and ($_.message -match "CommandLine.*.*\
\certutil.exe .* -decode .*" -or $_.message -match "CommandLine.*.*\\certutil.exe .* -decodehex .*" -or $_.message -match "CommandLine.*.*\\certutil.exe .*-urlc
ache.* http.*" -or $_.message -match "CommandLine.*.*\\certutil.exe .*-urlcache.* ftp.*" -or $_.message -match "CommandLine.*.*\\certutil.exe .*-URL.*" -or $_.m
essage -match "CommandLine.*.*\\certutil.exe .*-ping.*")) } | select TimeCreated,Id,RecordId,ProcessId,MachineName,Message}

Days              : 0
Hours             : 0
Minutes           : 4
Seconds           : 40
Milliseconds      : 155
Ticks             : 2801558363
TotalDays         : 0.00324254440162037
TotalHours        : 0.0778210656388889
TotalMinutes      : 4.66926393833333
TotalSeconds      : 280.1558363
TotalMilliseconds : 280155.8363

The other thing is the language of the event logs... unfortunately, some of the event logs read with Get-WinEvent are in the user's language that means that we can't rely on English words and the PowerShell configuration must be updated accordingly. Therefore, I currently only use the values and not everywhere the key from the value (because that differs).

And one last thing regarding group-object... it's the only way PowerShell allows grouping (there's a special grouping by format-table -groupby..., but, well that's not exactly what we want...). So, there's no straight with way with group-object for an aggregation with an aggregation field and an group field. I had to use the following hack (use ps as demonstration, distinct count of handles per process)... such a queries like | stats count(handles) by name or count(UserName) by SourceWorkstation > 3 (see https://github.com/Neo23x0/sigma/wiki/Specification) are not available directly with group-object.

ps | select name,handles | group name | % {[PSCustomObject]@{'Name'=$_.name;'Count'=($_.group.Handles | sort -u).count}} | sort count -desc

from sigma.

Hi Thomas, what's the state on this issue? Did you already started working on it?

from sigma.

Nothing done yet. Would you like to have it assigned?

from sigma.

No, I'm currently not able to implement it. If I work on the backend I'll post the progress here.

from sigma.

I started working on the issue and I should be able to have something done in the next days/weeks. You can assign me the issue if you like.

from sigma.

Hi! Great! 👍

@thomaspatzke Hope it's ok for you that I made the PR for the initial version, so it's easier to collaborate on the backend. If not, please advice accordingly.

Absolutely! That's the preferred way for new code. I will test and integrate it in the next days!

First of all, Get-WinEvent is very slow for large event logs like sysmon.

I think it's a bit like with the grep backend, which is inefficient and far away from being perfect, but very useful in some cases.

The other thing is the language of the event logs... unfortunately, some of the event logs read with Get-WinEvent are in the user's language that means that we can't rely on English words and the PowerShell configuration must be updated accordingly. Therefore, I currently only use the values and not everywhere the key from the value (because that differs).

This is more a general issue, where we possibly need a solution that is independent from the backend, like with field name mappings.

from sigma.

@thomaspatzke @Neo23x0 Thanks for already merging the PR. I will test it further and improve the backend where possible. Please ping me, if issues are raised with the backend.

One thing I missed mentioning yesterday in the comment above is that it's needed in the backend to differentiate between the message content (event log specific data, .e.g CommandLine in Sysmon) and the system fields (like Event Id, created time, ...). For system fields the where filter must use "$_.<fieldname>" and for all other fields it must match on the message field "$_.message -match "...". Furthermore, the event logs read by Get-WinEvent use "ID" instead of "EventID". Without using the config file the EventId is not "ID" and will be applied on the wrong field ($_.message -match "<ID>" instead of $_.ID -eq "<ID>").

Is the current implementation with mandatory config file (for EventId = ID) and some static code in the backend for special handling of the Id and logname fields ok? Or should I make the config file optional and translate the EventId name in the backend to "ID" because it's always only "ID"?

from sigma.