dod-cyber-crime-center / dc3-mwcp Goto Github PK

DC3 Malware Configuration Parser (DC3-MWCP) is a framework for parsing configuration information from malware. The information extracted from malware includes items such as addresses, passwords, filenames, and mutex names.

License: Other

Python 91.92% HTML 6.12% CSS 1.54% C 0.29% YARA 0.13%

python malware-analysis automation config-dump framework malware-automation

dc3-mwcp's People

Contributors

Stargazers

Watchers

dc3-mwcp's Issues

Cannot set decoderdir for run_kordesii_decoder

For a FileObject within dispatcher.py, it is not currently possible to set the decoderdir for a kordesii decoder to anything other than the default kordesii decoder directory.

Propose adding an argument to run_kordesii_decoder to enable users to specify the decoderdir when a kordesiireporter object is instantiated to run a decoder.

Fields in malwareconfigreporter and fields.json do not match, using certain fields results in error.

For example, malwareconfigreporter.py contains the fields "registrykeyvalue", "registrykey", "registryvalue", "key", but these do not match the fields in fields.json: "registrydata", "registrypath", "registrypathdata". Using any of these fields with 'add_metadata' results in "Error adding metadata because is not an allowed key".
Should the fields in fields.json, or a subset thereof, not match those in malwareconfigreporter.py?

Bug: Installation Broken with SetupTools>=70.0.0 on Windows

MWCP doesn't work with setuptools>=70.0.0. I have verified that this still works with setuptools==69.5.1. This issue is tracked upstream at pypa/setuptools#4399. If MWCP is updated to use a pyproject.toml for installation, then the setuptools version would be able to be specified and no longer cause opaque issues.

Traceback (most recent call last):
  File "\\?\C:\Users\analyst\AppData\Roaming\Python\Python310\Scripts\mwcp-script.py", line 33, in <module>
    sys.exit(load_entry_point('mwcp', 'console_scripts', 'mwcp')())
  File "\\?\C:\Users\analyst\AppData\Roaming\Python\Python310\Scripts\mwcp-script.py", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "C:\Python310\lib\importlib\metadata\__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "C:\Python310\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "c:\users\analyst\git\mwcp\mwcp\__init__.py", line 9, in <module>
    from mwcp.config import _config as config
  File "c:\users\analyst\git\mwcp\mwcp\config\__init__.py", line 6, in <module>
    import pkg_resources
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 3283, in <module>
    def _initialize_master_working_set():
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 3266, in _call_aside
    f(*args, **kwargs)
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 3295, in _initialize_master_working_set
    working_set = _declare_state('object', 'working_set', WorkingSet._build_master())
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 589, in _build_master
    ws.require(__requires__)
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 926, in require
    needed = self.resolve(parse_requirements(requirements))
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 787, in resolve
    dist = self._resolve_dist(
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 816, in _resolve_dist
    env = Environment(self.entries)
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 1014, in __init__
    self.scan(search_path)
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 1046, in scan
    for dist in find_distributions(item):
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 2091, in find_on_path
    yield from factory(fullpath)
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 2183, in resolve_egg_link
    return next(dist_groups, ())
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 2179, in <genexpr>
    resolved_paths = (
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 2167, in non_empty_lines
    for line in _read_utf8_with_fallback(path).splitlines():
NameError: name '_read_utf8_with_fallback' is not defined
Traceback (most recent call last):
  File "\\?\C:\Users\analyst\AppData\Roaming\Python\Python310\Scripts\kordesii-script.py", line 33, in <module>
    sys.exit(load_entry_point('kordesii', 'console_scripts', 'kordesii')())
  File "\\?\C:\Users\analyst\AppData\Roaming\Python\Python310\Scripts\kordesii-script.py", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "C:\Python310\lib\importlib\metadata\__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "C:\Python310\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "c:\users\analyst\git\kordesii\kordesii\__init__.py", line 7, in <module>
    from .tester import Tester
  File "c:\users\analyst\git\kordesii\kordesii\tester.py", line 17, in <module>
    from kordesii import registry
  File "c:\users\analyst\git\kordesii\kordesii\registry.py", line 9, in <module>
    import pkg_resources
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 3283, in <module>
    def _initialize_master_working_set():
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 3266, in _call_aside
    f(*args, **kwargs)
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 3295, in _initialize_master_working_set
    working_set = _declare_state('object', 'working_set', WorkingSet._build_master())
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 589, in _build_master
    ws.require(__requires__)
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 926, in require
    needed = self.resolve(parse_requirements(requirements))
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 787, in resolve
    dist = self._resolve_dist(
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 816, in _resolve_dist
    env = Environment(self.entries)
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 1014, in __init__
    self.scan(search_path)
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 1046, in scan
    for dist in find_distributions(item):
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 2091, in find_on_path
    yield from factory(fullpath)
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 2183, in resolve_egg_link
    return next(dist_groups, ())
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 2179, in <genexpr>
    resolved_paths = (
  File "C:\Users\analyst\AppData\Roaming\Python\Python310\site-packages\pkg_resources\__init__.py", line 2167, in non_empty_lines
    for line in _read_utf8_with_fallback(path).splitlines():
NameError: name '_read_utf8_with_fallback' is not defined

No support for python3

Would a PR for python3 support be desired?

Feature Request: Pytest Coverage Reports for Parser Test Cases

Hello! When adding parsers to MWCP, we do not have a solid and consistent way to establish code coverage for parser test cases. Can a CLI sub-command be added to generate a coverage report for a target parser? We found that the following commands, based partially on the existing mwcp test command, can be used to get a coverage report that gives code coverage for a parser or set of parser:

coverage run --source=mwcp\parsers -m pytest --disable-pytest-warnings --durations  10 --tb short -o cache_dir="C:\Users\vagrant\AppData\Local\mwcp\mwcp\.pytest_cache" --cache-clear mwcp\tests\test_parsers.py -k parser_name1

coverage run --source=mwcp\parsers -m pytest --disable-pytest-warnings --durations  10 --tb short -o cache_dir="C:\Users\vagrant\AppData\Local\mwcp\mwcp\.pytest_cache" --cache-clear mwcp\tests\test_parsers.py -k "parser_name1 or parser_name2"

For this feature, we ask that:

The CLI command works with a syntax so it is something like mwcp coverage [parser_name]
The coverage report contains only the parser files in the report, ideally just the ones specified in the command
The output is the coverage report, any additional processing like the command coverage html should be done by the user
If possible, the command should be able to process parser tests in parallel and not sequentially (the above command runs sequentially)

Thank you!

`mwcp test` doesn't allow using `--yara-repo`

In situations where a YARA_REPO has not been set in the mwcp configuration, the --yara-repo argument would enable setting the repository on the command line.

However, it is missing from the cli, as an option for pytest, and from usage when testing parsers.

Requesting its addition for the proposed use case, where it is otherwise a silent failure.

Error installing with pip

On a fresh Debian install, I get an error when trying to install with pip as it seems to be no longer available in the repository:

sudo pip install mwcp
Collecting mwcp
Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://pypi.org/simple/mwcp/

Scheduled Task Metadata Field

Requesting the following addition to the standard metadata fields, due to prominence for persistence (scheduled tasks)

Scheduled Tasks are a common method for persistence on a Windows system, and it would be desirable to have standard metadata for its reporting, particularly to link task names with at least taskrun (can be a metadata.Command) parameters.

Would request at least the following parameters (reflected in the options from MSDN above), where most are optional:

taskname (perhaps just as name
taskrun (would be a metadata.Command - perhaps evaluate for filepaths?)
description
author

Feature Request: allow formatted dictionary/list output in CLI

This issue cropped up when comparing the output of an alternative Cobalt Strike beacon parsing tool to that of an MWCP parser implementation.
The alternative tool outputs dictionaries like this:

HttpGet_Metadata                 - ConstHeaders
                                        Host: example.com
                                        Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
                                        Accept-Language: en-US,en;q=0.5
                                 ...

When an Other metadata object with the same data is added to a report, the MWCP CLI output looks like this:

Tags    Key                Value
------  -----------------  --------------------------------------------------------------------------------
        httpget_metadata   {'ConstHeaders': ['Host: example.com', 'Accept: text/html,application/xhtml+
                             xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language: en-US,en;q=0.5'] ... }

Having to condense all of the keys and values into a single string and printing it out like any other string is not ideal for human-readability.
If we could either implement support for printing list/dict items on different lines with visual nesting like the alternative tool I included an excerpt from or just use pprint on those objects before writing them to the CLI output, that would significantly improve the usability of the output for any parser which has objects it needs to output containing a set of keys that are not known beforehand.

Feature Request: Report Serialization

Hello. We would like to be able to store the mwcp.report.Report in a single format, then load that data back into mwcp for conversion to any of the other formats, depending on user preference. With just the canonical mwcp format and legacy format this wasn't to bad to do ourselves, but with the addition of stix and it's significantly different format it would require us to maintain multiple copies of data and code path.

Obviously we don't expect the full functionality of reports to work after loading, just access to utilities like as_stix, as_json would be sufficient.

Thanks,
Ryan

test.py

pefileutils.__obtain_exif_fname__ Error

DC3-MWCP does not pin the version of pefile and it appears that the manner in which pefileutils is attempting to use pefile.PE().FileInfo is no longer valid, and it will always throw an AttributeError.

Current implementation does:

        for file_info in pe.FileInfo:
            if file_info.Key == 'StringFileInfo':

Which will throw an AttributeError because file_info has no Attribute "Key". The following code will work for the latest version of pefile:

            for structs in file_object.pe.FileInfo:
                for file_info in structs:
                    if file_info.Key == "StringFileInfo":

Based on how this is leveraged in the file_object of DC3-MWCP, I imagine this will break test cases.

poshdeob.py detected as malicious (VirusTotal)

On VT today,
Kaspersky: HEUR:Trojan.PowerShell.Generic
Bkav Pro: ASP.Webshell
https://www.virustotal.com/gui/file/c044aa7e5851f152a734265e00677bd667dca5cad37a80335c4433d92b74b17b

Report writer UnicodeDecodeError

When attempting to add bytes formatted data using metadata.DecodedString or metadata.Other, a UnicodeDecodeError can be thrown because there is no special handling for representing the information like there is in metadata.EncryptionKey. This will result in a report not being generated.

An example stack trace follows:

Traceback (most recent call last):
  File "c:\python37\lib\site-packages\mwcp\cli.py", line 351, in parse
    print(report.as_text(format, split=split))
  File "c:\python37\lib\site-packages\mwcp\report.py", line 581, in as_text
    writer.write(self._report_model)
  File "c:\python37\lib\site-packages\mwcp\report_writers.py", line 219, in write
    self._write_table(elements)
  File "c:\python37\lib\site-packages\mwcp\report_writers.py", line 176, in _write_table
    self.table(tabular_data, headers="keys")
  File "c:\python37\lib\site-packages\mwcp\report_writers.py", line 159, in table
    self._stream.write(tabulate.tabulate(tabular_data, headers=headers, tablefmt=self._tablefmt))
  File "c:\python37\lib\site-packages\tabulate.py", line 1593, in tabulate
    for c, ct, fl_fmt, miss_v in zip(cols, coltypes, float_formats, missing_vals)
  File "c:\python37\lib\site-packages\tabulate.py", line 1593, in <listcomp>
    for c, ct, fl_fmt, miss_v in zip(cols, coltypes, float_formats, missing_vals)
  File "c:\python37\lib\site-packages\tabulate.py", line 1592, in <listcomp>
    [_format(v, ct, fl_fmt, miss_v, has_invisible) for v in c]
  File "c:\python37\lib\site-packages\tabulate.py", line 984, in _format
    return _text_type(val, "ascii")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfc in position 0: ordinal not in range(128)
Error running DC3-MWCP: 'ascii' codec can't decode byte 0xfc in position 0: ordinal not in range(128)

kordesii Reporter incompatible with 1.3.0

Within dispatcher, importing the kordesii reporter is not compatible with the updates for kordesii 1.3.0.

Line 23 would need to be from kordesii.reporter import Reporter as kordesiireporter, but this would not provide backwards compatibility with kordesii < 1.3.0

Recursion Bug

Discovered an edge-case bug while attempting to use the YARA recursion feature. If one of the parsers for a sub-component dispatches a new file with a parent of it's sibling, it may not get picked up because the parent was already processed.

"unhashable type 'list'" when adding a list to reporter metadata

Is it possible to add a list to reporter metadata? I would expect this to be as simple as outlined below, but apparently it is not.

  30         self.nodes = list()
  31         if self.results:
  32             for r in self.results:
  33                 self.nodes.append({"ipv4": str.split(r, '\x00')[0],
  34                                    "pub_key": binascii.hexlify(r[34:])})
  35             self.reporter.add_metadata("other", {"nodes": self.nodes})

Error:

Could not add object of <type 'list'> to metadata under other using key nodes

Pecon's PE object initialization is broken

Pecon's recommended way to initialize a PE object results in an error.
https://github.com/Defense-Cyber-Crime-Center/DC3-MWCP/blob/586707c7e98275a2177afd3d49af0ad79b127c29/mwcp/utils/pecon.py#L8

>>> from mwcp.utils import pecon
>>> pe = pecon.PE()
Traceback (most recent call last):
  File "<venv_path>\lib\site-packages\mwcp\utils\construct\version28.py", line 565, in _sizeof
    return sum(sc._sizeof(nest(context, sc), path) for sc in self.subcons)
  File "<venv_path>\lib\site-packages\mwcp\utils\construct\version28.py", line 565, in <genexpr>
    return sum(sc._sizeof(nest(context, sc), path) for sc in self.subcons)
  File "<venv_path>\lib\site-packages\construct\core.py", line 2476, in _sizeof
    return self.subcon._sizeof(context, path)
  File "<venv_path>\lib\site-packages\construct\core.py", line 3605, in _sizeof
    condfunc = condfunc(context)
  File "<venv_path>\lib\site-packages\construct\expr.py", line 160, in __call__
    lhs = self.lhs(obj) if callable(self.lhs) else self.lhs
  File "<venv_path>\lib\site-packages\construct\expr.py", line 188, in __call__
    return self.__parent(obj)[self.__field]
KeyError: 'Magic'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<venv_path>\lib\site-packages\mwcp\utils\pecon.py", line 284, in __init__
    'SizeOfOptionalHeader': construct.IMAGE_OPTIONAL_HEADER.sizeof(NumberOfRvaAndSizes=16),
  File "<venv_path>\lib\site-packages\construct\core.py", line 398, in sizeof
    return self._sizeof(context, "(sizeof)")
  File "<venv_path>\lib\site-packages\mwcp\utils\construct\version28.py", line 567, in _sizeof
    raise SizeofError("cannot calculate size, key not found in context")
construct.core.SizeofError: cannot calculate size, key not found in context

The exception happens because the initialization of the PE object attempts to calculate the size of the IMAGE_OPTIONAL_HEADER struct to set the SizeOfOptionalHeader variable. The size depends on whether the binary is 32-bit of 64-bit, set by the Magic value of the struct. This Magic value is not provided, causing the crash.
https://github.com/Defense-Cyber-Crime-Center/DC3-MWCP/blob/586707c7e98275a2177afd3d49af0ad79b127c29/mwcp/utils/pecon.py#L284

Simply providing the constant for the 32-bit option fixes the crash, but limits the library to generating 32-bit PE files.

'SizeOfOptionalHeader': construct.IMAGE_OPTIONAL_HEADER.sizeof(Magic=IMAGE_NT_OPTIONAL_HDR32_MAGIC, NumberOfRvaAndSizes=16),

To overcome the 32-bit limitation, constants also need to be changed in other places where it was assumed that the binary is 32-bit and the constant was hard-coded (often with an attached TODO comment):

the definition of the IMAGE_FILE_HEADER struct:
https://github.com/Defense-Cyber-Crime-Center/DC3-MWCP/blob/586707c7e98275a2177afd3d49af0ad79b127c29/mwcp/utils/construct/windows_structures.py#L200
the initialization of the optional header for the PE object in pecon: https://github.com/Defense-Cyber-Crime-Center/DC3-MWCP/blob/586707c7e98275a2177afd3d49af0ad79b127c29/mwcp/utils/pecon.py#L298
the initialization of the file header for the PE object in pecon: https://github.com/Defense-Cyber-Crime-Center/DC3-MWCP/blob/586707c7e98275a2177afd3d49af0ad79b127c29/mwcp/utils/pecon.py#L279
https://github.com/Defense-Cyber-Crime-Center/DC3-MWCP/blob/586707c7e98275a2177afd3d49af0ad79b127c29/mwcp/utils/pecon.py#L290

Perhaps the architecture of the PE file should be an argument of the PE object constructor, allowing setting the right constants for the architecture.

parsers not in list

When adding a new parser to the mwcp/parsers directory, and then attempting to run the mwcp-tool.py -l command, the new parser does not appear in the list, nor does it compile.

Additionally, when trying to run the parser using the -p option, mwcp indicates the parser does not exist.

However, when then trying to reference the parser by specifying the parser directory, using the --parserdir option, mwcp can run the parser successfully.

dod-cyber-crime-center / dc3-mwcp Goto Github PK

dc3-mwcp's People

Contributors

Stargazers

Watchers

Forkers

dc3-mwcp's Issues

Recommend Projects

Recommend Topics

Recommend Org