mandiant / flare-floss Goto Github PK

FLARE Obfuscated String Solver - Automatically extract obfuscated strings from malware.

License: Apache License 2.0

Python 100.00%

malware deobfuscation strings flare malware-analysis gsoc-2024

flare-floss's Introduction

FLARE Obfuscated String Solver

The FLARE Obfuscated String Solver (FLOSS, formerly FireEye Labs Obfuscated String Solver) uses advanced static analysis techniques to automatically extract and deobfuscate all strings from malware binaries. You can use it just like strings.exe to enhance the basic static analysis of unknown binaries.

Obfuscated Strings

Rather than heavily protecting backdoors with hardcore packers, many malware authors evade heuristic detections by obfuscating only key portions of an executable. Often, these portions are strings and resources used to configure domains, files, and other artifacts of an infection. These key features will not show up as plaintext in the output of the strings.exe utility that we commonly use during basic static analysis.

FLOSS extracts all the following string types:

static strings: "regular" ASCII and UTF-16LE strings
stack strings: strings constructed on the stack at run-time
tight strings: a special form of stack strings, decoded on the stack
decoded strings: strings decoded in a function

Please review the theory behind FLOSS here.

Our blog post talks more about the motivation behind FLOSS and details how the tool works.

FLOSS version 2.0 updates are detailed in this blog post.

Language-specific Strings

Not all compilers use string formats that the classic strings.exe algorithm supports. For example, if strings are UTF-8 encoded or stored without a NULL-terminator. FLOSS can identify and extract strings from programs compiled from the following languages:

Go
Rust

The strings FLOSS extracts specific to a compiler are much easier to inspect by humans.

Please consult the documentation to learn more about the language-specific string extraction.

Installation

To use FLOSS, download a standalone executable file from the releases page: https://github.com/mandiant/flare-floss/releases

See the installation documentation for a detailed description of all methods to install FLOSS.

Usage Examples

Extract obfuscated strings from a malware binary:

$ floss malware.exe

Only extract stack and tight strings:

$ floss --only stack tight -- suspicious.exe

Do not extract static strings:

$ floss --no static -- backdoor.exe

Display the help/usage screens:

$ floss -h  # show core arguments
$ floss -H  # show all supported arguments

For a detailed description of using FLOSS, review the documentation here.

Scripts

FLOSS also contains additional Python scripts in the scripts directory which can be used to load its output into other tools such as Binary Ninja or IDA Pro. For detailed description of these scripts review the documentation here.

flare-floss's People

Contributors

Stargazers

Watchers

Forkers

bouncyllama williamgibb michaeltestliu m00zh33 flycrane wflk lucabongiorni ww9210 cephurs binsrc kumaraguruv securextools crazykid199 redragonvn akiraaisha tun4 testzzzz chubbymaggie jaonlin ohio813 gleeda mikedawg guardianrg razuz secpersu sjy-nankai jadore melbshark 453483289 honeyflyfish pilate seth1002 cacker sbkiller hadesv embpgp thx0701 ioanlongjing qardeneden shalekesan maijin coffeesoft diggold kissthink nsxz nhfe zdone shizhx walterjia kernullist parknom nnamon tempuser1909 mattwatson123 zdzhjx h0wl deki0r gradetwo importantfork neighty hkingz redteamcaliber pombredanne b5y ch0psticks techlord-rce ostefano noobzero bugmeout amesianx malikvivek gitcollect y0d4a idkwim zh-explorer 01btc10 kopites shekkbuilder cerebralmischief ashr spagett fatgrass mrexodia jack51706 solertis hammerzeit javarockstar olivierh59500 null0x4d5a lovewitty i85yl64 fjh658 joinbaijun phpplay nexthacker james2060 crystalboy hackmycontrolsystem zhencang crltnhrrs

flare-floss's Issues

test: decode to allocated buffer

char *decode(const char *in, int in_len) {
    char *out = malloc(in_len);
    memcpy(out, in, in_len);
    return out;
}

test: decoded data on heap

test: split stackstrings

handle case in this image:

Unit tests fail

Unit tests fail since they rely on binaries which are not present in the git repo. Furthermore, not all of the binaries used for unit tests are present in VirusTotal, preventing anyone with VTI access from actually reproducing the full suite of unit tests.

Can you select binaries which are publicly shared and include them in the repo? Possibly under tests/assets/binaries/* ?

pip installation fails - can't copy 'floss/plugins': doesn't exist or not a regular file

Tried to install it on Ubuntu and OSX, and get the same error. Here is the log from OSX:

csaby:Downloads coe$ pip install https://github.com/fireeye/flare-floss/zipball/master
Collecting https://github.com/fireeye/flare-floss/zipball/master
  Downloading https://github.com/fireeye/flare-floss/zipball/master (76kB)
    100% |████████████████████████████████| 81kB 7.5kB/s 
Requirement already satisfied (use --upgrade to upgrade): vivisect in /usr/local/lib/python2.7/site-packages (from floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): plugnplay in /usr/local/lib/python2.7/site-packages (from floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): viv-utils in /usr/local/lib/python2.7/site-packages (from floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): pycparser in /usr/local/lib/python2.7/site-packages (from vivisect->floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): pefile in /usr/local/lib/python2.7/site-packages (from viv-utils->floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): argparse in /usr/local/lib/python2.7/site-packages (from viv-utils->floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): ipython in /usr/local/lib/python2.7/site-packages (from viv-utils->floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): funcy in /usr/local/lib/python2.7/site-packages (from viv-utils->floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): intervaltree in /usr/local/lib/python2.7/site-packages (from viv-utils->floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): traitlets in /usr/local/lib/python2.7/site-packages (from ipython->viv-utils->floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): pickleshare in /usr/local/lib/python2.7/site-packages (from ipython->viv-utils->floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): simplegeneric>0.8 in /usr/local/lib/python2.7/site-packages (from ipython->viv-utils->floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): setuptools>=18.5 in /usr/local/lib/python2.7/site-packages (from ipython->viv-utils->floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): decorator in /usr/local/lib/python2.7/site-packages (from ipython->viv-utils->floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): gnureadline in /usr/local/lib/python2.7/site-packages (from ipython->viv-utils->floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): appnope in /usr/local/lib/python2.7/site-packages (from ipython->viv-utils->floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): pexpect in /usr/local/lib/python2.7/site-packages (from ipython->viv-utils->floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): sortedcontainers in /usr/local/lib/python2.7/site-packages (from intervaltree->viv-utils->floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): ipython-genutils in /usr/local/lib/python2.7/site-packages (from traitlets->ipython->viv-utils->floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): path.py>=6.2 in /usr/local/lib/python2.7/site-packages (from pickleshare->ipython->viv-utils->floss==1.0.3)
Requirement already satisfied (use --upgrade to upgrade): ptyprocess>=0.5 in /usr/local/lib/python2.7/site-packages (from pexpect->ipython->viv-utils->floss==1.0.3)
Installing collected packages: floss
  Running setup.py install for floss ... error
    Complete output from command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/var/folders/_1/5vtzjytj43b23bn1z3brkwnm0000gn/T/pip-FoQMBp-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/_1/5vtzjytj43b23bn1z3brkwnm0000gn/T/pip-fcmxMN-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib
    creating build/lib/floss
    copying floss/__init__.py -> build/lib/floss
    copying floss/ArgumentMonitor.py -> build/lib/floss
    copying floss/DecodingManager.py -> build/lib/floss
    copying floss/FunctionArgumentGetter.py -> build/lib/floss
    copying floss/interfaces.py -> build/lib/floss
    copying floss/main.py -> build/lib/floss
    copying floss/utils.py -> build/lib/floss
    running egg_info
    creating floss.egg-info
    writing requirements to floss.egg-info/requires.txt
    writing floss.egg-info/PKG-INFO
    writing top-level names to floss.egg-info/top_level.txt
    writing dependency_links to floss.egg-info/dependency_links.txt
    writing entry points to floss.egg-info/entry_points.txt
    writing manifest file 'floss.egg-info/SOURCES.txt'
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'floss.egg-info/SOURCES.txt'
    writing manifest file 'floss.egg-info/SOURCES.txt'
    error: can't copy 'floss/plugins': doesn't exist or not a regular file

    ----------------------------------------
Command "/usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/var/folders/_1/5vtzjytj43b23bn1z3brkwnm0000gn/T/pip-FoQMBp-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/_1/5vtzjytj43b23bn1z3brkwnm0000gn/T/pip-fcmxMN-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /var/folders/_1/5vtzjytj43b23bn1z3brkwnm0000gn/T/pip-FoQMBp-build/

test: encoded data on heap

make output function names consistent

print vs output etc.

add -v/--version mode to print the current version of FLOSS

so that a user can quickly confirm if they have a feature or not.
also to provide a link to the project and authors pages.

test: encoded blob has header with key

typedef struct ENCODED_STRING {
    unsigned char rc4_key[0x10];
    uint32_t length;
    unsigned char encoded_data[length;
} ENCODED_STRING;

can't use FLOSS as a library

the StringDecoder class directly parses sys.argv for arguments, which prevents a client program from using FLOSS as a library. an alternative would be to pass in the configuration as parameters, which would be initially fetched in the main() routine.

Filter minimum string length before output

If -n is specified, FLOSS might display the wrong string count.

Plugins are not installed

I tried to install this following the guide, but the plugins never got installed. Tried the detailed method as well, but the same happens. The floss/plugins directory is completely missing.

use argparse over optparse

argparse is the replacement for optparse.

it better handles both flags and positional arguments.

test: decoded data on stack

test: encoded data on stack

test: base64 decoding routine

test: encoded data in global memory

test: wrapped decoding functions

int decode(void *out, size_t out_len, const void *in, size_t in_len) {
    return Base64decode(out, in) == in_len;
}

identification plugins should find Base64decode while the function argument getter needs to extract args to decode.

function index key error

sample that starts with: d759966b88e

Traceback (most recent call last):
  File "/home/user/env/bin/floss", line 9, in <module>
    load_entry_point('floss', 'console_scripts', 'floss')()
  File "/home/user/src/flare-floss/floss/main.py", line 374, in main
    decoded_strings = decode_strings(vw, function_index, decoding_functions_candidates)
  File "/home/user/src/flare-floss/floss/main.py", line 33, in decode_strings
    for ctx in string_decoder.extract_decoding_contexts(vw, fva):
  File "/home/user/src/flare-floss/floss/string_decoder.py", line 15, in extract_decoding_contexts
    return get_function_contexts(vw, function)
  File "/home/user/src/flare-floss/floss/function_argument_getter.py", line 84, in get_function_contexts
    return FunctionArgumentGetter(vw).get_all_function_contexts(fva)
  File "/home/user/src/flare-floss/floss/function_argument_getter.py", line 45, in get_all_function_contexts
    for caller_va in self.get_caller_vas(function_va):
  File "/home/user/src/flare-floss/floss/function_argument_getter.py", line 57, in get_caller_vas
    caller_function_va = self.index[caller_va]  # the address of the function that contains this instruction
  File "/home/user/env/local/lib/python2.7/site-packages/viv_utils/__init__.py", line 162, in __getitem__
    raise KeyError()
KeyError

break up main routine into bitesized chunks

add documentation on the algorithm and how the tool works

Catch error if FLOSS is run on invalid file

Right now meaning non-PE file, for example:

$ floss data.txt 
Traceback (most recent call last):
  File "/tmp/flare-floss/local/lib/python2.7/site-packages/vivisect/base.py", line 499, in _fireEvent
    self.ehand[event](einfo)
  File "/tmp/flare-floss/local/lib/python2.7/site-packages/vivisect/base.py", line 343, in _handleSETMETA
    mcb(name, value)
  File "/tmp/flare-floss/local/lib/python2.7/site-packages/vivisect/base.py", line 545, in _mcb_Architecture
    self.setMemArchitecture(archid)
  File "/tmp/flare-floss/local/lib/python2.7/site-packages/envi/memory.py", line 78, in setMemArchitecture
    archmod = self.imem_archs[arch >> 16]
TypeError: unsupported operand type(s) for >>: 'NoneType' and 'int'
[...]

update readme to document cli args like -a and -q

test: "global" stackstrings

string manually created in global memory using single byte movs

Remove pVA prefix in string output

For decoded strings and stackstrings, e.g., sample c5de...868e

denote heap and stack locations in output (vs 0x69690000)

move string decoding functions out of main.py

main.py should contain the logic needed for the cli tool floss.exe, while the string decoding routines should be placed elsewhere. recommend using the file string_decoder.py.

ZeroDivisionError: float division by zero

Used the compiled version on Ubuntu, and get the following for the sample with MD5: 575372f2ad8021b9d12da1213bc1d1bf

./floss Order\ 150216.exe
WARNING:__main__.StringDecoderConfig:load_plugins: failed to load extra plugins: The 'floss' distribution was not found and is required by the application
Traceback (most recent call last):
  File "<string>", line 428, in <module>
  File "<string>", line 414, in main
  File "<string>", line 51, in identify_decoding_functions
  File "<string>", line 341, in run_plugins
  File "floss/plugins/function_meta_data_plugin.py", line 40, in score
ZeroDivisionError: float division by zero

Support stackstrings

see title

release v1.1

tag and package v1.1

features:

can use as library
static strings
quiet mode
updated documentation
simplified code layout

does FLOSS handle wrapped decoding functions?

consider the following

int decode(void *out, size_t out_len, const void *in, size_t in_len) {
    return Base64decode(out, in) == in_len;
}

identification plugins should find Base64decode while the function argument getter needs to extract args to decode.

document functions in main

this is so new contributors can more easily understand whats going on

make doc dir for detailed documentation, and make readme very concise

add mode to disable output formatting/headers

perhaps cli arg -q.

output will be just the strings, with no section headers or formatting. this makes the output appropriate for grepping or piping to other tools.

test: decoding routine fetches encoded data by string index

char *encoded_strings[] = {
    "string_one",
    "string_two",
    "string_three",
    "string_four",
};

char *decode(unsigned int string_index);

rename ArgumentDecoder.py, DecodingManager.py, etc to be snake case

for consistency.
i think this is standard python practice, but i'll have to double check.

test: decoded data in global memory

Extra plugins warning

The standalone executables display the following non-fatal warning at the start of execution:

WARNING:__main__.StringDecoderConfig:load_plugins: failed to load extra plugins: The 'floss' distribution was not found and is required by the application

full traceback, via @williamgibb:

ERROR:__main__.StringDecoderConfig:
Traceback (most recent call last):
  File "/Users/nope/somepath/flare-floss/floss/main.py", line 156, in load_plugins
    plugins_path = pkg_resources.resource_filename(req, requested_directory)
  File "/usr/local/var/pyenv/versions/2.7.9/Python.framework/Versions/2.7/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1151, in resource_filename
    return get_provider(package_or_requirement).get_resource_filename(
  File "/usr/local/var/pyenv/versions/2.7.9/Python.framework/Versions/2.7/lib/python2.7/site-packages/pkg_resources/__init__.py", line 422, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/usr/local/var/pyenv/versions/2.7.9/Python.framework/Versions/2.7/lib/python2.7/site-packages/pkg_resources/__init__.py", line 943, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/var/pyenv/versions/2.7.9/Python.framework/Versions/2.7/lib/python2.7/site-packages/pkg_resources/__init__.py", line 830, in resolve
    raise DistributionNotFound(req, requirers)
DistributionNotFound: The 'floss' distribution was not found and is required by the application
WARNING:__main__.StringDecoderConfig:load_plugins: failed to load extra plugins: The 'floss' distribution was not found and is required by the application

This is an issue that stems from trying to dynamically load plugins from resources packaged from PyInstaller.

Things to do:

suppress the warning for casual users.
fix the underlying issue. perhaps disable dynamic plugins from standalone exe.

test: local stackstrings

test: single-byte xor decoding routine

elf support

are "global stackstrings" seen in the wild?

malware that uses a stackstrings-like technique to initialize a global string will not be detected by the stackstrings extractor, since we currently inspect only the active stack frame. the decoding routine plugins also probably won't consider it suspicious.

so, is this something that's seen in the wild and worth supporting? leave examples of sample with this behavior as comments to this issue.

jay's script supports this type of decoding, since he heavily uses the vivisect emulator writelog to reconstruct strings. we could also enable the writelog and inspect memory regions written during a function's execution. part of me worries about performance, but i don't think it will be serious enough to matter.

todo:

dev/find test case, then
track global stackstrings

decode(s);
gethostbyname(s);
encode(s);