cmu-sei / pharos Goto Github PK

Automated static analysis tools for binary programs

License: Other

C++ 84.84% Makefile 0.06% CMake 0.94% Shell 0.08% Prolog 10.61% Python 3.10% Dockerfile 0.03% C 0.29% OpenEdge ABL 0.01% Emacs Lisp 0.04%

binary-analysis reverse-engineering

pharos's Introduction

Pharos Static Binary Analysis Framework

The Pharos static binary analysis framework is a project of the Software Engineering Institute at Carnegie Mellon University. The framework is designed to facilitate the automated analysis of binary programs. It uses the ROSE compiler infrastructure developed by Lawrence Livermore National Laboratory for disassembly, control flow analysis, instruction semantics, and more. This software is released under a BSD license.

The current distribution is a substantial update to the previous version, and adds a variety of features including improvements to the OOAnalyzer tool, experimental path analysis code, partitioner improvements, multi-threading, and many other smaller features.

The Pharos framework is a research project, and the code is undergoing active development. No warranties of fitness for any purpose are provided. While this release provides build instructions, unit tests, and some documentation, much work remains to be done. We've tested a few select build configurations, but we have not actively tested the portability of the source code. See the installation instructions for more details.

Since the primary objective for releasing this code is to provide transparency into our research and stimulate conversation with other binary static analysis researchers, please feel free to contact Cory Cohen [email protected] with questions you may have about this work. I may be unable to respond in a timely manner, but I will do my best.

Pharos Static Binary Analysis Tools

APIAnalyzer

ApiAnalyzer is a tool for finding sequences of API calls with the specified data and control relationships. This capability is intended to be used to detect common operating system interaction paradigms like opening a file, writing to it, and the closing it.

OOAnalyzer

OOAnalyzer is a tool for the analysis and recovery of object oriented constructs. This tool was the subject of a paper titled "Using Logic Programming to Recover C++ Classes and Methods from Compiled Executables" which was published at the ACM Conference on Computer and Communications Security in 2018. The tool identifies object members and methods by tracking object pointers between functions in the program. A previous implementation of this tool was named "Objdigger", but it was renamed to reflect a substantial redesign using Prolog rules to recover the object attributes. The current version of the tool only supports analysis of 32-bit x86 executables compiled by Microsoft Visual C++. For more detailed instructons on how to run OOAnalyzer on very large executables, see these notes.

The Pharos distribution used to include a plugin that imported OO information exported by OOAnalayzer into the Ghidra reverse engineering tool set. To get that functionality now and in the future, install the Kaiju Ghidra plugin, which includes the functionality that was provided by the OOAnalayzer plugin.

CallAnalyzer

CallAnalyzer is a tool for reporting the static parameters to API calls in a binary program. It is largely a demonstration of our current calling convention, parameter analysis, and type detection capabilities, although it also provides useful analysis of the code in a program.

FN2Yara

FN2Yara is a tool to generate YARA signatures for matching functions in an executable program. Programs that share significant numbers of functions are are likely to have behavior in common.

FN2Hash

FN2Hash is tool for generating a variety of hashes and other descriptive properties for functions in an executable program. Like FN2Yara it can be used to support binary similarity analysis, or provide features for machine learning algorithms.

DumpMASM

DumpMASM is a tool for dumping disassembly listings from an executable using the Pharos framework in the same style as the other tools. It has not been actively maintained, and you should consider using ROSE's standard recursiveDisassemble instead http://rosecompiler.org/ROSE_HTML_Reference/rosetools.html.

pharos's People

Contributors

Stargazers

Watchers

Forkers

sei-ccohen chubbymaggie amesianx kursh avgirl uxmal aziem techlord-rce matzke1 brolerbin threatinteltest abuuuu hari5u alecakin ufwt thawsystems irivera007 wilsonfisk fourks nsxz jymcheong cacker cseq 4144 aln7 mewbak h0k5 sbambach charloco awesome-security dsbeaver de8gman1990 johnjohnsp1 4n6strider m41doror securitywarrior moriarty2016 grukz expertasif pentestingtools daodaoliang koncybernet ykankaya m00zh33 exh gavz mgq812 victorhdamian cryp706uru manishshukla d-ned edmcman step9 meesong sigma-random fcccode amattoo heruix henbar-hust potatogim 0xpit lallouslab 0xm3r 4b5f5f4b ckxckx bama4 wjrsonic 0x6b7966 trass3r wyu0hop certainlach loopyashy kuhar rocker9527 illbebach halvors janwielemaker boryotto uditagarwal97 softearth seabreg feiyunwill alex5160d joshroark bronxc stlcours morgan-etheredge bb33bb rommate chenghongm dberlin crackercat hello-xbugs olegjakushkin wannamak k3rnelninja happydpc 5l1v3r1 olaughlj smuthayan

pharos's Issues

add --delete-method option

PoC:

diff --git a/libpharos/ooanalyzer.cpp b/libpharos/ooanalyzer.cpp
index e656ea4..978bdb6 100644
--- a/libpharos/ooanalyzer.cpp
+++ b/libpharos/ooanalyzer.cpp
@@ -28,6 +28,7 @@ OOAnalyzer::OOAnalyzer(DescriptorSet& ds_, const ProgOptVarMap& vm_, AddrSet& ne
   delete_methods_found = 0;
   purecall_methods_found = 0;
   new_addrs = new_addrs_;
+  delete_addrs = option_addr_list(vm, "delete-method");
 
   // Initialize the new_hashes string set with the hashes of known methods.
   initialize_known_method_hashes();
diff --git a/tools/ooanalyzer/ooanalyzer.cpp b/tools/ooanalyzer/ooanalyzer.cpp
index 71c2aa1..19e395f 100644
--- a/tools/ooanalyzer/ooanalyzer.cpp
+++ b/tools/ooanalyzer/ooanalyzer.cpp
@@ -26,6 +26,9 @@ ProgOptDesc digger_options() {
     ("new-method,n",
      po::value<StrVector>(),
      "function at address is a new() method")
+    ("delete-method",
+     po::value<StrVector>(),
+     "function at address is a delete() method")
     ("no-guessing",
      "do not perform hypothetical reasoning.  never use except for experiments")
     ("ignore-rtti",

global destructors

I saw functions being marked as constructors which are actually destructors.
It could have been deduced from atexit usage: https://godbolt.org/z/rndZOs
(Also in this particular case the function called operator delete which shouldn't happen in a ctor.)

FSEM[ERROR]: Function 0x00000940 has no out edges.

I got such error when I run the testcase ooex5.cpp from dir pharos/src, any suggestions would be greatly appreciated.

icy@ubuntu:~/cpp_test$ ooanalyzer --json ooex5.json ./ooex5 --allow-64bit
OPTI[INFO ]: Analyzing executable: ./ooex5
OPTI[INFO ]: OOAnalyzer version 0.13.
OPTI[INFO ]: ROSE stock partitioning took 0.687482 seconds.
OPTI[INFO ]: Long delays until the next time stamp are caused by the Pharos custom partitioning
OPTI[INFO ]: algorithm and may be resolved by using the --stockpart option, but at the expense
OPTI[INFO ]: of possibly less complete function detection. Using --no-semantics may also help.
OPTI[INFO ]: Function partitioning took 0.829896 seconds.
FSEM[ERROR]: Function 0x00000940 has no out edges.
OPTI[ERROR]: bad operand size found @ 0x000009D3 : 0
OPTI[ERROR]: bad operand size found @ 0x00000D19 : 0
OOAN[ERROR]: No new() methods were found. Heap objects may not be detected.
OOAN[ERROR]: No delete() methods were found. Object analysis may be impaired.
OPTI[INFO ]: Function analysis complete, analyzed 36 functions in 0.354419 seconds.
OPTI[ERROR]: No C++ classes were detected in the program.
OPTI[INFO ]: OOAnalyzer analysis complete.

Import into eclipse

The GhidraDev plugin does not have an import option for the project. Eclipse does not recognize the plugin directory as a project. How are you importing this into Eclipse to build the plugin?

object arrays

msvc transforms something like

struct Foo
{
	Foo();
	~Foo();

	int a[20] = {};
};
return new Foo[50];

into

t = (uint32_t*)operator new(4004);
if (!t)
    return 0;
*t = 50;
a = t + 1;
eh_vector_constructor(a, 80, 50, Foo::Foo, guard_check_icall_nop);
return a;

https://www.geoffchappell.com/studies/msvc/language/compgen/_j.htm

Looks like pharos can't handle that case yet.

This pattern can also arise inside of a constructor if you have an array of objects as part of another object: https://godbolt.org/z/Ml61jT

OOAnalyzer doesn't support kext files

Used revision: 2ff59e7

$ ooanalyzer -j test.json test
OPTI[INFO ]: Analyzing executable: test
OPTI[INFO ]: OOAnalyzer version 1.0.
OOAN[FATAL]: Pharos main error: unrecognized file format for "test": Mach-O 64-bit x86_64 kext bundle, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL>
$

apianalyzer all api sequences

Can not you extract the entire sequence of api that runs as the program runs in apianalyzer? Without exactly defined api signature

switch table confused as function start

https://godbolt.org/z/4yL7RO

// cl -nologo -O2 -GS- -Gy- -Z7 main.cpp -MD -link /opt:ref
struct A
{
	int b = 4;
	int foo(int num);
	int bar(int a);
};

int A::foo(int num)
{
	switch (num)
	{
	case 0:
	case 1:
	case 12:
	case 3:
	case 22:
	case 13:
	case 20:
		return num;
		break;
	case 8:
	case 11:
	case 17:
	case 4:
	case 10:
		return num;
		break;
	}
	return num * num;
}
int A::bar(int a) { return b * a; }
int main(int argc, const char* argv[]) { return ((A*)argv)->foo(5); }

Produces a switch table between foo and bar.

.text:00401028                 db      0,     1,     0
.text:0040103F                 align 10h
.text:00401040 A::bar       proc

But bar is recognized by pharos as starting at offset 40103c.

libboost-all-dev pulls in too many dependencies

We can probably cut down on the Docker image size by explicitly enumerating the boost libraries we need.

x87 support

Should be a Rose issue so this is just to document this. Not sure if it's actually limiting pharos' analysis.

48.16040s FSEM[WARN ]: Semantics exception: no dispatch ability for "fild" instruction: 4152BD: fild      [esp+0]
48.16092s FSEM[WARN ]: Semantics exception: no dispatch ability for "fdivp" instruction: 4152C1: fdivp     (1), (0)
48.16153s FSEM[WARN ]: Semantics exception: 64-bit FP values not supported yet: 4152C3: fstp      [esp+0]
48.16400s FSEM[WARN ]: Replaced excessively large expression with v18926212[1]<unspec,f=10000>

Btw the only other unsupported instructions I've seen so far are:
https://www.felixcloutier.com/x86/sahf
https://www.felixcloutier.com/x86/xlat:xlatb

Unable to load API database: /usr/local/share/pharos/configs/shell32.json

Operating system used to compile and run Pharos Tools
Kali Linux 2017.2
Got the following error when analyzing a PE32 executable file

/* CALA[ERROR]: Unable to load API database: /usr/local/share/pharos/configs/msvcrt.json
CALA[ERROR]: Unable to load API database: /usr/local/share/pharos/configs/winmm.json
CALA[ERROR]: Unable to load API database: /usr/local/share/pharos/configs/imm32.json
CALA[ERROR]: Unable to load API database: /usr/local/share/pharos/configs/shell32.json */

no file (shell32.json etc) in the configs directory

Plugin not working ghidra version 9.1.1

Hey,

I have compiled the OOAnalyzer plugin successfully for ghidra, imported the plugin but it seems that ghidra doesn't recognize the plugin so the CERT dropdown is not available.

Any guidelines?

Regards!

Docker use ubuntu:latest cause error on package not exist

Hi,

Your docker file use image of "ubuntu:latest" but i think than image changed since you make Dockerfile (xenial to bionic). Bionic version have packages that not disponible. For fix fastest, you change FROM ubuntu:latest to FROM ubuntu:xenial.

Thanks
Lionel

Extension version for <OOAnalyzerPlugin> is incompatible with Ghidra 9.1.2

Am getting with error message when i try to install the plugin

OOAnalyzer reports DLL built by Visual C++ 6.0 contains no classes

I'm attempting to analyze a DLL I know to have been written in C++ and compiled by Visual C++ 6.0 (is this supported?), but OOAnalyzer reports that it doesn't contain any classes at all (disproven by basic examination in Ghidra, as well as a copy of the linker mapfile).

Output from analysis as below:

OPTI[INFO ]: Analyzing executable: engine-netbabel.dll
OPTI[INFO ]: OOAnalyzer version 1.0.
OPTI[INFO ]: ROSE stock partitioning took 70.525 seconds.
OPTI[INFO ]: Partitioned 187777 bytes, 67592 instructions, 15381 basic blocks, 3 data blocks and 1092 functions.
OPTI[INFO ]: Pharos function partitioning took 93.7035 seconds.
OPTI[INFO ]: Partitioned 212992 bytes, 71498 instructions, 17081 basic blocks, 1112 data blocks and 2492 functions.
[INFO ]: Function 0x10005080 has no out edges.
OOAN[ERROR]: Found only 2266 functions of 2268 specifically requested for analysis.
OPTI[INFO ]: Function analysis complete, analyzed 2266 functions in 61.6782 seconds.
PLOG[FATAL]: No complete solution was found!
OPTI[ERROR]: No C++ classes were detected in the program.
OPTI[INFO ]: Successfully exported to JSON file 'engine-netbabel.json'.
OPTI[INFO ]: OOAnalyzer analysis complete.

It's a release binary, with no embedded debugging info (and obviously it's an x86 Win32 binary).

use RTTI inheritance data

Seems like RTTI Class Hierarchy Descriptor / RTTI Base Class Array information is not used to merge facts about object layout. And after export related classes are independent instead of reusing base class objects.

Ghidra plugin Bug

when the config file's vftable include a null entries, json parse Exception with "Expected BEGIN_ARRAY but was STRING at path $[0].entries" and thus return a null vftList.
field cls3 for example:

{"Name":".?AVCls3@@","DemangledName":"Cls3","Size":"84","Members":[{"name":"vfptr_0","type":"vfptr","offset":"0","count":"1"},{"name":"vfptr_c","type":"vfptr","offset":"c","count":"1"},{"name":"mbr_50","type":"dword","offset":"50","count":"1"},{"name":".?AVCls1@@_0","type":"struc","struc":".?AVCls1@@","parent":"yes","offset":"0","count":"1"},{"name":".?AVCls2@@_c","type":"struc","struc":".?AVCls2@@","parent":"yes","offset":"c","count":"1"}],"Methods":[{"ea":"412930","name":"ctor_412930","demangled_name":"","import":"no","type":"ctor"}],"Vftables":[{"ea":"41dcd0","vfptr":"0","entries":[{"ea":"4149f0","offset":"0","name":"virt_meth_4149f0","demangled_name":"","import":"no","type":"meth"},{"ea":"414b70","offset":"3","name":"virt_meth_414b70","demangled_name":"","import":"no","type":"meth"}]},{"ea":"41dcdc","vfptr":"12","entries":""}]}

vftable parse occur some error: scom.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_ARRAY but was STRING at path $[1].entries

line /pharos/tools/ooanalyzer/ghidra/OOAnalyzerPlugin/src/main/java/ooanalyzer/OOAnalyzer.java:1508 cause this issue.

ooanalyzer timeouts

When not setting any timeouts via the commandline some functions still time out:

Function 0x004508A0 relative wall clock exceeded
Function analysis convergence failed for: 0x004508A0
Function 0x0040D020 relative memory exceeded

This is quite weird looking at

pharos/libpharos/limit.cpp

Line 56 in 70fac02

if (relative_clock_limit > duration() || absolute_clock_limit > duration()) {

pharos/libpharos/limit.cpp

Line 38 in 70fac02

set_clock_limits(duration(), duration());

how it work ??

i need tutorial ??

Lots of error "OOAN[ERROR]: update_return_values, no output state for 0x004CFC00"

I don't know what this error means. The software I tested can be download from the link blow:

https://drive.google.com/file/d/1VgdH0IJguqLjyT_m-b-CCHAN9slbZ4Yk/view?usp=sharing

icy@ubuntu:$ ooanalyzer -j yundetectservice.json yundetectservice.exe
OPTI[INFO ]: Analyzing executable: yundetectservice.exe
OPTI[INFO ]: OOAnalyzer version 0.13.
OPTI[INFO ]: ROSE stock partitioning took 314.121 seconds.
OPTI[INFO ]: Long delays until the next time stamp are caused by the Pharos custom partitioning
OPTI[INFO ]: algorithm and may be resolved by using the --stockpart option, but at the expense
OPTI[INFO ]: of possibly less complete function detection. Using --no-semantics may also help.
OPTI[INFO ]: Function partitioning took 2117.42 seconds.
OAN[ERROR]: update_return_values, no output state for 0x00401000
...
OOAN[ERROR]: update_return_values, no output state for 0x004CFC00
OOAN[ERROR]: No new() methods were found. Heap objects may not be detected.
OOAN[ERROR]: No delete() methods were found. Object analysis may be impaired.
OPTI[INFO ]: Function analysis complete, analyzed 6037 functions in 49.2624 seconds.
OPTI[INFO ]: Successfully exported to JSON file 'yundetectservice.json'.
OPTI[INFO ]: OOAnalyzer analysis complete.

Memory limits

I am having a hard time getting ooanalyzer to finish.

This is what i get:

[28839601.841345] Out of memory: Kill process 30340 (ooanalyzer) score 736 or sacrifice child
[28839601.842291] Killed process 30340 (ooanalyzer) total-vm:18192656kB, anon-rss:13268556kB, file-rss:4kB

This is how i execute it:

ooanalyzer --json result.json -f Sample.exe --serialize Sample.exe.rose --maxmem 4096 --relmaxmem 500

How am i supposed to use these memory limits? They do not seem to work.

Recursive clone errors

See the end of the log.

~/docker % git clone --recursive https://github.com/cmu-sei/pharos.git
Cloning into 'pharos'...
remote: Enumerating objects: 1038, done.
remote: Total 1038 (delta 0), reused 0 (delta 0), pack-reused 1038
Receiving objects: 100% (1038/1038), 8.16 MiB | 10.08 MiB/s, done.
Resolving deltas: 100% (447/447), done.
Submodule 'rose' (https://github.com/rose-compiler/rose-develop) registered for path 'rose'
Submodule 'z3' (https://github.com/Z3Prover/z3.git) registered for path 'z3'
Cloning into '/home/rk/docker/pharos/rose'...
remote: Enumerating objects: 2531, done.        
remote: Counting objects: 100% (2531/2531), done.        
remote: Compressing objects: 100% (1224/1224), done.        
remote: Total 318158 (delta 1658), reused 1770 (delta 1252), pack-reused 315627        
Receiving objects: 100% (318158/318158), 373.06 MiB | 15.32 MiB/s, done.
Resolving deltas: 100% (238823/238823), done.
Cloning into '/home/rk/docker/pharos/z3'...
remote: Enumerating objects: 113, done.        
remote: Counting objects: 100% (113/113), done.        
remote: Compressing objects: 100% (67/67), done.        
remote: Total 86941 (delta 47), reused 70 (delta 44), pack-reused 86828        
Receiving objects: 100% (86941/86941), 36.03 MiB | 18.15 MiB/s, done.
Resolving deltas: 100% (71038/71038), done.
Submodule path 'rose': checked out 'd3eaef2ad21687c294827d4471f2b0163af86978'
Submodule 'projects/vulnerabilitySeeding' ([email protected]:rose/vulnerabilitySeeding.git) registered for path 'rose/projects/vulnerabilitySeeding'
Submodule 'src/frontend/CxxFrontend/EDG' ([email protected]:rose/edg4x/edg.git) registered for path 'rose/src/frontend/CxxFrontend/EDG'
Cloning into '/home/rk/docker/pharos/rose/projects/vulnerabilitySeeding'...
ssh: Could not resolve hostname rosecompiler1.llnl.gov: No address associated with hostname
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:rose/vulnerabilitySeeding.git' into submodule path '/home/rk/docker/pharos/rose/projects/vulnerabilitySeeding' failed
Failed to clone 'projects/vulnerabilitySeeding'. Retry scheduled
Cloning into '/home/rk/docker/pharos/rose/src/frontend/CxxFrontend/EDG'...
ssh: Could not resolve hostname rosecompiler1.llnl.gov: No address associated with hostname
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:rose/edg4x/edg.git' into submodule path '/home/rk/docker/pharos/rose/src/frontend/CxxFrontend/EDG' failed
Failed to clone 'src/frontend/CxxFrontend/EDG'. Retry scheduled
Cloning into '/home/rk/docker/pharos/rose/projects/vulnerabilitySeeding'...
ssh: Could not resolve hostname rosecompiler1.llnl.gov: No address associated with hostname
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:rose/vulnerabilitySeeding.git' into submodule path '/home/rk/docker/pharos/rose/projects/vulnerabilitySeeding' failed
Failed to clone 'projects/vulnerabilitySeeding' a second time, aborting
Cloning into '/home/rk/docker/pharos/rose/src/frontend/CxxFrontend/EDG'...
ssh: Could not resolve hostname rosecompiler1.llnl.gov: No address associated with hostname
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:rose/edg4x/edg.git' into submodule path '/home/rk/docker/pharos/rose/src/frontend/CxxFrontend/EDG' failed
Failed to clone 'src/frontend/CxxFrontend/EDG' a second time, aborting
Submodule path 'z3': checked out 'b81165167304c20e28bc42549c94399d70c8ae65'
Failed to recurse into submodule path 'rose'

Working with Ghidra. json

How do I use this tool on windows?
I built the ghidra plugin but i could do nothing else without the json file.

reserved identifier violation

I would like to point out that identifiers like “_APIGRAPH_H_” and “_MATCHERTAGS_HPP_” do not fit to the expected naming convention of the C++ language standard.
Would you like to adjust your selection for unique names?

Very long analysis and important drift between expected blocks & functions counts

Hi,

So I had this one running for 9 hours now and while it seemed to progress very fast until 90% mark, it's now progressing quite very slow. what strikes me if the different between expected bytes, blocks and functions which more than doubled for functions for example:

root@339ab94ee21f:/tmp/share# ooanalyzer --maximum-memory 0 --per-function-timeout 0 --partitioner-timeout 0 --timeout 0 --json ida.json -F facts.txt -R results.txt xxxx.exe
OPTI[INFO ]: Analyzing executable: exe.exe
OPTI[INFO ]: OOAnalyzer version 1.0.
OPTI[INFO ]: ROSE stock partitioning took 5317.74 seconds.
OPTI[INFO ]: Partitioned 14033584 bytes, 4963808 instructions, 994447 basic blocks, 534 data blocks and 76710 functions.
PRT2[MARCH]: cfg: 91% [##############-] 16968921 bytes 1209943 blks 172878 funcs

Now at every tick I can see 5-15 bytes were processed and functions count increments by one as well as block count and memory consumption is progressing very slowly as well.

I am running latest version your docker auto build.

Is there anything wrong? At that rate I did calculate that it would finish a few weeks. Thank you in advance for your advices.

prebuilt binaries

Would be nice to have. An occasional binaries drop as a Github release would be enough.

Pharos Slow Build On Ubuntu 18.04

I am attempting to build pharos in a VmWare Ubuntu 18.04 VM. The build takes over 1 day and then starts slowing down to the point where the VM freezes. I have 4 GB RAM and 250 GB hard drive for this VM. None of the builds have completed. I tried building on Ubuntu 16.04 as well and had the same issue.

Can't build ROSE with ninja

root@c95617ca8a3b:~/pharos/scripts/rose/release# (cd /root/pharos/scripts/rose/release/src/ROSETTA/src && /usr/bin/cmake -E copy_directory /root/pharos/scripts/rose/src/ROSETTA/Grammar /root/pharos/scripts/rose/release/src/ROSETTA/Grammar && /usr/bin/cmake -E make_directory /root/pharos/scripts/rose/release/src/frontend/SageIII/astFileIO && /root/pharos/scripts/rose/release/bin/CxxGrammarMetaProgram /root/pharos/scripts/rose/release/src/frontend/SageIII/)
ERROR: File not found -- ../src/ROSETTA/src/binaryInstruction.C

root@c95617ca8a3b:~/pharos/scripts/rose/release/src/ROSETTA/src# /root/pharos/scripts/rose/release/bin/CxxGrammarMetaProgram /root/pharos/scripts/rose/release/src/frontend/SageIII/
ERROR: File not found -- ../src/ROSETTA/src/binaryInstruction.C 
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >'
Aborted
root@c95617ca8a3b:~/pharos/scripts/rose/release/src/ROSETTA/src# less ../src/ROSETTA/src/binaryInstruction.C
../src/ROSETTA/src/binaryInstruction.C: No such file or directory
root@c95617ca8a3b:~/pharos/scripts/rose/release/src/ROSETTA/src# find ../../../ -name binaryInstruction.C
root@c95617ca8a3b:~/pharos/scripts/rose/release/src/ROSETTA/src# find ../../../.. -name binaryInstruction.C
../../../../src/ROSETTA/src/binaryInstruction.C

So it looks like when using Ninja, binaryInstruction.C is not copied into the build directory for some reason.

Duplicate merging of classes

Background information:
I am processing a 7 MB file. Around 68k functions.
I serialized the partitioning on a server, using 72 threads; then took the 12GB file to another server and resumed from here (using 1 core as the analysis can't be parallel).
Before canceling ooanalyzer on that first server, it was using 140GB of ram (was my limit). On the second server, I set no limit, using a 300GB swap, but is currently using only 47GB of ram (does the first pass does not release memory correctly or is it related to threads? But that's not my issue here)
I am using the eschwartz/pharos docker image with tag 089f94992923 from 3 or 4 weeks ago.
It has been processing for around 14 days at the moment.

Issue:
Even if it seems like I can't set the log level greater than 4 (crashes, can't remember failure, but can find it if needed), logs display the merging of classes.
I can see that some classes are merged into the same other one a couple of time. I am wondering if this is a wanted behavior or not.

You can check the attached log and look for the string "Merging class 0x71ad00 into 0x854be0" and see that it happened 10 times these last 5 days.

logs_ooanalyzer.txt

verbose mode does not show progress

Usually you get a sense of progress out of the "processing function %u/%u" messages. But with -v3 or -v4 it only prints "Processing function %p"

Also there are no timestamps / time remaining according to the limits shown.

Suboptimal build configuration

There is this line in build.bash :

NCPU="${NCPU:-1}"

I'm no bash expert, but I noticed that building the Docker container didn't fully utilize my CPU, so I did a test:

$ echo "${NCPU:-1}"
1

Based on this comment (and reason) NCPU is supposed to be set to -1, but as you can see above, it is set to 1 instead. This SO answer also confirms that the above code is semantically incorrect. The following code gives the expected result on my system (note the double dash):

$ echo "${NCPU:--1}"
-1

Errors while running CTest

Operating system used to compile and run Pharos Tools
Kali Linux 2017.2
Ctest result
73% tests passed, 38 tests failed out of 139

FAILED tests

ooanalyzer crash + remarks

Hello,
I wanted to try OOanalyzer, using the docker file, here is the result:

OOAN[ERROR]: No new() methods were found.  Heap objects may not be detected.
OOAN[ERROR]: No delete() methods were found.  Object analysis may be impaired.
OPTI[INFO ]: Function analysis complete, analyzed 491 functions in 12.6402 seconds.
OOAN[FATAL]: Unable to start Prolog session.
OOAN[FATAL]: XSB Initialization Error: init_error/XSB configuration file /root/pharos/scripts/build/XSB/xsb-3.8.0/config/x86_64-unknown-linux-gnu/lib/xsb_configuration.P does not exist or is not readable by you.
Segmentation fault (core dumped)

Besides, getting pharos to run is a pain:

building from source did not work (Z3 linking errors)
the docker build took 2.5 hours

Maybe you could provide either:

static (or partially static) binaries
pre-built docker images.

Also, is OOAnalyzer.py compatible with IDA 7 ?

OOAnalyzer plugin will not compile with Ghidra 9.1

The OOAnalyzer Ghidra plugin doesn't seem to compile with Ghidra 9.1 The same environment builds against Ghidra 9.04 just fine.

gradle -PGHIDRA_INSTALL_DIR=/home/tapioca/in/ghidra_9.1_PUBLIC

> Task :compileJava FAILED
/home/tapioca/in/pharos/tools/ooanalyzer/ghidra/OOAnalyzerPlugin/src/main/java/ooanalyzer/OOAnalyzer.java:1406: error: no suitable constructor found for OptionDialog(String,String,String,int,<null>)
                                var contDialog = new OptionDialog("Careful",
                                                 ^
    constructor OptionDialog.OptionDialog(String,String,int,Icon) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,String,int,Icon,boolean) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,String,int,Icon,boolean,String) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,int,Icon,String) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,String,String,int,Icon,boolean) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,int,Icon,boolean,DialogRememberOption,List<String>,String) is not applicable
      (actual and formal argument lists differ in length)
/home/tapioca/in/pharos/tools/ooanalyzer/ghidra/OOAnalyzerPlugin/src/main/java/ooanalyzer/OOAnalyzerGhidraPlugin.java:64: error: no suitable constructor found for OptionDialog(String,String,int,<null>)
                        new OptionDialog("Error", "Invalid JSON file", OptionDialog.ERROR_MESSAGE, null).show();
                        ^
    constructor OptionDialog.OptionDialog(String,String,String,String,int,Icon,boolean) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,String,int,Icon,boolean,String) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,int,Icon) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,int,Icon,String) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,String,String,int,Icon,boolean) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,int,Icon,boolean,DialogRememberOption,List<String>,String) is not applicable
      (actual and formal argument lists differ in length)
/home/tapioca/in/pharos/tools/ooanalyzer/ghidra/OOAnalyzerPlugin/src/main/java/ooanalyzer/OOAnalyzerGhidraPlugin.java:74: error: no suitable constructor found for OptionDialog(String,String,int,<null>)
                        new OptionDialog("Error", "Could not load/parse JSON file " + jsonFile.getName(),
                        ^
    constructor OptionDialog.OptionDialog(String,String,String,String,int,Icon,boolean) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,String,int,Icon,boolean,String) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,int,Icon) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,int,Icon,String) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,String,String,int,Icon,boolean) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,int,Icon,boolean,DialogRememberOption,List<String>,String) is not applicable
      (actual and formal argument lists differ in length)
/home/tapioca/in/pharos/tools/ooanalyzer/ghidra/OOAnalyzerPlugin/src/main/java/ooanalyzer/OOAnalyzerGhidraPlugin.java:87: error: no suitable constructor found for OptionDialog(String,String,int,<null>)
                                                new OptionDialog("Error", "No current program for OOAnalyzer.", OptionDialog.ERROR_MESSAGE,
                                                ^
    constructor OptionDialog.OptionDialog(String,String,String,String,int,Icon,boolean) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,String,int,Icon,boolean,String) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,int,Icon) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,int,Icon,String) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,String,String,int,Icon,boolean) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,int,Icon,boolean,DialogRememberOption,List<String>,String) is not applicable
      (actual and formal argument lists differ in length)
/home/tapioca/in/pharos/tools/ooanalyzer/ghidra/OOAnalyzerPlugin/src/main/java/ooanalyzer/OOAnalyzerGhidraPlugin.java:90: error: no suitable constructor found for OptionDialog(String,String,int,<null>)
                                                new OptionDialog("Results", "OOAnalyzer loaded " + result + " classes.", 
                                                ^
    constructor OptionDialog.OptionDialog(String,String,String,String,int,Icon,boolean) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,String,int,Icon,boolean,String) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,int,Icon) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,int,Icon,String) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,String,String,int,Icon,boolean) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,int,Icon,boolean,DialogRememberOption,List<String>,String) is not applicable
      (actual and formal argument lists differ in length)
/home/tapioca/in/pharos/tools/ooanalyzer/ghidra/OOAnalyzerPlugin/src/main/java/ooanalyzer/OOAnalyzerGhidraPlugin.java:93: error: no suitable constructor found for OptionDialog(String,String,int,<null>)
                                                new OptionDialog("Results", "OOAnalyzer could not load any classes",
                                                ^
    constructor OptionDialog.OptionDialog(String,String,String,String,int,Icon,boolean) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,String,int,Icon,boolean,String) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,int,Icon) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,int,Icon,String) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,String,String,String,int,Icon,boolean) is not applicable
      (actual and formal argument lists differ in length)
    constructor OptionDialog.OptionDialog(String,String,int,Icon,boolean,DialogRememberOption,List<String>,String) is not applicable
      (actual and formal argument lists differ in length)
6 errors


FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 1s
2 actionable tasks: 1 executed, 1 up-to-date

OOAnalyzer plugin

  File OOAnalyzer.py, line 797, in apply_all_structs
    c.id = self.__apply_class(c)
  File OOAnalyzer.py, line 573, in __apply_class
    u.apply()
  File OOAnalyzer.py, line 937, in apply
    idc.OpStroffEx(self.__ea, n, self.__cid, 0)
  File idc_bc695.py, line 107, in OpStroffEx
    def OpStroffEx(ea, n, strid, delta): return op_stroff(ea, n, strid, delta)
  File idc.py, line 1267, in op_stroff
    return ida_bytes.op_stroff(ea, n, path.cast(), 1, delta)
  File ida_bytes.py, line 1007, in op_stroff
    return _ida_bytes.op_stroff(*args)
TypeError: in method 'op_stroff', argument 1 of type 'insn_t const &'

Is it incompatible with the v7 API?

resume after analysis

Is it possible to resume after analysis has finished?
I had to stop after it dumped the Prolog facts file.

how can i run pharos

what is command to start ??

installation problem

I installed it using command "$ docker build -t pharos .",
but it finally returned
"The command '/bin/sh -c /root/pharos/scripts/build.bash -reclaim && rm -rf /root/pharos && cd /usr/local/lib && find /usr/local/lib | xargs file | grep 'current ar archive' | awk -F':' '{print $1}' | xargs strip' returned a non-zero code: 2"
my OS is ubuntu 16.04, and I have tried it for many times, the results is same :(
is that because I'm lake of some dependencies?

Some polymorphic calls are not recognizable

When I test file /pharos/src/oo.cpp, OOAnalyzer unable to recognize some calls to vtable functions.More specifically,

Calls e->func3(), e->func4() and f->func1() not recognized by OOAnalyzer cause the config file seems like below:

Only vcall e->func1() can be recognized.

Support for Linux 64 bit ELF files

From scanning through other issues it seems this may be something that is being actively worked. This would be a wonderful feature. Any chance there is a ball park estimate for when this feature would be added? I refer particularly to the OOanalyzer tool.

Mach-O x86_64 Support?

# ooanalyzer --allow-64bit macho_binary --json macho_binary.json
OPTI[INFO ]: Analyzing executable: macho_binary
OPTI[INFO ]: OOAnalyzer version 1.0.
OOAN[FATAL]: Pharos main error: unrecognized file format for "macho_binary": Mach-O 64-bit x86_64 bundle, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|WEAK_DEFINES|BINDS_TO_WEAK>

It’s not clear to me from the documentation I can find whether or not macOS binaries are meant to be supported, or whether or not I’m doing something wrong. Are they? Am I?

Update?

Hi. I tried to install all by your guides. I have tried docker and bash script. Both ways are failed. Can you please update scripts?
For example, rose-develop is not downloaded correctly by GIt, only one file was downloaded, it says, that develop branch is not used, release shoud be used. I tried fix this, but in release ./build fails with "aclocal" and etc...

parallel analysis

Would it be possible to parallelize the function analysis?
Even for an exe from the 90s with only 2-3MB it takes lots of hours.

Default resource limits are unexpected and poorly communicated

The default timeouts continue to be a source of confusion for users. We should probably make them longer by default, and resource constrained users can set them shorter as desired. At the very least the error messages should be improved.

no warning for missing API functions

As soon as there is a single function declaration for a particular dll in your json DB it does not warn anymore for any other missing functions. Not sure if that's intended.
(Also, is there a tool to generate a json db from header files?)

Windows support

Has anyone ever tried to compile it for Windows?

Add Cutter/Rizin/Radare2 plugins

Since you have now plugins for IDA Pro and GHIDRA, what about doing that for rizin/Cutter too?

Rizin is a highly-portable cross-platform reverse engineering framework and a toolkit without dependencies. It has support for analyzing binaries, disassembling code, debugging programs, attaching to remote GDB/LLDB, WinDbg servers, rich plugin system (see rz-pm), and integration with various decompilers. For example, ghidra decompiler plugin - rz-ghidra. It is actively developed and can be easily integrated in various open source and commercial products.

For documentation on writing plugins for rizin see Scripting and Plugins Rizin Book chapters.

Cutter is a crossplatform Qt/C++ GUI frontend to rizin:

For documentation on writing plugins for Cutter see the official tutorial and the curated list of various popular plugins.

git clone --depth 1 -b z3-4.8.6 https://github.com/Z3Prover/z3.git z3

the capital Z results in an error.

warning: Could not find remote branch Z3-4.8.6 to clone. fatal: Remote branch Z3-4.8.6 not found in upstream origin

thx