joxeankoret / pyew Goto Github PK

I found an strange behaviour while changing pyew.bsize value.

After the change of the block size, it won't be applied until a command like 
'g' or 's' is used. Attached there is a txt file with an example of the bug.

Regards

Hello every one and authors of pyew tool.
I read the general information of pyew tool and know that pyew 2.2 support 
debugger feauture.
i am doing research on malware analize and i need this feature to monitor virus 
behavior which be shown on variable value change.
Can you help me?
Where i can get this version of pyew tool. Because in the project home site, i 
can not find it.
This feature is very important and neccessary for my graduate thesis.
Thank you very much. 

Some URLs return 404 using wget, but chkurl still reports those as OK. Have you 
noticed this problem?

What steps will reproduce the problem?
1. Trying to load an ELF 64 bit binary with relocatable section
2. For example:

Platform info:
{{{
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 10.04 LTS
Release:    10.04
Codename:   lucid
$ arch
x86_64
$ file /bin/ls
/bin/ls: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically 
linked (uses shared libs), for GNU/Linux 2.6.15, stripped
}}}

Python interactive repeat-by
{{{
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyew.pyew_core import CPyew
>>> pyew = CPyew(batch=True)
>>> pyew.loadFile("/bin/ls")
}}}

results in an traceback that ends with:
{{{
/usr/local/python_environments/map/pyew/Elf/__init__.py in 
parseRelocs(self=<pyew.Elf.Elf instance>)
  453                         reloc = Elf32Reloca(bytes)
  454                     else:
  455                         raise "FIXME"

  456                     index = reloc.getSymTabIndex()
  457                     try:

<type 'exceptions.TypeError'>: exceptions must be old-style classes or derived 
from BaseException, not str
}}}



What is the expected output? What do you see instead?

Ideally, the file would have loaded. Lacking that, a useful exception. An 
exception related to exception syntax is less than useful, as it distracts from 
the root cause of the problem.


What version of the product are you using? On what operating system?

Latest checkout from mercurial (0e0ecce43c74). OS details listed above.


Please provide any additional information below.

The attached patch replaces the "FIXME" with an Exception instance that gives 
more details. Note that loading a 64 bit ELF still causes an exception, but 
I'll open a new ticket for that.

What steps will reproduce the problem?
1. ./pyew /path\ with/space/file


What is the expected output? What do you see instead?
IOError: [Errno 2] No such file or directory: <file name>

What version of the product are you using? On what operating system?
Current source on Fedora 17

Please provide any additional information below.

Fix applied to m3gat0nn4ge-pyew clone

for those who need it

ex :
on linux 64bit

/path/to/distorm3/make/linux % make 
sudo mv libdistorm3.so /usr/lib64

patch :

diff --git a/pydistorm.py b/pydistorm.py
--- a/pydistorm.py
+++ b/pydistorm.py
@@ -32,11 +32,11 @@

 osVer = platform.system()
 if osVer == "Windows":
-    LIB_FILENAME = "distorm64.dll"
+    LIB_FILENAME = "distorm3.dll"
 else:
-    LIB_FILENAME = 'libdistorm64.so'
+    LIB_FILENAME = 'libdistorm3.so'

-distorm = cdll.LoadLibrary(LIB_FILENAME)
+distorm3 = cdll.LoadLibrary(LIB_FILENAME)
 Decode16Bits = 0
 Decode32Bits = 1
 Decode64Bits = 2
@@ -44,11 +44,11 @@

 if osVer == "Windows":
     if SUPPORT_64BIT_OFFSET:
-        decode_func = distorm.distorm_decode64
+        decode_func = distorm3.distorm_decode64
     else:
-        decode_func = distorm.distorm_decode32
+        decode_func = distorm3.distorm_decode32
 else:
-    decode_func = distorm.internal_decode
+    decode_func = distorm3.distorm_decode64

 DECRES_NONE = 0
 DECRES_SUCCESS = 1

What steps will reproduce the problem?
1. Load /bin/ls from a 64 bit ELF platform.

What is the expected output? What do you see instead?

Ideally, the file would load with no error. After applying the patch in issue 
#7, we now get an exception ending in:


  File "/usr/local/python_environments/map/lib/python2.6/site-packages/pyew/Elf/__init__.py", line 456, in parseRelocs
    "bits == %r (not 32)" % self.bits)
Exception: FIXME: Section type is SHT_RELA and bits == 64 (not 32)



What version of the product are you using? On what operating system?

Latest from revision control. On Ubuntu 10.04 (on an x86_64 platform). Oddly, 
when I try to load the same 64 bit ELF file on from an x86 system, I get a 
different error:

  File "/home/wam/projs/map/pyew/Elf/__init__.py", line 118, in __init__
    strsec = self.sections[self.e_shstrndx]
IndexError: list index out of range



Please provide any additional information below.

I'm attaching a copy of Ubuntu 10.04's /bin/ls from an x86_64 system to 
possibly aid in debugging.

What steps will reproduce the problem?
1. load any reported malware
2. type in "vt"


What is the expected output? What do you see instead?
I expected the AV software + malware-"nicknames"
I see only the AV software who detects the malware

What version of the product are you using? On what operating system?
pyew version: (2.3.0.0)
OS: Linux trudy 3.15.8-1-ARCH #1 SMP PREEMPT Fri Aug 1 08:53:42 CEST 2014 i686 
GNU/Linux


I attached a jpg 

What steps will reproduce the problem?
1. Load (as part of bokken) pyew with a binary
2. Click "Visualize binary"
3. Nothing appears in GUI, TypeError traceback appears in console

What is the expected output? What do you see instead?
 Visualization of a binary. The traceback is:


Traceback (most recent call last):
  File "/usr/share/pyshared/bokken/ui/pyew_toolbar.py", line 462, in execute
    self.uicore.execute_plugin(plugin)
  File "/usr/share/pyshared/bokken/ui/core.py", line 522, in execute_plugin
    self.core.plugins[plg[0]](self.core)
  File "/usr/share/pyshared/pyew/plugins/graphs.py", line 110, in showBinaryImage
    img = Image.new("RGB", (size, size), "red")
  File "/usr/lib/python2.7/dist-packages/PIL/Image.py", line 1763, in new
    return Image()._new(core.fill(mode, size, color))
TypeError: integer argument expected, got float




What version of the product are you using? On what operating system?
Kali, version as described in pyew.py: 

VERSION=0x01020000
HUMAN_VERSION="1.2.0.0"


Please provide any additional information below.

The issue seems to be the result of a typo. graphs.py contains this code (lines 
108-110):

    buf = pyew.getBuffer()
    size = len(buf)**(1./3)+1
    img = Image.new("RGB", (size, size), "red")

It seems that someone was trying to either trying to type "len(buf)**(1//3)+1" 
or "len(buf)**(1/3)+1", but their finger slipped. The type of "1." is float, a 
float + 1 is also a float, and finally, an int to the power of a float is also 
a float.


Given that BOTH 1/3 and 1//3 evaluate to 0 in python 2.x, I'm more confused, as 
len(buf)**(0+1) == len(buf), why the exponentiation?

$ sqlite3 pyew-files.sqlite .schema
CREATE TABLE antidebugs (
                        id integer not null primary key,
                        sample_id, addr, mnemonic
                        );
CREATE TABLE function_stats (
                        id integer not null primary key,
                        sample_id, addr, nodes, edges, cc);
CREATE TABLE samples (id integer not null primary key,
                                       md5, sha1, sha256, filename, type);

What steps will reproduce the problem?
1. installing using hg
2. ./pyew <pdf file name>


after loading the pdf the pdf filet plugins are not present

im using it on Debian 6 64bit (same results with the 32bit versions as well) 
python 2.7 stock.


during loading i get the following warning:
Error loading file: 'pdf'

when trying to invoke pdfilter i get the following error:

Error: name 'pdfilter' is not defined

Hello,

I have tried to perform a malware analysis using pyew as given in the Wiki. 
However, I am getting the following error.

--------------------------------------------------------------------------------
-----$ pyew b6bd1640dcbd7b81970f8e4606b215e1
Traceback (most recent call last):
  File "/usr/share/pyshared/pyew/pyew.py", line 525, in <module>
    main(sys.argv[1])
  File "/usr/share/pyshared/pyew/pyew.py", line 232, in main
    pyew.loadFile(filename, "rb")
  File "/usr/share/pyshared/pyew/pyew_core.py", line 291, in loadFile
    self.f = file(filename, mode)
IOError: [Errno 2] No such file or directory: 'b6bd1640dcbd7b81970f8e4606b215e1'
--------------------------------------------------------------------------------
-----


Further, I have copied the content in a text file  and followed the steps given 
in the Wiki. I am getting the following error:

--------------------------------------------------------------------------------
-----
$ cat /home/user/example.txt
b6bd1640dcbd7b81970f8e4606b215e1


$ python pyew.py /home/user/example.txt 
0000   62 36 62 64 31 36 34 30 64 63 62 64 37 62 38 31    b6bd1640dcbd7b81
0010   39 37 30 66 38 65 34 36 30 36 62 32 31 35 65 31    970f8e4606b215e1
0020   0A                                                 .

[0x00000000]> packer
[0x00000000]> 
[0x00000000]> print len(pyew.functions)
0
[0x00000000]> s ep
[0x00000000]> c
Error: global name 'Decode32Bits' is not defined
[0x00000000]>
--------------------------------------------------------------------------------
-----

Please le me know if this is the correct way to do ? 

What steps will reproduce the problem?
1.open a malware using pyew
2.try to use vt on it

We should have the result here. Instead we have a 403 forbidden message.


I'm using the latest mercurial version at this date under GNU linux Sabayon.


I managed to come over the problem using this url string in 
plugins/virustotal.py :
baseUrl = "https://www.virustotal.com/search?query=%s"
instead of :
baseUrl = "http://www.virustotal.com/en/file/%s/analysis/"

Regards.

from pyew import CPyew

def main(f):
    pyew = CPyew(plugins=True, batch=True)
    pyew.loadFile(f)
    ca = pyew._anal
    for _, block in ca.basic_blocks.iteritems():
        for instr in block.instructions:
            print "%s %s" % (instr.mnemonic, instr.operands)
        print '--------------------------------------------'


if __name__ == "__main__":
    if len(sys.argv) == 1:
        print "Usage:", sys.argv[0], "<program file>"
    else:
        main(sys.argv[1])

What steps will reproduce the problem?
1. Create a file named virustotal.py in c:\Python27\lib\site-packages\
2. Run pyew from cmd : python pyew FILENAME
3. The virustotal plugin isn't loaded (so you don't have the options to use the 
"vt" plugin)



What version of the product are you using? On what operating system?
Latest pyew version from svn.
Windows 8.1 x64 bit, Pyhton 2.7 32bit

Please provide any additional information below.
It seems like the procedure that loads the plugins needs to be change so that 
it use only modules from the PLUGINS_PATH directory.

[0x00004abc:0x004056bc]> b
Error: name 'b' is not defined
[0x00004b08:0x00405708]> b
Error: name 'b' is not defined
[0x00032d88:0x00433988]> b
Error: name 'b' is not defined
[0x00000000]> b

Well it works OK :)

Would you consider changing the licensing being used for pyew to something less 
restrictive (like LGPL or MPL)? I want to use your library in my MPL'ed project 
and I don't want it to be as restrictive as the GPL. Thanks for your 
consideration.

The url.py plugin should iterate over the found Unicode urls but instead it 
iterate over the ascii urls again.

You can see the bug at line 62 of the url.py plugin file:
 for url in ret:

instead it should looks like this:
 for url in uniret:

I am starter with this program so this might be newbie mistake.

"""
[0x00000000]> vt
File 002.exe with MD5 ea58eefb31cfc7e866a771bf13294347
------------------------------------------------------

Error: local variable 'match' referenced before assignment
"""

Here is the report if I do send it using browser: 
http://www.virustotal.com/file-scan/report.html?id=7897ce606aac0e8186a68909a51e8
4dd298a5e590e80235000128cf004ff0c2b-1323035150

Am I doing something wrong or is this programming error?

What steps will reproduce the problem?
1. pyew notepad.exe
2. s 0x0000a7b4
3. d

What is the expected output? What do you see instead?

I expect to see all of the digits for the call and jump target addresses.  For 
example:

0x0000a7cd (05) e8 5660ffff          CALL 0x00000828    ; 1 sub_00000828
0x0000a841 (02) 74 53                JZ 0x0000a896  ; 2 

Instead, the last digit is missing:

0x0000a7cd (05) e85660ffff             CALL 0x00000082  ; 1 
0x0000a841 (02) 7453                   JZ 0x00000a89    ; 2 

What version of the product are you using? On what operating system?

The problem occurs with the current version of the code in the Mercurial 
repository, but not with version 2.0.  I am using Ubuntu 14.04 64-bit.

Please provide any additional information below.

The problem appears to be caused by lines 1116 and 1119 in pyew_core.py.  The 
current code is:

                        hex_pos = ops.find("[0x")
                        if ops > -1:
                          ops = ops[hex_pos+3:]
                        hex_pos = ops.find("]")
                        if ops > -1:
                          ops = ops[:hex_pos]
                        ops = int(ops, 16)

However, it works when changed to:

                        hex_pos = ops.find("[0x")
                        if hex_pos > -1:
                          ops = ops[hex_pos+3:]
                        hex_pos = ops.find("]")
                        if hex_pos > -1:
                          ops = ops[:hex_pos]
                        ops = int(ops, 16)

Hello,

When scanning a directory with gcluster :

---

hash:filename:primes_hash:nodes_total:nodes_max:nodes_avg:nodes_min:edges_total:
edges_max:edges_avg:edges_min:ccs_total:ccs_max:ccs_avg:ccs_min:functions:adjace
ncy_list
Traceback (most recent call last):
  File "./gcluster.py", line 353, in <module>
    compareDirectory(sys.argv[1])
  File "./gcluster.py", line 334, in compareDirectory
    print "%s:%s:%s:%s%d:%s" % (hash, pyew.f.name, str(phash.as_integer_ratio()[0]), data, len(pyew.functions), str(alist.adjacency_lists(pyew)))
TypeError: 'dict' object is not callable

---

Hi,

I added the pyew.case option so we can select if we prefer high or low case on 
disassembly with command:

pyew.case = 'high/low'

By default is set to 'high'. Not the best approach for performance but I don't 
think it will be a problem.

hi,

It is possible the streams of pdf contains '\x20', '\x0d' and '\x0a' after
the "stream", so simply intercept the stream buf with "buf[pos+8:pos2-1]"
may cause fail to decode it via filters. The following is my awkward fix,
works for me.

         # -8 means -len("stream")
-        tmp = buf[pos+8:pos2-1]
+        tmp = buf[pos+6:pos2-1]
+
+        # FIXME found some samples start with a SPACE, does it possible
with multi spaces?
+        if tmp[0] == ' ': tmp = tmp[1:]
+        # twice check, it can be '\x0D\x0A' or '\x0D' or '\x0A' follow the
'stream'
+        if tmp[0] == '\r' or tmp[0] == '\n': tmp = tmp[1:]
+        if tmp[0] == '\r' or tmp[0] == '\n': tmp = tmp[1:]
+

HTH,

Binjo

Hi,

I coded the scripting support for pyew. This offers the option to create a file 
with a list of pyew commands and to run them on pyew start or at any moment 
once working.

On the file attached you can find the patch, two sample scripts and a txt 
document with some examples on how to use scripts. The text file could be 
removed if added to the project wiki :)

Regards

There is an error on trying to run pdf_sample.py:

Traceback (most recent call last):
  File "pdf_example.py", line 7, in <module>
    from easygui import choicebox, fileopenbox, msgbox
ImportError: No module named easygui

It's easy to solve:

@@ -4,7 +4,7 @@
 import sys

 from pyew_core import CPyew
-from easygui import choicebox, fileopenbox, msgbox
+from plugins.easygui import choicebox, fileopenbox, msgbox

 def main(filename=None):

Regards

Hello every one
i am try to get the function call of pe file
and i want to export this result into text file for my processing later
but i can add the code :
file = open("text.txt", 'w') into pyew_core.py
can i help me?
thank you very much for readling mt isses!
good bye

What steps will reproduce the problem?
1. Analysis/Deepanalysis calc.exe (win xp sp0)
2. compare basicblocks found with pyew with IDA 

What is the expected output? What do you see instead?
 I found 26 basic blocks. should be 32 (4 which might be hard to catch (SEH blocks))

What version of the product are you using? On what operating system?
PYEW "1.2.0.0", ubuntu 12.04 64bit 

Please provide any additional information below.
it misses a short jump @ addr 0x010124c8 to basicblock @ 0x010124d8
and a short jump @ addr 0x010125cd to basicblock @ 0x010125dd

both jumps have same offset 0x0e: opcode EB 0E

What steps will reproduce the problem?
1. Try to load an Elf object using a binary string. E.g.:

>>> from pyew.Elf import Elf
>>> data = file("/bin/ls").read()
>>> Elf(data)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyew/Elf/__init__.py", line 77, in __init__
    if os.path.exists(initstr):
  File "MY_VIRTUALENV_PATH/lib/python2.6/genericpath.py", line 18, in exists
    st = os.stat(path)
<type 'exceptions.TypeError'>: stat() argument 1 must be encoded string without 
NULL bytes, not str


What is the expected output? What do you see instead?

The problem comes from the fact that the initializer is using os.path.exists() 
to determine if the passed in string is a filename that should be loaded or is 
otherwise considered a string containing already loaded bytes. The problem 
though is that os.path.exists() chokes on characters that aren't 
expected/allowed to be part of a filename (such a 0x00).


What version of the product are you using? On what operating system?

Latest pyew checkout, on Ubuntu Linux (9.10 and 10.04).

Please provide any additional information below.

Attached is a small patch that fixes this issue.

Hello,
When I try to analyze an ELF binary I get the following traceback (For your 
information: pysqlite and sqlite3 are installed.) :

user@trudy /opt/pyew % python2 pyew.py /usr/bin/ls
ELF Information

Entry Point at 0x1ce0
CodAnalyzing address 0x00011e40 - 0 in queue / 3 total                          
                                Analyzing address 0x0000cc50 - 1 in queue / 2 
total
Traceback (most recent call last):n queue / 1 total                             
  File "pyew.py", line 532, in <module>
    main(sys.argv[1])
  File "pyew.py", line 219, in main
    saveAndCompareInDatabase(pyew)
  File "pyew.py", line 164, in saveAndCompareInDatabase
    db = sqlite3.connect(DATABASE_PATH)

Are there any plans to support elf64 bits?

What steps will reproduce the problem?
1.  Disassemble attachmen in IDA and goto .text:100019A8   
2.  Notice the destination of this instruction isnt created as a basic block?  
jz      short loc_100019BD
3.

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?
Latest pyew from source via hg clone.  Operating system on windows 7.

Please provide any additional information below.
Here is the toy script/harness I used to analyze basic blocks for the given 
program.

from pyew import *

def main(f):
    pyew = CPyew(plugins=True, batch=True)
    pyew.loadFile(f)
    print "Total basic blocks %d" % len(pyew.basic_blocks)
    for bb in pyew.basic_blocks:
      print "[*]", hex(pyew.getVirtualAddressFromOffset(bb))
if __name__ == "__main__":
    if len(sys.argv) == 1:
        print "Usage:", sys.argv[0], "<program file>"
    else:
        main(sys.argv[1])

When I output stdout to a text file I noticed that 100019BD  is not included in 
the basic blocks set.  It 'should' be as it is a code block has 1 direction in 
and 1 direction out.  I tried to look and debug the code in X86Analyzer within  
   def createFunction(self, addr):
 but couldn't figure out why the analysis code wasn't following the destination of the jump?  

Going to continue poking and update here accordingly.