areizen / unpyc Goto Github PK

What steps will reproduce the problem?
decompiling a program that uses:

import a.b.c.d as e

will fail, because the code expects no more then one LOAD_ATTR after a 
IMPORT_FROM statement.

A work around is to add the following, up to the number of LOAD_ATTR lines you 
are seeing - I don't currently see a way to tell unpyc to accept any number of 
them.  Starting at line 189 in Parser.py, add:
import_as ::= IMPORT_NAME LOAD_ATTR LOAD_ATTR designator
import_as ::= IMPORT_NAME LOAD_ATTR LOAD_ATTR LOAD_ATTR designator

etc, etc, etc

Yes, I know this doesn't fit the 0,1,n pattern, but it is a quick fix.

What steps will reproduce the problem?
1. checkout read-only
2. find 'svn-commit.tmp' file in root
3. also ROOT\.svn\text-base\svn-commit.tmp.svn-base 

What is the expected output? What do you see instead?
no file 'svn-commit.tmp', it's a temporary file for uncommitted changes

What version of the product are you using? On what operating system?
Doesnt matter.

Please provide any additional information below.
None.

When I tried to install unpyc I get this error:


Microsoft Windows XP [Versão 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\>cd C:\Python26\decompyle

C:\Python26\decompyle>setup.py install
running install
running build
running build_py
running build_ext
building 'unpyc/marshal_20' extension
error: Unable to find vcvarsall.bat


I'm using Windows XP SP3 32bits
Python 2.6
And I have Microsoft Visual C++ 2005 Express Edition



I don't know what I'm doing wrong...

File "C:\Python26\lib\site-packages\unpyc\__init__.py", line 183, in main
    unpyc_file(infile, outstream, showasm, showast)
  File "C:\Python26\lib\site-packages\unpyc\__init__.py", line 120, in 
unpyc_fil
e
    unpyc(version, co, outstream, showasm, showast)
  File "C:\Python26\lib\site-packages\unpyc\__init__.py", line 113, in 
unpyc
    walker.gen_source(ast, customize)
  File "C:\Python26\lib\site-packages\unpyc\Walker.py", line 832, in 
gen_source
    self.print_(self.traverse(ast, isLambda=isLambda))
  File "C:\Python26\lib\site-packages\unpyc\Walker.py", line 362, in 
traverse
    self.preorder(node)
  File "C:\Python26\lib\site-packages\unpyc\spark.py", line 682, in 
preorder
    self.preorder(kid)
  File "C:\Python26\lib\site-packages\unpyc\spark.py", line 682, in 
preorder
    self.preorder(kid)
  File "C:\Python26\lib\site-packages\unpyc\spark.py", line 677, in 
preorder
    self.default(node)
  File "C:\Python26\lib\site-packages\unpyc\Walker.py", line 630, in 
default
    self.engine(table[key], node)
  File "C:\Python26\lib\site-packages\unpyc\Walker.py", line 598, in engine
    self.preorder(node[entry[arg]])
  File "C:\Python26\lib\site-packages\unpyc\spark.py", line 675, in 
preorder
    func(node)
  File "C:\Python26\lib\site-packages\unpyc\Walker.py", line 489, in 
n_mkfunc
    self.make_function(node, isLambda=0)
  File "C:\Python26\lib\site-packages\unpyc\Walker.py", line 733, in 
make_functi
on
    ast = self.build_ast(code._tokens, code._customize)
  File "C:\Python26\lib\site-packages\unpyc\Walker.py", line 843, in 
build_ast
    raise ParserError(e, tokens)
unpyc.Walker.ParserError: --- This code section failed: ---
0       LOAD_CONST        ''
3       STORE_FAST        'i1'
6       BUILD_LIST_0      ''
9       STORE_FAST        'result'
12      SETUP_LOOP        '115'
15      LOAD_FAST         'where'
18      LOAD_ATTR         'find'
21      LOAD_FAST         'start'
24      LOAD_FAST         'i1'
27      CALL_FUNCTION_2   ''
30      STORE_FAST        'i0'
33      LOAD_FAST         'where'
36      LOAD_ATTR         'find'
39      LOAD_FAST         'end'
42      LOAD_FAST         'i0'
45      CALL_FUNCTION_2   ''
48      STORE_FAST        'i1'
51      LOAD_FAST         'i0'
54      LOAD_CONST        -1
57      COMPARE_OP        '=='
60      JUMP_IF_TRUE      '76'
63      POP_TOP           ''
64      LOAD_FAST         'i1'
67      LOAD_CONST        -1
70      COMPARE_OP        '=='
73      JUMP_IF_FALSE     '81'
76_0    COME_FROM         ''
76      POP_TOP           ''
77      BREAK_LOOP        ''
78      JUMP_FORWARD      '82'
81_0    COME_FROM         ''
81      POP_TOP           ''
82_0    COME_FROM         ''
82      LOAD_FAST         'result'
85      LOAD_ATTR         'append'
88      LOAD_FAST         'where'
91      LOAD_FAST         'i0'
94      LOAD_GLOBAL       'len'
97      LOAD_FAST         'start'
100     CALL_FUNCTION_1   ''
103     BINARY_ADD        ''
104     LOAD_FAST         'i1'
107     SLICE+3           ''
108     CALL_FUNCTION_1   ''
111     POP_TOP           ''
112     JUMP_ABSOLUTE     '15'
115_0   COME_FROM         ''
115     LOAD_FAST         'result'
118     RETURN_VALUE      ''

Syntax error at or near `STORE_FAST' token at offset 30

how can I fix it?

What steps will reproduce the problem?
1.
2.
3.

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?


Please provide any additional information below.

Everywhere I look I see reference to Dan Pascu's patch that added
recognition of Python 2.3 optimization structures.  Debian has long given
up on maintaining old repositories of decompyle.

Does anyone have the exact text of Dan Pascu's patch?

Also, what's the difference between this project and "depyc" (also on
Google code) except that 2 people forked decompyle and nobody can make it
work with Python 2.5?

My mission: To point out the issue and help fix it.

Note: I'm not invalidating anyone, this not an attempt to troll.
unpyc is free, and I'm not owed anything.
Like a lot of people comming here, we are looking for a free robust method to 
decompile Python files.

Fact:
This decompiler is very broke and is not fit for beyond the most primitive 
decompiling as it stands on July 31, 2011.


What steps will reproduce the problem?:
Attempted to decompile several Python 2.6a1 files.

What is the expected output? What do you see instead?:
Was expected that they decompile.
At best maybe 1 to 3 out of ten actually decompiled with out error.
Most more then 90% failed.

What version of the product are you using?:
The latest and then with the 2.7 patch applied

On what operating system?:
Windows XP 32bit.

Please provide any additional information below:

First Some background.  I found some Korean sources for a 2.3 update, and then 
I attempted to update it to 2.4.
Admittedly for me, my biggest problem is my lack of the Python language.

It seems to me the root problem is at least with this decompile paradigm is 
that you end up with blocks (groups of opcodes between if/while/for what ever 
loops and branch constructs).
Then to put the actual byte code back into Python code its looked at as 
groups/patterns.  In other words a a group of byte codes will form a construct.

To do this properly one must know the patterns. At least most of them to 
approach a near 100% accuracy.
On top of this, this can be different per Python version.
From my work and understanding the bulk of the work resides in "Parser.py".
It's apparent that not all patterns are covered.
Unexpected patterns cause the parser to break.

I believe John Aycock's original stuff covered most of this and public attempts 
at updating have not been as through enough.

Furthermore if you look at the pay decompile site "crazy-compilers" quote:
"* Our unit-tests include more than 3900 test-patterns. Each of it is 
successfully decompiled for all supported versions, both normal (.pyc) and 
optimized (.pyo) bytecode. This is a total of about 138,000 test-cases."
"* decompyles and successfully verifies 100% of the Python 1.5, 2.0, 2.1, 2.2, 
2.3, 2.4, 2.5 and 2.6 library"

Solution:
I don't know if there needs to be close to 4k test cases but I think this is 
the key thing to make this decompiler fit.
I believe it will require a similar methodology.

Many simple construct and complex real world samples need to be applied and 
patterns needed to be tweaked and added.
Prehaps broken down on a per Python version basis.
This should obviously be automated as much as possible to make it easier.

I added support for Python 2.7

ff.py code:

a, b = 0, 1
a, b = b, a+b

after compile by python26

can not be decompiled, unpyc wrote:

  File "c:\python\26\lib\site-packages\unpyc\Walker.py", line 843, in build_ast
    raise ParserError(e, tokens)
unpyc.Walker.ParserError: --- This code section failed: ---
0       LOAD_CONST        (0, 1)
3       UNPACK_SEQUENCE_2 ''
6       STORE_NAME        'a'
9       STORE_NAME        'b'
12      LOAD_NAME         'b'
15      LOAD_NAME         'a'
18      LOAD_NAME         'b'
21      BINARY_ADD        ''
22      ROT_TWO           ''
23      STORE_NAME        'a'
26      STORE_NAME        'b'
29      LOAD_CONST        ''
32      RETURN_VALUE      ''

Syntax error at or near `ROT_TWO' token at offset 22

ff.py & ff.pyc attached.

I use :
BACKTRACK 2 LINUX.
PYTHON 2.5\2.4.
UNPYC R24.  


scripts-2.5 # python2.5 unpyc -o /root -c  downloader.pyc

unpyc.Walker.ParserError: --- This code section failed: ---
0       LOAD_FAST         'self'
3       LOAD_ATTR         'decoder'
6       LOAD_ATTR         'start'
9       CALL_FUNCTION_0   ''
12      POP_TOP           ''
13      SETUP_LOOP        '1988'
16      SETUP_LOOP        '545'
19      LOAD_FAST         'self'
22      LOAD_ATTR         'servers'
25      GET_ITER          ''
26      FOR_ITER          '544'
29      STORE_FAST        'server'
32      SETUP_LOOP        '105'
35      LOAD_FAST         'server'
38      LOAD_ATTR         'busy_threads'
41      SLICE+0           ''
42      GET_ITER          ''
....
1980    POP_TOP           ''
1981_0  COME_FROM         ''
1981    JUMP_ABSOLUTE     '1193'
1984_0  COME_FROM         ''
1984    POP_BLOCK         ''
1985_0  COME_FROM         ''
1985    JUMP_ABSOLUTE     '16'
1988_0  COME_FROM         ''
1988    LOAD_CONST        ''
1991    RETURN_VALUE      ''

Syntax error at or near `SETUP_LOOP' token at offset 16.


The file I'm trying to decompile is from SABNZBD+ 0.4.6.  I modified
downloader.py late 2008 and I accidentally lost it after I compiled it.  I
include a unmodified downloader.py and my compiled\modified downloader.pyo.

Please fix this as soon as possible!

I ATTECHED FILES!!!!!!!!!!1
DOWNLOAD THEM TO HELP ME!!

I include a unmodified downloader.py and my compiled\modified downloader.pyo.

What steps will reproduce the problem?

Problem on Python 2.5
Has error on decompile at next files:
1. test_listComprehensions.pyc
(http://code.google.com/p/unpyc/source/browse/trunk/test/bytecode_2.5/test_listC
omprehensions.pyc)
2. test_tuples.pyc
(http://code.google.com/p/unpyc/source/browse/trunk/test/bytecode_2.5/test_tuple
s.pyc)
3. test_yield.pyc
(http://code.google.com/p/unpyc/source/browse/trunk/test/bytecode_2.5/test_yield
.pyc)

What is the expected output? What do you see instead?

Promlem with get source code for module "test_yield.pyc":
unpyc.Walker.ParserError: --- This code section failed: ---
0       LOAD_FAST         't'
3       JUMP_IF_FALSE     '80'
6       POP_TOP           ''
7       SETUP_LOOP        '38'
10      LOAD_GLOBAL       'inorder'
13      LOAD_FAST         't'
16      LOAD_ATTR         'left'
19      CALL_FUNCTION_1   ''
22      GET_ITER          ''
23      FOR_ITER          '37'
26      STORE_FAST        'x'
29      LOAD_FAST         'x'
32      YIELD_VALUE       ''
33      POP_TOP           ''
34      JUMP_ABSOLUTE     '23'
37_0    COME_FROM         ''
37      POP_BLOCK         ''
38_0    COME_FROM         ''
38      LOAD_FAST         't'
41      LOAD_ATTR         'label'
44      YIELD_VALUE       ''
45      POP_TOP           ''
46      SETUP_LOOP        '81'
49      LOAD_GLOBAL       'inorder'
52      LOAD_FAST         't'
55      LOAD_ATTR         'right'
58      CALL_FUNCTION_1   ''
61      GET_ITER          ''
62      FOR_ITER          '76'
65      STORE_FAST        'x'
68      LOAD_FAST         'x'
71      YIELD_VALUE       ''
72      POP_TOP           ''
73      JUMP_ABSOLUTE     '62'
76_0    COME_FROM         ''
76      POP_BLOCK         ''
77_0    COME_FROM         ''
77      JUMP_FORWARD      '81'
80_0    COME_FROM         ''
80      POP_TOP           ''
81_0    COME_FROM         ''
81      LOAD_CONST        ''
84      RETURN_VALUE      ''
Syntax error at or near `POP_TOP' token at offset 33

Promlem with get source code for module "test_listComprehensions.pyc":

unpyc.Walker.ParserError: --- This code section failed: ---
0       BUILD_LIST_0      ''
3       DUP_TOP           ''
4       STORE_FAST        '_[1]'
7       LOAD_FAST         'seq'
10      GET_ITER          ''
11      FOR_ITER          '40'
14      STORE_FAST        'subseq'
17      LOAD_FAST         'subseq'
20      GET_ITER          ''
21      FOR_ITER          '37'
24      STORE_FAST        'x'
27      LOAD_FAST         '_[1]'
30      LOAD_FAST         'x'
33      LIST_APPEND       ''
34      JUMP_ABSOLUTE     '21'
37_0    COME_FROM         ''
37      JUMP_ABSOLUTE     '11'
40_0    COME_FROM         ''
40      DELETE_FAST       '_[1]'
43      RETURN_VALUE      ''

Syntax error at or near `GET_ITER' token at offset 10

Promlem with get source code for module "test_tuples.py":
33      LOAD_CONST        (3, 4, 5)
36      ROT_TWO           ''
37      STORE_NAME        'a'
40      STORE_NAME        'b'
43      BUILD_MAP         ''
46      STORE_NAME        'x'
49      SETUP_EXCEPT      '64'
52      LOAD_NAME         'x'
55      LOAD_CONST        (1, 2, 3)
58      BINARY_SUBSCR     ''
59      POP_TOP           ''
60      POP_BLOCK         ''
61      JUMP_FORWARD      '71'
64_0    COME_FROM         ''
64      POP_TOP           ''
65      POP_TOP           ''
66      POP_TOP           ''
67      JUMP_FORWARD      '71'
70      END_FINALLY       ''
71_0    COME_FROM         ''
71_1    COME_FROM         ''
71      LOAD_CONST        42
74      LOAD_NAME         'x'
77      LOAD_CONST        (1, 2, 3)
80      STORE_SUBSCR      ''
81      LOAD_NAME         'x'
84      LOAD_CONST        (1, 2, 3)
87      BINARY_SUBSCR     ''
88      PRINT_ITEM        ''
89      PRINT_NEWLINE     ''
90      LOAD_NAME         'x'
93      LOAD_CONST        (1, 2, 3)
96      BINARY_SUBSCR     ''
97      PRINT_ITEM        ''
98      PRINT_NEWLINE     ''
99      LOAD_NAME         'x'
102     LOAD_CONST        (1, 2, 3)
105     BINARY_SUBSCR     ''
106     LOAD_NAME         'x'
109     LOAD_CONST        (1, 2, 3)
112     BINARY_SUBSCR     ''
113     COMPARE_OP        '=='
116     JUMP_IF_TRUE      '126'
119     POP_TOP           ''
120     LOAD_GLOBAL       'AssertionError'
123     RAISE_VARARGS     ''
126_0   COME_FROM         ''
126     POP_TOP           ''
127     LOAD_NAME         'x'
130     LOAD_CONST        (1, 2, 3)
133     DELETE_SUBSCR     ''
134     LOAD_CONST        ''
137     RETURN_VALUE      ''

Syntax error at or near `ROT_TWO' token at offset 36

What version of the product are you using? On what operating system?
using last version of "unpyc", WIN2003
Python 2.5.4 (r254:67916) [MSC v.1310 32 bit (Intel)] on win32

Please provide any additional information below.

other files decompile successfully (directory "test")