This is most notable when using the "what function is calling this function" queries in cscope.

pycscope cannot parse
print(f"Found {num} devices.")

I hacked init.py to have f as a keyword.
kwlist.extend(("True", "False", "None", "f"))

Works now.. but I am sure there is a better fix.

Reproduce:

E:\Lab\casuald\lib>pycscope -R
E:\Lab\casuald\lib>cscope -dl -f cscope.out
cscope: cannot read trailer offset from file cscope.out

Expect:
cscope should load "cscope.out" correctly on windows, just like on linux

Result:
cscope report an error: "cscope: cannot read trailer offset from file cscope.out"

Version:
cscope.exe: 15.8a (from: cscope-win32)
pycscope: 1.2.1 (installed by pip)
Windows: Windows 7
Python: 2.7.12 32bits

Hi Peter,
I haven't checked in for a while and looks like you've done a lot of new work to get Python3 and some new things working. Nice!

I checked out the latest code and ran on a fairly large P3 code base and things seemed to work as expected except that at one place the decoding of assignment was wrong.

[Edit: it is a little more complicated than I first realized. Tracing back Python grammar the problem appears to be that you can stack trailers which confuses isTrailerFuncCall.]
The problem is that you can have a gettor call on the lhs of an assignment, which could create a comma that is not part of a tuple assignment, as in:

x.get(0, "sth", c=True).text = "ok"

This used to work but now it produces an error "Comma encountered, but an assignment has not already taken place". The error message indicates that it is related to the following change:

...  ...    
@@ -811,7 +835,7 @@ def processTerminal(ctx, cst):
811  835    
         # comma which means the symbol following is also being assigned
812  836    
         # to, so we need to bump the assigned count back up to properly
813  837    
         # handle that below.
814     
-        if ctx.mark == Mark.ASSIGN:
     838    
+        if (ctx.in_trailer == None) and (ctx.mark == Mark.ASSIGN):
815  839    
             assert ctx.equal_cnt > 0, "Comma encountered, but assignment marker in place without an equal count"
816  840    
             assert (ctx.assigned_cnt + 1) == ctx.equal_cnt, "Comma encountered, but an assignment has not already taken place"
817  841    
             ctx.assigned_cnt += 1

The new error reporting with file names and line numbers provided really helped finding this in a cinch. Great feature! Question: Does pycscope now chug along even after encountering errors? I seem to remember that it just broke down on syntax errors. Now it still proceeds to produce cscope.out despite the errors encountered. Is the cscope.out expected to be legit in such a case?

With this code:

class MyClass(object):

    @property
    def get_bar(self):
        return 'foo'

    def my_method(self):
        from datetime import datetime

pyscope produces this traceback:

Traceback (most recent call last):
  File "/home/mk/src/pycscope/pycscope.py", line 914, in <module>
    main()
  File "/home/mk/src/pycscope/pycscope.py", line 154, in main
    indexbuff, fnamesbuff = doWork(basepath, gen, lock)
  File "/home/mk/src/pycscope/pycscope.py", line 202, in work
    indexbuff_len = parseFile(basepath, fname, indexbuff, indexbuff_len, fnamesbuff)
  File "/home/mk/src/pycscope/pycscope.py", line 288, in parseFile
    indexbuff_len = parseSource(filecontents, indexbuff, indexbuff_len)
  File "/home/mk/src/pycscope/pycscope.py", line 887, in parseSource
    walkCst(ctx, cst.totuple(True))
  File "/home/mk/src/pycscope/pycscope.py", line 857, in walkCst
    processCst(ctx, cst)
  File "/home/mk/src/pycscope/pycscope.py", line 844, in processCst
    processNonTerminal(ctx, cst)
  File "/home/mk/src/pycscope/pycscope.py", line 668, in processNonTerminal
    ctx.setMark(cst[-1], Mark.FUNC_CALL)
  File "/home/mk/src/pycscope/pycscope.py", line 543, in setMark
    assert idx not in self.marks
AssertionError

If you swap the order of the methods, the assertion goes away.
If you use "bar = property(get_bar)" instead of the decorator, the assertion goes away.
If you use "import x" instead of "from x import y", the assertion goes away.

Hi Peter,
I wrote a tool that reads cscope.out with only the -c parameter, and it has one problem with the cscope.out emitted by pycscope, but not by the cscope.out emitted by cscope 15.7+. If you look at the cscope source code, there are some spaces intentionally written into the header after -c. I suggest to include this in pycscope.

I have taken the liberty to add it myself to the function writeIndex of pycscope. Please see my changes below. The changes are quite simple, so I didn't make a pull request.

    hdr_len = len(basepath) + 39
    fout.write("cscope 15 %s -c               %010d" % (basepath, hdr_len + index_len))

Please see my project, CodeQuery. I'm currently trying to add pycscope support to it so that it will would understand Python. It uses both cscope and ctags databases.

Using the AST tree might simplify the code (removing all kinds of pattern matching we do today on the CST tree) and allow additional features not possible with the CST tree.

It might also speed up the processing of source files (or not).

This is a place holder for tracking the pros and cons of making that switch. It should be noted that such a switch should have a significant enough benefit to out weigh any destabilization of the code in the mean time.

See also Issue #12.

Exception in thread Worker-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in bootstrap_inner
self.run()
File "/home/steven/.local/lib/python2.7/site-packages/pycscope/__init.py", l
ine 191, in run
indexbuff, fnamesbuff = workT(self.basepath, self.gen, self.lock, self.debug
)
TypeError: workT() takes exactly 3 arguments (4 given)

When executing pycscope -i 'srclistfile' an error occurs.
It seems that the string.rstrip section of line 118 in init.py brings up the error "string has no attribute rstrip".

I personally fixed this by changing that section to read "str.rstrip" and now the -i function works fine.

I am using cscope 15-8.a with vim. When I do cs add cscope.out on the db created by pycscope it complains about "cannot read list size from cscope.out". This error message can be traced to main.c in cscope, defined in skiplist. The call to skiplist is to "skip the source and include directory lists" according to the comment in main.c

The same executable worked with cscope.out generated by cscope itself.

So we can jump to install libraries. That is useful.

From Free Speech:

diff --git a/pycscope.py b/pycscope.py
index 4e9d07f..39c5b2b 100755
--- a/pycscope.py
+++ b/pycscope.py
@@ -607,7 +607,7 @@ def isTrailerFuncCall(cst):
     """
     if len(cst) < 4:
         return False
-
+    printCst(cst)
     return (cst[-2][0] == symbol.trailer) \
             and (cst[-2][1][0] == token.DOT) \
             and (cst[-2][2][0] == token.NAME) \
@@ -769,6 +769,7 @@ def processNonTerminal(ctx, cst):
             ctx.setMark(cst[-2][2], Mark.FUNC_CALL)
             # Suspend COMMA processing in trailers
             ctx.in_trailer = ctx.spb_lvl[token.LPAR];
+        print("in_trailer: %s" % ctx.in_trailer)

 def processTerminal(ctx, cst):
     """ Process a given CST tuple representing a terminal symbol
@@ -914,7 +915,7 @@ def processTerminal(ctx, cst):

     return lineno

-def walkCst(ctx, cst):
+def walkCst(ctx, cst, processing=True):
     """ Scan the CST (tuple) for tokens, appending index lines to the buffer.
     """
     indent = 0
@@ -924,12 +925,13 @@ def walkCst(ctx, cst):
         while stack:
             cst, indent = stack.pop()

-            #print("%5d%s%s" % (lineno, " " * indent, nodeNames[cst[0]]))
+            print("%5d%s%s" % (lineno, " " * indent, nodeNames[cst[0]]))

-            if token.ISNONTERMINAL(cst[0]):
-                processNonTerminal(ctx, cst) 
-            else:
-                lineno = processTerminal(ctx, cst)
+            if processing:
+                if token.ISNONTERMINAL(cst[0]):
+                    processNonTerminal(ctx, cst) 
+                else:
+                    lineno = processTerminal(ctx, cst)

             indented = False
             for i in range(len(cst)-1, 0, -1):
@@ -944,6 +946,11 @@ def walkCst(ctx, cst):
         e.lineno = lineno
         raise e

+def printCst(cst):
+    print("CST subtree starts:")
+    walkCst(None, cst, False) # just print the tree
+    print("CST subtree ends")
+
 def parseSource(sourcecode, indexbuff, indexbuff_len, dump=False):
     """Parses python source code and puts the resulting index information into the buffer.
     """

When we have an assignment statement, like the following: x.y = 5, we find that x is marked as the symbol with the assignment, but we probably want y to be marked.

Python3 support is broken right now over symbol.listmaker
http://python.readthedocs.org/en/latest/reference/grammar.html

These are the kind of things that could bite over and over again due to the cst implementation (vs ast).

pycscope works fine with vim on non-windows system. but on windows, gvim can not recognize the "cscope.out" generated by pycscope.

Reproduce:

1. generate cscope.out with pycscope -R
2. open gvim.exe, input : "cs add cscope.out"

Result:

GVim prompts error

E609: Cscope error: cscope: cannot read trailer offset from file E:\Lab\casuald\.cscopy

expect:
GVim should load "cscope.out" correctly

Version:
GVim: 8.0.55 32bits
cscope.exe: 15.8a
pycscope: 1.2.1 (installed by pip)
Windows: Windows 7
Python: 2.7.12 32bits

I have searched a lot on the web, it says, path name shouldn't contain spaces, but there isn't any space in my path name.

BTW: My gvim can load the file generated by cscope.exe.

Hi,

I find that using the keyword-only arguments syntax causes pycscope to fail (I use version 1.1).

For example:

def foo(pos1, pos2, *, bar=None):
    pass

causes this error:
pycscope.py: /home/beka/thesis/ucca/try.py: Line 1: invalid syntax (try.py, line 1)

Would it be useful to just invoke cscope in interactive mode on the database that pyscope just build if the user did not specify -b?

The new broken pattern is now this:

x.get([], "123", c=[1,2]).text = "ok"

However this issue could affect both LHS and RHS. It just does not cause assertion errors on the RHS.

It looks like now isTrailerFuncCall is correctly setting in_trailer. However it then got turned off prematurely by the RSQB terminal (of the first argument). Further comma in the argument list then caused the error. Since the argumentlist can be arbitrarily complex this protocol of turning on/off in_trailer won't work without more context? Basically in_trailer is set to the spb_lvl of the entering bracket and reset when a leaving bracket is at the same level. This can't handle the intermingling of brackets.

Using these printing codes inside walkCst should make the problem easy to see:

            print(ctx.spb_lvl)
            print(ctx.in_trailer)
            print("%5d%s%s" % (lineno, " " * indent, nodeNames[cst[0]]))

The effect on RHS is probably currently non-existent. However is it possible that once in_trailer can be correctly set some RHS symbols can be recognized better? I do notice that sometimes cscope doesn't recognize what appear to me valid symbols but I haven't really documented these irregular encounters.

Given that people have tried cscope for python files and have found various short comings.

This issue will probably stay open for longer than last one :)

I am interested in running pycscope on some Python3 codes so I can jump around easier. I did a quick port of the code to Python3. The git diff will be attached at the end. I will make some comments here on what happened.

First I ran 2to3. This addressed the prints, string typing and exception syntax issues. Some other minor issues were also easily addressed:

1. Got rid of the import from compiler (not used and obsolete in python3);
2. Strings have now type str instead of bytes. A couple of type comparison to bytes were changed to str.
3. There is one place where I had to change string.replace(sourcecode, ..) to sourcecode.replace(...).
4. The fix for DOS line endings in the output files is best addressed in P3 with the newline argument for open.

However the nonterminal symbol decoding of the augmented assignment had a problem that took quite a bit of work to figure out. Here the parse tree has changed in Python. From grammar.html:

P3.2:

expr_stmt: testlist_star_expr (augassign (yield_expr|testlist) |
                     ('=' (yield_expr|testlist_star_expr))*)

P2.7:

expr_stmt: testlist (augassign (yield_expr|testlist) |
                     ('=' (yield_expr|testlist))*)

The end fix was just to change one symbol comparison from testlist (for P2) to testlist_star_expr (for P3) and it has worked so far. However working through this I also realized that the current implementation using raw parser output is a bit hairy and many supported grammar features are not actually implemented. For example the implementation for the above augassign only supports testlist on the RHS (hardcoded in pycscope) but the language actually allows for yield_expr as well.

Does Python AST provide enough information for pycscope to work with instead of dealing with the raw parse tree? That should simplify the logic and make the whole thing safer with complete coverage? AST provides lineno and column offset. Cross compilers seem to use AST instead of parser ST objects.

There is also the question of whether one code base should be maintained for both P2/P3. The symbol decoding difference above is definitely not compatible and need to be handled differently depending on whether P2 or P3 is at issue.

Anyway here is the patch for P3 that works for me:

diff --git a/pycscope.py b/pycscope.py
index 254d313..533ceb8 100755
--- a/pycscope.py
+++ b/pycscope.py
@@ -22,3 +22,3 @@ __usage__ = """Usage: pycscope.py [-R] [-t cnt] [-f reffile] [-i srclistfile] [f
 import getopt, sys, os, os.path, string, types
-import keyword, parser, symbol, token, compiler
+import keyword, parser, symbol, token
 from threading import Lock, Thread
@@ -94,3 +94,3 @@ def main():
     except getopt.GetoptError:
-        print __usage__
+        print(__usage__)
         sys.exit(2)
@@ -111,4 +111,4 @@ def main():
                     threadCount = val
-            except Exception, e:
-                print __usage__
+            except Exception as e:
+                print(__usage__)
                 sys.exit(2)
@@ -117,3 +117,3 @@ def main():
         if o == "-i":
-            args.extend(map(string.rstrip, open(a, 'r').readlines()))
+            args.extend(list(map(string.rstrip, open(a, 'r').readlines())))
 
@@ -202,4 +202,4 @@ def work(basepath, gen, lock):
             indexbuff_len = parseFile(basepath, fname, indexbuff, indexbuff_len, fnamesbuff)
-        except SyntaxError, se:
-            print "pycscope.py: %s: %s" % (se.filename, se)
+        except SyntaxError as se:
+            print("pycscope.py: %s: %s" % (se.filename, se))
             pass
@@ -225,4 +225,4 @@ def workT(basepath, gen, lock):
             indexbuff_len = parseFile(basepath, fname, indexbuff, indexbuff_len, fnamesbuff)
-        except SyntaxError, se:
-            print "pycscope.py: %s: %s" % (se.filename, se)
+        except SyntaxError as se:
+            print("pycscope.py: %s: %s" % (se.filename, se))
             pass
@@ -288,3 +288,3 @@ def parseFile(basepath, relpath, indexbuff, indexbuff_len, fnamesbuff):
             indexbuff_len = parseSource(filecontents, indexbuff, indexbuff_len)
-        except SyntaxError, se:
+        except SyntaxError as se:
             se.filename = fullpath
@@ -308,3 +308,3 @@ def replaceNodeType(treeList):
     for i in range(1, len(treeList)):
-        if type(treeList[i]) == types.ListType:
+        if type(treeList[i]) == list:
             replaceNodeType(treeList[i])
@@ -324,3 +324,3 @@ class Symbol(object):
         """
-        assert (mark == Mark.FUNC_END or name) and (type(name) == types.StringType), "Must have an actual symbol name as a string (unless marking function end)."
+        assert (mark == Mark.FUNC_END or name) and (type(name) == str), "Must have an actual symbol name as a string (unless marking function end)."
 
@@ -358,3 +358,3 @@ class Symbol(object):
         else:
-            print "Symbol(): does not have attribute <%s>" % name
+            print("Symbol(): does not have attribute <%s>" % name)
             raise AttributeError(name)
@@ -368,3 +368,3 @@ class Symbol(object):
 
-    def __nonzero__(self):
+    def __bool__(self):
         """ Defined so that the interpretter won't invoke
@@ -385,3 +385,3 @@ class NonSymbol(object):
         """
-        assert val and (type(val) == types.StringType), "Must have an actual string."
+        assert val and (type(val) == str), "Must have an actual string."
         self.__text = str(val)
@@ -408,3 +408,3 @@ class Line(object):
     def __init__(self, num):
-        assert ((type(num) == types.IntType) or (type(num) == types.LongType)) and num > 0, "Requires a positive, non-zero integer for a line number"
+        assert ((type(num) == int) or (type(num) == int)) and num > 0, "Requires a positive, non-zero integer for a line number"
         self.lineno = num
@@ -692,3 +692,3 @@ def processNonTerminal(ctx, cst):
         if (l >= 4):
-            if (cst[1][0] == symbol.testlist) and (cst[2][0] == symbol.augassign) and (cst[3][0] == symbol.testlist):
+            if (cst[1][0] == symbol.testlist_star_expr) and (cst[2][0] == symbol.augassign) and (cst[3][0] == symbol.testlist):
                 # testlist, augassign, testlist
@@ -878,3 +878,3 @@ def walkCst(ctx, cst):
         for i in range(len(cst)-1, 0, -1):
-            if type(cst[i]) == types.TupleType:
+            if type(cst[i]) == tuple:
                 # Push it onto the processing stack
@@ -893,3 +893,3 @@ def parseSource(sourcecode, indexbuff, indexbuff_len):
     # Parse the source to an Abstract Syntax Tree
-    sourcecode = string.replace(sourcecode, '\r\n', '\n')
+    sourcecode = sourcecode.replace('\r\n', '\n')
     if sourcecode[-1] != '\n':
@@ -912,3 +912,3 @@ def writeIndex(basepath, indexfn, indexbuff, fnamesbuff):
     """
-    fout = open(os.path.join(basepath, indexfn), 'wb')
+    fout = open(os.path.join(basepath, indexfn), 'w', newline='\n')

I realize this is the same error message as issue #4 , but it's still keeping me from using pycscope.

After putting the (restricted) list of python files I want to index into ./cscope.files, and then successfully using /usr/local/bin/pycscope -i ./cscope.files to generate ./cscope.out, I'm still unable to use the generated index:

$ cscope -d -i ./cscope.files -f ./cscope.out
cscope: cannot read list size from file ./cscope.out
$ du -sh cscope.out
25M     cscope.out
$ cscope -V
cscope: version 15.8a
$ 

This is with an older Ubuntu system. I do find multiple references of other people hitting this issue recently-ish, so it can't just by my system. The complaints I've found are about paths containing spaces and about the index being "too large", both of which seem like solvable issues...?

I tried using pycscope on a much smaller subset of python files (making sure none had spaces), yielding a cscope.out index of less than 1 MB, and sure enough now it works. However, I know I've successfully used cscope in C and C++ projects where the cscope.out index was much larger, so this still seems like an issue where pycscope isn't correctly generating the index once it gets "large enough".

After updating emacs, I found that xpycscope.el no longer works because of an elisp function that no longer works. I found a simple fix, replacing two lines (2010, 2142):

(process-kill-without-query xxxx)

is replaced with

(set-process-query-on-exit-flag xxxx nil)

Thanks,
Jeff

When I try to install, I get the following output:

[user ~/github/pycscope]$ python setup.py install
Traceback (most recent call last):
  File "/home/user/github/pycscope/setup.py", line 5, in <module>
    from pycscope import __version__
  File "/home/user/github/pycscope/pycscope/__init__.py", line 25, in <module>
    import keyword, parser, symbol, token
ModuleNotFoundError: No module named 'parser'

The -L option is useful for streaming/offline operation.

From cscope documentation:

-L
Do a single search with line-oriented output when used with the -num pattern option.

Example:
pycscope.py -L3myfunction
Finds functions calling myfunction

Options that cscope supports:

     n  cscope operation
    --- ---------------------------------------
     0  Find this C symbol:
     1  Find this function definition:
     2  Find functions called by this function:
     3  Find functions calling this function:
     4  Find this text string:
     5  Change this text string:
     6  Find this egrep pattern:
     7  Find this file:
     8  Find files #including this file:
     9  Find assignments to this symbol:

It seems that pycscope-1.2.1.tar.gz generates the following error on install for a python2.6 environment:

File "setup.py", line 5, in
from pycscope import version
File "/my_current_directory/pycscope-1.2.1/pycscope/init.py", line 597, in
testlist_comp = (symbol.testlist_comp, symbol.listmaker)
AttributeError: 'module' object has no attribute 'testlist_comp'

Traceback (most recent call last):
File "/home/skumaran/pycscope.py", line 914, in
main()
File "/home/skumaran/pycscope.py", line 154, in main
indexbuff, fnamesbuff = doWork(basepath, gen, lock)
File "/home/skumaran/pycscope.py", line 202, in work
indexbuff_len = parseFile(basepath, fname, indexbuff, indexbuff_len, fnamesbuff)
File "/home/skumaran/pycscope.py", line 288, in parseFile
indexbuff_len = parseSource(filecontents, indexbuff, indexbuff_len)
File "/home/skumaran/pycscope.py", line 887, in parseSource
walkCst(ctx, cst.totuple(True))
File "/home/skumaran/pycscope.py", line 857, in walkCst
processCst(ctx, cst)
File "/home/skumaran/pycscope.py", line 844, in processCst
processNonTerminal(ctx, cst)
File "/home/skumaran/pycscope.py", line 668, in processNonTerminal
ctx.setMark(cst[-1], Mark.FUNC_CALL)
File "/home/skumaran/pycscope.py", line 543, in setMark
assert idx not in self.marks
AssertionError

This was mixed in with the comments on issue 6, which is now closed. I pulled this part out so that it won't get forgotten too easily :-)

I enhanced the dumpCst so that now you can dump any subtree (any cst along the walkCst) rather than just the top cst object. Is it okay to include this in trunk? I find it convenient to be able to look at any subtree (say from inside isTrailerFuncCall) for debugging.

diff --git a/pycscope.py b/pycscope.py
index 58d472c..8b31d22 100755
--- a/pycscope.py
+++ b/pycscope.py
@@ -321,2 +321,4 @@ def replaceNodeType(treeList):
     for i in range(1, len(treeList)):
+        if type(treeList[i]) == tuple:
+            treeList[i] = list(treeList[i])
         if type(treeList[i]) == list:
@@ -330,3 +332,6 @@ def dumpCst(cst):
     try:
-        pprint.pprint(replaceNodeType(cst.tolist(True)))
+        if type(cst) == tuple:
+            pprint.pprint(replaceNodeType(list(cst)))
+        else:
+            pprint.pprint(replaceNodeType(cst.tolist(True)))
     except IOError as e: