I'm trying to profile the following code:

https://github.com/rueckstiess/mtools/blob/8ee80ac14fa1425cfbdaa7c0c0a2c69f36ecae21/mtools/util/logevent.py

I have added @profile to the duration method as follows:

    @profile
    @property
    def duration(self):
        """ calculate duration if available (lazy) """

However, when I run the code via kernprof, it gives me the following error:

kernprof -l -v mloginfo --queries mongodb.log
/usr/local/lib/python2.7/site-packages/line_profiler.py:62: UserWarning: Could not extract a code object for the object <property object at 0x110181fc8>
  self.add_function(func)
Wrote profile results to mloginfo.lprof
Timer unit: 1e-06 s

Traceback (most recent call last):
  File "/usr/local/bin/kernprof", line 9, in <module>
    load_entry_point('line-profiler==1.0', 'console_scripts', 'kernprof')()
  File "/usr/local/lib/python2.7/site-packages/kernprof.py", line 221, in main
    execfile(script_file, ns, ns)
  File "/usr/local/bin/mloginfo", line 9, in <module>
    load_entry_point('mtools==1.1.8', 'console_scripts', 'mloginfo')()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 521, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2632, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2312, in load
    return self.resolve()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2318, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python2.7/site-packages/mtools/mloginfo/mloginfo.py", line 3, in <module>
    from mtools.util.logfile import LogFile
  File "/usr/local/lib/python2.7/site-packages/mtools/util/logfile.py", line 1, in <module>
    from mtools.util.logevent import LogEvent
  File "/usr/local/lib/python2.7/site-packages/mtools/util/logevent.py", line 18, in <module>
    class LogEvent(object):
  File "/usr/local/lib/python2.7/site-packages/mtools/util/logevent.py", line 155, in LogEvent
    @property
  File "/usr/local/lib/python2.7/site-packages/line_profiler.py", line 63, in __call__
    if is_generator(func):
  File "/usr/local/lib/python2.7/site-packages/line_profiler.py", line 50, in is_generator
    isgen = (f.__code__.co_flags & CO_GENERATOR) != 0
AttributeError: 'property' object has no attribute '__code__'

Is this to do with the fact it's a method? Or something else?

FYI, if you use os._exit(0) on at least Ubuntu 14.04 with Python 3.5.1 then there will be no profiler output on your console with kernprof -l -v my_script.py my_data.txt.

if __name__ == "__main__":
    from os import _exit
    main()
    _exit(0)

I am trying to install line profiler via pip and getting the aforementioned error.
This is the output of pip install line_profiler.

Downloading/unpacking line-profiler
  Downloading line_profiler-1.0.tar.gz (65kB): 65kB downloaded
  Running setup.py (path:/tmp/pip_build_chris/line-profiler/setup.py) egg_info for package line-profiler
    Could not import Cython. Using the available pre-generated C file.

Installing collected packages: line-profiler
  Running setup.py install for line-profiler
    Could not import Cython. Using the available pre-generated C file.
    building '_line_profiler' extension
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c _line_profiler.c -o build/temp.linux-x86_64-2.7/_line_profiler.o
    In file included from _line_profiler.c:353:0:
    unset_trace.h:1:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
     void unset_trace();
     ^
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c timers.c -o build/temp.linux-x86_64-2.7/timers.o
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c unset_trace.c -o build/temp.linux-x86_64-2.7/unset_trace.o
    unset_trace.c:5:6: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
     void unset_trace() {
          ^
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/_line_profiler.o build/temp.linux-x86_64-2.7/timers.o build/temp.linux-x86_64-2.7/unset_trace.o -o build/lib.linux-x86_64-2.7/_line_profiler.so
    error: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/line_profiler.py'
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip_build_chris/line-profiler/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-JtMfZ_-record/install-record.txt --single-version-externally-managed --compile:
    Could not import Cython. Using the available pre-generated C file.

running install

running build

running build_py

creating build

creating build/lib.linux-x86_64-2.7

copying line_profiler.py -> build/lib.linux-x86_64-2.7

copying kernprof.py -> build/lib.linux-x86_64-2.7

running build_ext

building '_line_profiler' extension

creating build/temp.linux-x86_64-2.7

x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c _line_profiler.c -o build/temp.linux-x86_64-2.7/_line_profiler.o

In file included from _line_profiler.c:353:0:

unset_trace.h:1:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]

 void unset_trace();

 ^

x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c timers.c -o build/temp.linux-x86_64-2.7/timers.o

x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c unset_trace.c -o build/temp.linux-x86_64-2.7/unset_trace.o

unset_trace.c:5:6: warning: function declaration isn’t a prototype [-Wstrict-prototypes]

 void unset_trace() {

      ^

x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/_line_profiler.o build/temp.linux-x86_64-2.7/timers.o build/temp.linux-x86_64-2.7/unset_trace.o -o build/lib.linux-x86_64-2.7/_line_profiler.so

running install_lib

copying build/lib.linux-x86_64-2.7/line_profiler.py -> /usr/local/lib/python2.7/dist-packages

error: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/line_profiler.py'

----------------------------------------
Cleaning up...
Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip_build_chris/line-profiler/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-JtMfZ_-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip_build_chris/line-profiler
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    load_entry_point('pip==1.5.4', 'console_scripts', 'pip')()
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 235, in main
    return command.main(cmd_args)
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 161, in main
    text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 56: ordinal not in range(128)

An error occurs when starting a gunicorn server with kernprof:

kernprof  -l ./bin/gunicorn --paste etc/development.ini --forwarded-allow-ips="127.0.0.1" -t 6000

Error: gunicorn
Wrote profile results to gunicorn.lprof

gunicorn.lprof is then empty:

cat gunicorn.lprof 
���F�_line_profiler�� LineStats���)}���}�(��unit�G>�����퍌timings�}�ub.⏎ 

I'm using Python 3.4.3

Sorry for dumb question, but I couldn't find the answer...

How can I profile function imported from some package?
I've tried putting @Profile decorator into the imported file, but it doesn't work.

Thanks

I use line_profiler on a regular basis, but with a specific python script::

$ kernprof -l scripts/test -n -a7 ~/data/20151128_1M.csv
...
Wrote profile results to test.lprof
$

It run quite well ~1 min but the lprof is almost empty.

$ strings test.lprof
c_line_profiler
LineStats
unitq
timingsq
}ub.
$

There is no multi thread/processus involved but the test script import a function from a .so made in c++ with pythran, can that be related ?

$ python -m line_profiler test.lprof
Timer unit: 1e-06 s

When running kernprof -l -v test.py on the following test.py file:

@profile
def test():
    a = 1
    b = 1
    return a + b

if __name__ == '__main__':
    print test()

I get:

$ kernprof -l -v test.py
Wrote profile results to test.py.lprof
Timer unit: 1e-06 s

Traceback (most recent call last):
  File "/usr/local/bin/kernprof", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/kernprof.py", line 221, in main
    execfile(script_file, ns, ns)
  File "test.py", line 1, in <module>
    @profile
NameError: name 'profile' is not defined

This error appears on:

OS X El Capitan 10.11.2

Python 2.7.11 (default, Dec  8 2015, 09:38:05)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)] on darwin

line-profiler (1.0)

but not on:

Windows 7 SP1
Python 2.7.11 | Anaconda 2.4.0 (64bits)
line-profiler (1.0)

WHY ???

Thanks.

It seems that the RST parser doesn't like section headers with numbers. As a result, the links at the table of contents in README.rst to the changes of the different versions are not working (the id seems to be wrong)

I am using the line_profiler extension in IPython 3.0.0 (Jupyter) notebook from Anaconda, but do not get the line numbers in the simplest example:

%load_ext line_profiler
def myfun():
    a = 0
    for i in xrange(1000):
        a = a + i
%lprun -f myfun myfun()

But get the output without function lines:

Timer unit: 1e-06 s

File: <ipython-input-50-9459fcdfcb3c>
Function: myfun at line 1
Total time: 0.002182 s

Could not find file <ipython-input-50-9459fcdfcb3c>
Are you sure you are running this program from the same directory
that you ran the profiler from?
Continuing without the function's contents.

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     1                                           
     2         1            3      3.0      0.1  
     3      1001         1011      1.0     46.3  
     4      1000         1168      1.2     53.5

What am I doing wrong?

Currently, if a line takes a very long time to execute, the profiler output gets misaligned vertically:

import time

@profile
def func():
    1  # just to show the alignment
    time.sleep(5)

if __name__ == "__main__":
    func()

results in

<elided>
Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     3                                           @profile
     4                                           def func():
     5                                               1
     6         1      5002679 5002679.0    100.0      time.sleep(5) # <-- oops!

Switching to exponential format for very long times should help.

line_profiler installed in anaconda distribution. %lprun runs fine.
however, when I try to kerprof in the following situation:

file: primes.py

@Profile
def primes(n):
if n==2:
return [2]
elif n<2:
return []
s=range(3,n+1,2)
mroot = n ** 0.5
half=(n+1)/2-1
i=0
m=3
while m <= mroot:
if s[i]:
j=(m_m-3)/2
s[j]=0
while j<half:
s[j]=0
j+=m
i=i+1
m=2_i+3
return [2]+[x for x in s if x]

primes(100)

in terminal:
kernprof -l -v primes.py

has the following info:
Wrote profile results to primes.py.lprof
Timer unit: 1e-06 s

Traceback (most recent call last):
File "/Users/Retina/anaconda/envs/py2k/bin/kernprof", line 9, in
load_entry_point('line-profiler==1.0', 'console_scripts', 'kernprof')()
File "build/bdist.macosx-10.5-x86_64/egg/kernprof.py", line 221, in main
File "primes.py", line 8, in
@Profile
NameError: name 'profile' is not defined

The @profile decorator needs to be the most proximate decorator to the function since it registers the function object itself with the profiler. Also, since it replaces the function with a wrapper, subsequent decorators may not work since it necessarily mucks with the signature.

It might be nice to get some better statistics to set some number of times to run the function like what is done with timeit. This way if there is some drift from run to run it will hopefully cancel out on the average. This would also allow for some error bars to be added. If there is already a way to do this, please let me know.

Also, thanks for such an awesome tool. This is my favorite Python profiling tool by far.

@rkern - I have recently installed line_profiler using pip, with no apparent problems. I attempt to run the exact trivial test in #35 and run into the exact same error message. #35 refers to #25 as the resolution to the problem, but reading through #25 I do not see where the problem was resolved. I use OS X with python 2.7.11

Traceback (most recent call last):
File "/Users/aphearin/anaconda/bin/kernprof", line 11, in
sys.exit(main())
File "/Users/aphearin/anaconda/lib/python2.7/site-packages/kernprof.py", line 221, in main
execfile(script_file, ns, ns)
File "hello_world.py", line 2, in
@Profile
NameError: name 'profile' is not defined

In [1]: %lprun -mfoo bar
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/usr/lib/python3.5/site-packages/line_profiler.py in magic_lprun(self, parameter_s)
    302         try:
--> 303             mod = __import__(modname, fromlist=[''])
    304             profile.add_module(mod)

ImportError: No module named 'foo'

During handling of the above exception, another exception occurred:

UnboundLocalError                         Traceback (most recent call last)
<ipython-input-1-52fba8d475dc> in <module>()
----> 1 get_ipython().magic('lprun -mfoo bar')

/home/antony/.local/lib/python3.5/site-packages/IPython/core/interactiveshell.py in magic(self, arg_s)
   2334         magic_name, _, magic_arg_s = arg_s.partition(' ')
   2335         magic_name = magic_name.lstrip(prefilter.ESC_MAGIC)
-> 2336         return self.run_line_magic(magic_name, magic_arg_s)
   2337 
   2338     #-------------------------------------------------------------------------

/home/antony/.local/lib/python3.5/site-packages/IPython/core/interactiveshell.py in run_line_magic(self, magic_name, line)
   2255                 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
   2256             with self.builtin_trap:
-> 2257                 result = fn(*args,**kwargs)
   2258             return result
   2259 

/usr/lib/python3.5/site-packages/line_profiler.py in magic_lprun(self, parameter_s)
    304             profile.add_module(mod)
    305         except Exception as e:
--> 306             raise UsageError('Could not find module %r.\n%s: %s' % (name,
    307                 e.__class__.__name__, e))
    308 

UnboundLocalError: local variable 'name' referenced before assignment

The fix is probably just name->modname in line_profiler.py line 306.

I'm new to Python, and I've been searching for a profiler that can produce flame graphs, like perl's NYTProf does.

Is there an easy way for line_profiler to generate output suitable for input to flamegraphs.pl ?
See http://www.brendangregg.com/flamegraphs.html for more.

Thanks!

I use this script to profile.

But when I run kernprof, get the error like this:

R:\>kernprof -l example.py
Traceback (most recent call last):
  File "C:\Anaconda\Scripts\kernprof-script.py", line 9, in <module>
    load_entry_point('line-profiler==1.0', 'console_scripts', 'kernprof')()
TypeError: main() takes exactly 1 argument (0 given)

I use python(anaconda) on windows x64, and have installed line-profiler 1.0 already.

Can you give me some advises?

I installed line_profiler with "pip install line_profiler" command, and tried to use it by calling "kernprof.py -l -v MYFILE.py". I then got an error "-bash: kernprof.py: command not found".

I located kernprof.py at ~/anaconda/lib/python2.7/site-packages/ so I don't know why the command is not found. I couldn't find the answer from either stack overflow or the close issues here. Does anyone know what is the problem?

The future package provides a builtins module that masquerades as a Python 3 builtins module, but it is a read-only namespace. Injecting into it doesn't affect the actual __builtin__ namespace.

I write lots of exploratory data processing scripts with a lot of code at the top level outside any function, and I desperately want to profile this code with line_profiler. I came up with a sort of a monkey patch solution to allow profiling top-level code which I've put into a gist here:

https://gist.github.com/jmuhlich/b6e349c5d80ec3897a77

I would be willing to work up a pull request if it would be considered.

Some reason I have a tough time counting columns (zeros), so I am wondering if there is an option to convert timer units to seconds when using python -m line_profiler ....

line_profiler has saved me a lot of time, by the way, so keep up the great work!

I have not been able to get line_profiler working. As an example, I tried to do line-by-line profiling of a simple script that finds all prime numbers less than 100. The script is as follows:

@profile                                                                         
def primes(n):                                                                   
    if n==2:                                                                    
        return [2]                                                              
    elif n<2:                                                                   
        return []                                                                
    s=range(3,n+1,2)                                                            
    mroot = n ** 0.5                                                            
    half=(n+1)/2-1                                                               
    i=0                                                                          
    m=3                                                                          
    while m <= mroot:                                                            
        if s[i]:                                                                 
            j=(m*m-3)/2                                                          
            s[j]=0                                                               
            while j<half:                                                        
                s[j]=0                                                           
                j+=m                                                             
        i=i+1                                                                    
        m=2*i+3                                                                  
    return [2]+[x for x in s if x]                                               
print primes(100)                                                                

I get the following error when calling the following in terminal:

[kcrum]coding_space/test_code > kernprof -l -v primes.py 
Wrote profile results to primes.py.lprof
Timer unit: 1e-06 s

Traceback (most recent call last):
  File "/usr/local/bin/kernprof", line 9, in <module>
    load_entry_point('line-profiler==1.0', 'console_scripts', 'kernprof')()
  File "/usr/local/lib/python2.7/site-packages/kernprof.py", line 221, in main
    execfile(script_file, ns, ns)
  File "primes.py", line 1, in <module>
    @profile
NameError: name 'profile' is not defined

Any help understanding this error would be greatly appreciated. For the record, I'm running python 2.7.9 on a Mac, and I'm running kernprof 1.0b2, which is what pip installed.

After execute:
alan@ar2:~/Document/$ python -m line_profiler S-SEBI_Orientado.py.lprof
I got this error:

Function: getEntrada at line 75
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/lib/python2.7/dist-packages/line_profiler.py", line 394, in
main()
File "/usr/local/lib/python2.7/dist-packages/line_profiler.py", line 391, in main
show_text(lstats.timings, lstats.unit)
File "/usr/local/lib/python2.7/dist-packages/line_profiler.py", line 227, in show_text
show_func(fn, lineno, name, stats[fn, lineno, name], unit, stream=stream, stripzeros=stripzeros)
File "/usr/local/lib/python2.7/dist-packages/line_profiler.py", line 201, in show_func
'%5.1f' % (100*time / total_time))
ZeroDivisionError: float division by zero

Thanks for the line_profilers -- does exactly what I needed: wanted to see what takes most of the time in the unittest. But to get such information I needed 4 lines at minimum...

+import line_profiler
+prof = line_profiler.LineProfiler()

+@prof
 def test_test():
@@ -108,10 +111,12 @@ 
      # end of the test_test
+    prof.print_stats()

I think it would be neat if I could just do

+from line_profiler import LineProfiler

+@LineProfiler(print_stats=True)
 def test_test()

to accomplish the mission, i.e. if decorator could just print those stats after completing the test.

This is what the .lprof file looks like after I run it on my script:

I get no warnings or errors. What can I do?

I'm using OSX Yosemite, and trying to time a script of mine in Python 3.4.1. I have 2 issues:

• if I don't run as sudo, I get an error when running kernprof
• even as sudo, the .lprof file comes up empty of times. Here's a dump of the screen (i deleted some sensitive lines)

$ sudo kernprof -lv ./emai-extractor.py
Wrote profile results to emai-extractor.py.lprof
Timer unit: 1e-06 s

Total time: 0 s
File: ./emai-extractor.py
Function: log_to_db at line 232

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   232                                           @profile
   233                                           def log_to_db(data_in):
   234                                               try:
   235                                                   log = {"datetime":datetime.datetime.now().strftime('%x %X'),
   236                                                        "sendername":data_in[0],
   237                                                        "senderemail":data_in[1][0],
   238                                                        "subject": data_in[2],
   239                                                        "recordscleaned": data_in[3],
   240                                                        "email_list": data_in[4],
   241                                                        "seconds_elapsed": data_in[5]
   242                                                   }
   243                                                   return db.general_log.insert(log)
   244                                               except Exception as e:
   245                                                   logging.error("{} (exception: {})".format("failed to write general_log in DB", e))

Total time: 0 s
File: ./emai-extractor.py
Function: send_results at line 264

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   264                                           @profile
   265                                           def send_results(to_inp, extra_message=None, file_inp=None):
   266                                           
   267                                               recipients = [to_inp[1]] + CC + BCC
   268                                               msg = MIMEMultipart('mixed')
   269                                               msg['From'] = FROM_TEXT
   270                                               msg['To'] = to_inp[1]
   271                                               msg['Date'] = formatdate(localtime=True)
   272                                               msg['Subject'] = 'Your cleaned up email list is ready!'
   273                                           
   274                                               try:
   275                                                   fname = to_inp[0].split()[0]
   276                                               except:
   277                                                   fname = ''
   278                                           
  
   304                                           
   305                                               part1 = MIMEText(plain_text, 'plain')
   306                                               part2 = MIMEText(html, 'html')
   307                                           
   308                                               msg.attach(part2)
   309                                               #msg.attach(part1)
   310                                           
   311                                               attachment = MIMEBase('application', "octet-stream")
   312                                               attachment.set_payload(file_inp.getvalue())
   313                                               encoders.encode_base64(attachment)
   314                                               attachment.add_header('Content-Disposition', 'attachment; filename="' +
   315                                                               datetime.datetime.now().strftime('%x') + ' EMAI-List.csv"')
   316                                               msg.attach(attachment)
   317                                           
   318                                               try:
   319                                                   server = smtplib.SMTP(IMAP_HOST, 587)
   320                                                   server.ehlo()
   321                                                   server.starttls()
   322                                                   server.login(LOGIN, PASSWORD)
   323                                                   server.sendmail(LOGIN, recipients, msg.as_string())
   324                                                   server.close()
   325                                                   logging.info('send_results(): Successfully sent mail!')
   326                                                   return True
   327                                               except Exception as e:
   328                                                   logging.error("{} (exception: {})".format("send_results(): failed to send mail", e))
   329                                                   raise

Total time: 0 s
File: ./emai-extractor.py
Function: make_extra_msg at line 331

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   331                                           @profile
   332                                           def make_extra_msg(senderemail, **kwargs):
   333                                               try:
   334                                                   buf = []
   335                                           
   336                                                   pipe = []
   337                                                   pipe.append({'$match': {'senderemail': senderemail}})
   338                                                   pipe.append({'$group': {'_id':'$senderemail', 'RecordsCleaned':{'$sum':'$recordscleaned'}}})
   339                                           
   340                                                   res = db.general_log.aggregate(pipeline=pipe)
   341                                                   if res:
   342                                                       records_cleaned = res['result'][0]['RecordsCleaned']
   343                                           
   344                                                   s = db.detailed_log.find({'sender': senderemail}).distinct('email')
   345                                                   if s:
   346                                                       unique_emails = len(s)
   347                                           
   348                                                   #if records_cleaned and unique_emails:
   349                                                   #    buf = "
Stats for your account " + str(to_inp[1]) + ":
 " \
   350                                                   #               "- " + str(unique_emails) + " unique emails
" \
   351                                                   #                "-  " + str(records_cleaned) + " total emails" \
   352                                                   #               "
"
   353                                           
   354                                                   if unique_emails:
   355                                                       txt = "- " + str(unique_emails) + " unique emails processed
"
   356                                                       buf.append(txt)
   357                                           
   358                                                   if records_cleaned:
   359                                                       txt = "- " + str(records_cleaned) + " total emails sent our way
"
   360                                                       buf.append(txt)
   361                                           
   362                                                   if kwargs is not None:
   363                                                       for key, value in kwargs.items():
   364                                                           txt = "- " + str(value) + " " + key + "
"
   365                                                           buf.append(txt)
   366                                           
   367                                                   if len(buf) > 0:
   368                                                       buf.append("
")
   369                                                       txt = "
Global info for your account " + str(senderemail) + ":
"
   370                                                       buf.insert(0, txt)
   371                                           
   372                                                   return ''.join(buf)
   373                                               except Exception as e:
   374                                                   logging.error(e, exc_info=True)
   375                                                   return ''

Yes, I have followed directions in #25

That is, I have installed future package and installed line_profiler from master branch (by running python setup.py install).

However, I still get:

  File "/export/home/myuser/mysite/mysite/views/aview.py", line 207, in <module>
    @profile
NameError: name 'profile' is not defined

Command:

 kernprof -l -v /export/home/myuser/mysite/ve/bin/pserve development.ini --reload   

System: RHEL 6.3, Python 2.7.10 x64.

Hi

Anyone seen this error before with readsettings?

File "/volume1/Downloads/Scripts/readSettings.py", line 11, in <module> from babelfish import Language ImportError: No module named babelfish

Cant seem to figure out what the issue is, or if i am missing some installation.

I used the following readsetting - https://github.com/mdhiggins/sickbeard_mp4_automator/blob/master/readSettings.py

My system: python 3.4 (64 bit) via Anaconda3 - 2.1.0 on Windows 7. I installed line_profiler via pip

When I run kernprof script.py I don't have any problems. However, when I try to run kernprof -l script.py, I get a message saying "python.exe has stopped working". The error seems happen when executing import line_profiler, which kills python.

I was previously using the fork for Python 3 that Mikhail Korobov had created, and didn't have any issues.

Current version on PyPI (1.0) errors out when loading the pickled data. This is a virtualenv'ed Python on Fedora 20.

(virtualenv)rbu@saul ~ $ cat bla.py 
@profile
def hello():
    print('Hello, world!')

hello()


(virtualenv)rbu@saul ~ $ kernprof --line-by-line bla.py
Hello, world!
Wrote profile results to bla.py.lprof

(virtualenv)rbu@saul ~ $ python -m line_profiler bla.py
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/rbu/devel/virtualenvs/virtualenv/lib/python2.7/site-packages/line_profiler.py", line 394, in <module>
    main()
  File "/home/rbu/devel/virtualenvs/virtualenv/lib/python2.7/site-packages/line_profiler.py", line 390, in main
    lstats = load_stats(args[0])
  File "/home/rbu/devel/virtualenvs/virtualenv/lib/python2.7/site-packages/line_profiler.py", line 380, in load_stats
    return pickle.load(f)
cPickle.UnpicklingError: invalid load key, '@'.

(virtualenv)1 rbu@saul ~ $ python --version
Python 2.7.5

(virtualenv)rbu@saul ~ $ hexdump -C  bla.py.lprof    
00000000  80 02 63 5f 6c 69 6e 65  5f 70 72 6f 66 69 6c 65  |..c_line_profile|
00000010  72 0a 4c 69 6e 65 53 74  61 74 73 0a 71 01 29 81  |r.LineStats.q.).|
00000020  71 02 7d 71 03 28 55 04  75 6e 69 74 71 04 47 3e  |q.}q.(U.unitq.G>|
00000030  b0 c6 f7 a0 b5 ed 8d 55  07 74 69 6d 69 6e 67 73  |.......U.timings|
00000040  71 05 7d 71 06 55 06 62  6c 61 2e 70 79 71 07 4b  |q.}q.U.bla.pyq.K|
00000050  01 55 05 68 65 6c 6c 6f  71 08 87 5d 71 09 4b 03  |.U.helloq..]q.K.|
00000060  4b 01 4b 1e 87 71 0a 61  73 75 62 2e              |K.K..q.asub.|
0000006c

I haven't been using line_profiler for a long time so I don't know if it is a dumb question

Let's say this is prof.py

import os

@profile
def foo():
    print('hello')
    print('hello')
    os.chdir('/home/')
    print('hello')
    print('hello')

if __name__ == '__main__':
    foo()

Now I do

sudo kernprof -l prof.py

Output:

hello
hello
hello
hello
Wrote profile results to prof.py.lprof

And then

python -m line_profiler prof.py lprof

Output:

Timer unit: 1e-06 s

Total time: 7.6e-05 s
File: prof.py
Function: foo at line 3

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     3                                           @profile
     4                                           def foo():
     5         1           53     53.0     69.7     print('hello')
     6         1           13     13.0     17.1     print('hello')
     7         1           10     10.0     13.2     os.chdir('/home/')
     8                                              print('hello')
     9                                              print('hello')

I don't know what's happened in here. Is this solvable?

On windows, if the python script is encoded with utf-8 while the system default encoding is gbk, when running kernprof -l it would throw error like:

Traceback (most recent call last):
  File "C:\Anaconda3\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Anaconda3\Scripts\kernprof.exe\__main__.py", line 9, in <module>
  File "C:\Anaconda3\lib\site-packages\kernprof.py", line 221, in main
    execfile(script_file, ns, ns)
  File "C:\Anaconda3\lib\site-packages\kernprof.py", line 34, in execfile
    exec_(compile(f.read(), filename, 'exec'), globals, locals)
UnicodeDecodeError: 'gbk' codec can't decode byte 0xaa in position 553: illegal multibyte sequence

I currently workaround this by converting the python file to gbk first, run kernprof. If I try to view the FILE.lprof file with python -m line_profiler FILE.lprof now, it would also give encoding error, and then I have to convert the python script back to utf-8 and run python -m line_profiler FILE.lprof to view the results. Is there a better way?

Hi

I have installed sabnzbd and sickbeard on my DS112+ and been trying to set up a few things to things get automatically get transcoded once downloaded.

At the moment i have been having trouble with the following script:

sabToSickBeardwithCoverter.py

It runs but then i get the following error when it does post processing:

       Trying to install SetupTools
        Traceback (most recent call last):
       File "/volume1/Downloads/Scripts/sabToSickBeardwithConverter.py", line 6, in <module>
       from readSettings import ReadSettings
        File "/volume1/Downloads/Scripts/readSettings.py", line 9, in <module>
       setup.ez_setup.main()
         NameError: name 'setup' is not defined

I have tried to install the setup tool now a few times and it shows as it has downloaded and installed fine but still getting that error on post processing. I am very new to all of this so been googling for last few days but anything i try to install with cmd just doesnt seem to be working.

Any had same experience?

For example how would I line profile this view function: https://github.com/tomchristie/django-rest-framework/blob/1e9ea377e3f0521ec5d9c21527cfad3c7626baaa/rest_framework/viewsets.py#L61 ?

I want to run kernprof from the root of my package source like kernprof -l mylib\test\test.py -v but then any import within test.py fails, e.g. import mylib.foo because the current working dir is not added to the python path when running kernprof, which in my opinion should happen by default. The workaround is to temporarily hack sys.path within this specific module, but that's annoying.

Hello

I wrote the following code:

@profile
def hello():
  print('Testing')

hello()

And called

kernprof -l /tmp/hello.py

Here is the output

Wrote profile results to alignAll.py.lprof
Traceback (most recent call last):
  File "/usr/local/bin/kernprof", line 9, in <module>
    load_entry_point('line-profiler==1.0', 'console_scripts', 'kernprof')()
  File "/usr/local/lib/python2.7/dist-packages/kernprof.py", line 221, in main
    execfile(script_file, ns, ns)
  File "directPixelBased/alignAll.py", line 9, in <module>
    @profile
NameError: name 'profile' is not defined

Ubuntu 14.04 with Python 2.7.6

Thanks
Ajay

import unittest
class Foo(unittest.TestCase):
       @profile
       def test_me(self):
            print "hi there"
if __name__ == "__main__":
    unittest.main()

if invoked with kernprof -vvl test.py it correctly shows the code to be profiled, but does not execute anything. Maybe I did something wrong or is this expected not to work?

It's unclear to me as to whether this is required, but after running without using this decorator anywhere, I got no results, so I'm assuming it is. I tried importing line_profiler in ipython and seeing if the decorator was there, but I didn't see it. I'm digging through the code now to see if I can figure it out.

When I try to run the command python -m line_profiler 1e3fdc80f5419f682e65b410f2a2ef92135d3d046_1800.py.lprof

I get the error:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/local/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/lib/python2.7/dist-packages/line_profiler.py", line 20, in
from _line_profiler import LineProfiler as CLineProfiler
ImportError: /usr/local/lib/python2.7/dist-packages/_line_profiler.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8

Is this a problem with my pip or is it a bug? I am betting that it is more of a problem with pip, however how do I go about fixing this

After running kernprof -l myscript.py,
then python -m line_profiler myscript.py.lprof

this is all that is displayed
Timer unit: 3.30314e-07 s

I'm using Python 2.7.6 32-Bit on Windows 7 Pro

I often use strings with non-ascii characters, both in code and in documentation (comments or docstrings). When I run line_profiler using kernprof -lv myscript.py, the output prints until the first line with a non-ascii character where I get this error:

Traceback (most recent call last):
  File "/usr/bin/kernprof", line 9, in <module>
    load_entry_point('line-profiler==1.0', 'console_scripts', 'kernprof')()
  File "/usr/lib/python2.7/site-packages/kernprof.py", line 230, in main
    prof.print_stats()
  File "/usr/lib/python2.7/site-packages/line_profiler.py", line 117, in print_stats
    show_text(lstats.timings, lstats.unit, stream=stream, stripzeros=stripzeros)
  File "/usr/lib/python2.7/site-packages/line_profiler.py", line 227, in show_text
    show_func(fn, lineno, name, stats[fn, lineno, name], unit, stream=stream, stripzeros=stripzeros)
  File "/usr/lib/python2.7/site-packages/line_profiler.py", line 215, in show_func
    stream.write(txt)
  File "/usr/lib/python2.7/codecs.py", line 351, in write
    data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 84: ordinal not in range(128)

My files are in utf-8, and declared as such using # -*- coding: utf-8 -*-.

define_magic has apparently been deprecated for a while and is now removed: https://github.com/ipython/ipython/pull/9504/files
Starting IPython with the line_profiler extension will fail with

Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/IPython/core/shellapp.py", line 246, in init_extensions
    self.shell.extension_manager.load_extension(ext)
  File "/usr/lib/python3.5/site-packages/IPython/core/extensions.py", line 85, in load_extension
    if self._call_load_ipython_extension(mod):
  File "/usr/lib/python3.5/site-packages/IPython/core/extensions.py", line 132, in _call_load_ipython_extension
    mod.load_ipython_extension(self.shell)
  File "/usr/lib/python3.5/site-packages/line_profiler.py", line 372, in load_ipython_extension
    ip.define_magic('lprun', magic_lprun)
AttributeError: 'TerminalInteractiveShell' object has no attribute 'define_magic'

Support for running package.__main__ via kernprof.py -m package would be appreciated, thanks.

In my hands, line_profiler is failing to record information when multiprocessing.Pool.map is used to call the functions I'm targetting. I can probably put together a minimal test case, but I thought first I would check whether this is a known issue.

Just a quick note, I had an exception until I edited show_func() to have a guard for max(linenos) like this:

    if len(linenos):
        nlines = max(linenos) - min(min(linenos), start_lineno) + 1
    else:
        nlines = 1

Hope this helps. Great module btw, amazing how short a line profiler can be.

Hi,

Changelog includes this line:

• ENH: Support for IPython cell blocks. Thanks to Michael Forbes for adding this feature.

Does this mean that we can run profiler in Notebook via cell block magic (i.e. profile cell block without making it function), as I don't see that it is currently possible?

Searching for @mforbes I cant see that he contributed to this repo.

Thanks

I've found it a bit annoying to comment out the @Profile decorations so I wrote a really simple decorator to use instead of profile. Actually, it’s a bit obvious so maybe there is another way to do this already which is better; I just couldn't find it ☺

It took an hour to look up the syntax and get it working so I thought it might be useful. I use another decorator to check if the profile decorator is available. Separating the second profiled_func assignment from its execution is to ensure that NameError exceptions, from func, are still propagated and the try...except block only detects if profile is present.

def proxyprofile(func):
    def profile_check_proxy(*args, **kwargs):
        profiled_func = func
        try:
            profiled_func = profile(func)
        except NameError:
            pass
        return profiled_func(*args, **kwargs)
    return profile_check_proxy

I'm working on an atom plugin capable of parsing output from line_profiler. At times it can be painful to explicitly mark all the functions to profile. Would it be possible to add an option to record samples from all functions (or maybe whenever the 'added' function is part of the stack trace) ?

Hi Robert. I'm having trouble getting kernprof to run on a Windows box (). The first confusing bit was kernprof --version showing 1.0b2 when I'd built from source, I was expecting something like 1.0 (having gone via pip and thinking it was an old release).

Looking at kernprof.py's OptionParser it looks like 1.0b2 is hard-coded. Is that a minor oversight? Or does the 1.0 code live somewhere else? I wanted to rule out any Python 3 issues.
https://github.com/rkern/line_profiler/blob/master/kernprof.py#L152

Regarding my Windows issue, that's for Python 3.4 via Anaconda on a recent Windows, I'll file a report for that in the morning if we can't get it fixed (it segfaults, I see someone else posted a light bug report that looks similar). I'm using a similarly recent build of line_profiler on my linux machine with no crashes.