dill.source.getsource() does not return expected results in IPython terminal or Notebook about dill HOT 11 CLOSED

This would be helpful to have the traceback, but I'm sure I will be able to reproduce this.

dill.source.getsource uses readline to get a function from history. @ivanov mentioned that ipython uses it's own history buffer and not readline, so I'm guessing that you are seeing a NameError… and your code would work in IPython -- just not the notebook. @vinc456: is that correct?

from dill.

Actually it's an "IOError: could not get source code" error.

The follow getsource() call works as expected in an IPython session running from the Windows command line.

In [1]: def doubler(f):
   ...:         def func(*args, **kwds):
   ...:                 return 2 * f(*args, **kwds)
   ...:         return func
   ...:

In [2]: @doubler
   ...:
   ...: def double_squared(x):
   ...:         return x**2
   ...:

In [3]: double_squared(5)
Out[3]: 50

In [4]: import dill

In [5]: print dill.source.getsource(double_squared) # returns the expected result
%%javascript
from ctypes import CDLL
# This will crash a linux system; equivalent calls can be made on Windows or Mac

libc = CDLL("libc.so.6")
libc.time(-1)  # BOOM!!
from time import sleep
sleep(2)
print 'test'
from time import sleep
sleep(2)
print 'hi'
%load soln/load.py
import dill
%clear
import dill
>>> def foo(x):

However, when running in the IPython Notebook or the IPython QT Console there is the following IOError.

print dill.source.getsource(double_squared)
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-5-a2b1f49c66be> in <module>()
----> 1 print dill.source.getsource(double_squared)

C:\Users\vinc456\Documents\GitHub\dill\dill\source.pyc in getsource(object, alias, lstrip, enclosing, force, builtin)
    324     # get source lines; if fail, try to 'force' an import
    325     try: # fails for builtins, and other assorted object types
--> 326         lines, lnum = getsourcelines(object, enclosing=enclosing)
    327     except (TypeError, IOError): # failed to get source, resort to import hooks
    328         if not force: # don't try to get types that findsource can't get

C:\Users\vinc456\Documents\GitHub\dill\dill\source.pyc in getsourcelines(object, lstrip, enclosing)
    295     If lstrip=True, ensure there is no indentation in the first line of code.
    296     If enclosing=True, then also return any enclosing code."""
--> 297     code, n = getblocks(object, lstrip=lstrip, enclosing=enclosing, locate=True)
    298     return code[-1], n[-1]
    299 

C:\Users\vinc456\Documents\GitHub\dill\dill\source.pyc in getblocks(object, lstrip, enclosing, locate)
    221     DEPRECATED: use 'getsourcelines' instead
    222     """
--> 223     lines, lnum = findsource(object)
    224 
    225     if ismodule(object):

C:\Users\vinc456\Documents\GitHub\dill\dill\source.pyc in findsource(object)
    125 
    126     if not lines:
--> 127         raise IOError('could not get source code')
    128 
    129     #FIXME: all below may fail if exec used (i.e. exec('f = lambda x:x') )

IOError: could not get source code

from dill.

I'll clean up the code and send you the IPython notebook file I was working during my layover today.

from dill.

Could you add triple backticks before and after the code, so that github doesn't interpret it as md.

from dill.

@vinc456: thanks. It's hard to parse your comment above with the traceback. Can you edit the markup as suggested?

from dill.

I've cleaned up the Markdown.

By the way, the Dill Primer I was working on can be found at https://dl.dropboxusercontent.com/u/1047492/Dill%20Primer.ipynb

There's probably a better way to share IPython Notebook files but I emailed a copy to John Tyree, who was going to have a look and maybe convert it to a .rst file. It's based on the typescript demo session Mike showed me at the first day of the SciPy 2014 Dill Sprint.

from dill.

Actually now that I have a closer look it seems like dill.source.getsource(double_squared) is not returning the expected result even in the case of the IPython terminal. I'm not sure what is going on here.

In [5]: print dill.source.getsource(double_squared) # not sure what is going on here
%%javascript
from ctypes import CDLL
# This will crash a linux system; equivalent calls can be made on Windows or Mac

libc = CDLL("libc.so.6")
libc.time(-1)  # BOOM!!
from time import sleep
sleep(2)
print 'test'
from time import sleep
sleep(2)
print 'hi'
%load soln/load.py
import dill
%clear
import dil

But the same code will return the following results in a python terminal.

>>> print dill.source.getsource(double_squared, enclosing=True) # why should this be any different when called from the Python terminal?
def doubler(f):
    def func(*args, **kwds):
        return 2 * f(*args, **kwds)
    return func

from dill.

@vinc456: Like I said, IPython is not using readline directly, and dill.source assumes using readline for history. IPython apparently has it's own history mechanism, and it's different for both the terminal and the notebook. So, we'll need to figure out what exactly they do.

When dill.source looks at readline history, it converts the buffer into a big string, as it it had been read from a file… and then a good bit of dill.source basically works like inspect does on a file. The part with readline can be abstracted out to a function (it's two lines), and I should be able to swap in whatever IPython needs.

from dill.

@vinc456: I'm assuming this was a suitable conclusion…? or would you prefer a ticket with an option to remove IPython content so it looks like the same you'd get from raw python?

from dill.

Actually it's an "IOError: could not get source code" error.

The follow getsource() call works as expected in an IPython session running from the Windows command line.

In [1]: def doubler(f):
   ...:         def func(*args, **kwds):
   ...:                 return 2 * f(*args, **kwds)
   ...:         return func
   ...:

In [2]: @doubler
   ...:
   ...: def double_squared(x):
   ...:         return x**2
   ...:

In [3]: double_squared(5)
Out[3]: 50

In [4]: import dill

In [5]: print dill.source.getsource(double_squared) # returns the expected result
%%javascript
from ctypes import CDLL
# This will crash a linux system; equivalent calls can be made on Windows or Mac

libc = CDLL("libc.so.6")
libc.time(-1)  # BOOM!!
from time import sleep
sleep(2)
print 'test'
from time import sleep
sleep(2)
print 'hi'
%load soln/load.py
import dill
%clear
import dill
>>> def foo(x):

However, when running in the IPython Notebook or the IPython QT Console there is the following IOError.

print dill.source.getsource(double_squared)
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-5-a2b1f49c66be> in <module>()
----> 1 print dill.source.getsource(double_squared)

C:\Users\vinc456\Documents\GitHub\dill\dill\source.pyc in getsource(object, alias, lstrip, enclosing, force, builtin)
    324     # get source lines; if fail, try to 'force' an import
    325     try: # fails for builtins, and other assorted object types
--> 326         lines, lnum = getsourcelines(object, enclosing=enclosing)
    327     except (TypeError, IOError): # failed to get source, resort to import hooks
    328         if not force: # don't try to get types that findsource can't get

C:\Users\vinc456\Documents\GitHub\dill\dill\source.pyc in getsourcelines(object, lstrip, enclosing)
    295     If lstrip=True, ensure there is no indentation in the first line of code.
    296     If enclosing=True, then also return any enclosing code."""
--> 297     code, n = getblocks(object, lstrip=lstrip, enclosing=enclosing, locate=True)
    298     return code[-1], n[-1]
    299 

C:\Users\vinc456\Documents\GitHub\dill\dill\source.pyc in getblocks(object, lstrip, enclosing, locate)
    221     DEPRECATED: use 'getsourcelines' instead
    222     """
--> 223     lines, lnum = findsource(object)
    224 
    225     if ismodule(object):

C:\Users\vinc456\Documents\GitHub\dill\dill\source.pyc in findsource(object)
    125 
    126     if not lines:
--> 127         raise IOError('could not get source code')
    128 
    129     #FIXME: all below may fail if exec used (i.e. exec('f = lambda x:x') )

IOError: could not get source code

this is still an issue for me, getsource still fails in ipython (jupyter notebook)

from dill.

@pcko1: are you seeing the exact error above with the exact same code, or what?

from dill.