Importing local module fails about pycall.jl HOT 30 OPEN

py"""
import sys
sys.path.insert(0, "./directory/path/containing/module")
"""

function_name = pyimport("module_name")["function_name"]

from pycall.jl.

👍 for this. If any one needs a workaround the following can be done

@pyimport importlib.machinery as machinery
loader = machinery.SourceFileLoader("module.name","/abs/path/to/file.py")
my_mod = loader[:load_module]("module.name")
my_mod[:myfunc]()

This loads python files by path. I got the idea from: http://stackoverflow.com/questions/67631/how-to-import-a-module-given-the-full-path

from pycall.jl.

python prepends '' to sys.path, whereas initializing libpython does not, by default.

More technically, we are passing 0 for the updatepath parameter of PySys_SetArgvEx as recommended in the Python library (see here) and as recommended for security reasons.

I'm not sure of the best option here. One option would be to just prepend '' to sys.path automatically all of the time (similar to the python executable?). Another would be to only do this in interactive mode, somehow. Or....?

@loladiro, how does Julia decide whether to import modules from the current directory?

from pycall.jl.

Maybe a custom option to pyinitialize? This should be available from both interactive and noninteractive modes, as python users would expect.

from pycall.jl.

@stevengj Not sure what you mean. If you're talking about include it depends the most recently included file in the current task (the directory of that, or, if none cwd).

from pycall.jl.

@loladiro, if you type using Foo at the Julia prompt, it will load the module from a Foo.jl file in the current directory if it exists. This seems like the equivalent of having '' in the Python path. I'm not sure where this behavior is specified in the Julia source...

Furthermore, it looks like Julia has a similar security issue to Python... it looks like the current directory is searched before the other standard package directories.

from pycall.jl.

@stevengj I think it just does require which falls back to include which uses the mechanism mentioned above.

from pycall.jl.

@loladiro, see my amended comment above... it looks like we may have the same security issue as in Python after all. e.g. if I add a PyPlot.jl to my current directory and do using PyPlot, Julia gets confused.

Or maybe because it is task-dependent this is okay?

from pycall.jl.

It think this is different because using PyPlot in a package will search package directory for a PyPlot.jl. cwd is only applicable at the prompt (I guess we could say the source directory of the prompt is implicitly cwd). I think the issue in python is that the search path gets set globally.

from pycall.jl.

@mlubin, regarding adding an option to pyinitialize, I don't see why this is better than just telling people to modify sys.path, e.g.

unshift!(PyVector(pyimport("sys")["path"]), "")

although one might provide a convenience function for this. (e.g. I could just define PyCall.path as a PyVector mirror of sys.path.) (Note that this only works with the latest version of PyCall, which adds unshift! to PyVectors.)

However, I'd like to do something sensible by default. If Python users are used to being able to load things in the current directory in python, they should be able to do that in PyCall without manually modifying the path. The question is, how can I do this in a way that is reasonably secure?

from pycall.jl.

I could mimic Julia's behavior: every pyimport call could modify sys.path based on the (Julia) directory of the current task. This seems reasonable to me, but I'm not sure whether it might have unexpected consequences in Python.

from pycall.jl.

What if we called Python's os.chdir whenever Julia's cd was called (as well as in pyinitialize)?
EDIT: NM, it's necessary and sufficient to modify sys.path.

from pycall.jl.

How would we modify Julia's cd?

from pycall.jl.

I think I'm not understanding the security issue with always prepending sys.path with "" in pyinitialize. If I execute a file test.jl with python like

import numpy
...

and a file called numpy.py is in the same directory as test.py, then that local numpy.py will be imported. PyCall would then have the same behavior. Is that what you're trying to avoid?

from pycall.jl.

It's not where test.jl is located that is the problem. What we want to avoid is having the current directory in the default path, independent of the location of the file that is being executed. You don't want unexpected behavior to result just from executing Julia from the wrong path.

from pycall.jl.

It seems the path that julia is run in is independent of the search path of python modules (or I'm still not quite getting the issue):

~/a/test.jl:

using PyCall
println(pwd())
unshift!(PyVector(pyimport("sys")["path"]), "")
@pyimport numpy

~/numpy.py:

print "numpy hijacked"

Then in the shell,

cd ~
julia a/test.jl

loads the numpy package, not my rogue version.

from pycall.jl.

What do you think, @stevengj?

from pycall.jl.

You're right, I can't get it to override the numpy module, but maybe I'm missing something in how Python decides what modules to load. See also the Python bug discussion.

from pycall.jl.

Github should have a system to just say: "+1" for a request.
I'd just like to add a: +1 for searching the local directory first.
I just started looking at Julia (I mostly use Python), and so pretty much the first thing I did was trying to import a module in the local directory, which failed.

PyCall seems like a brilliant stroke of genius by the way. Thanks!

from pycall.jl.

@dhoegh that solution doesn't work for me. I get the following error:

julia>@pyimport importlib.machinery as machinery
ERROR: PyError (:PyImport_ImportModule) <type 'exceptions.ImportError'>
ImportError('No module named machinery',)

in pyerr_check at /home/shrif/.julia/v0.3/PyCall/src/exception.jl:61
in pyimport at /home/shrif/.julia/v0.3/PyCall/src/PyCall.jl:81

Any idea if there a machinery equivalent I can use?

from pycall.jl.

What python version are you using? It depends on the python version. I have a version that works for python 2.7 at: https://github.com/dhoegh/Hawk.jl/blob/master/src/Hawk.jl#L13 alternative see: http://stackoverflow.com/questions/19009932/import-abitrary-python-source-file-python-3-3.

from pycall.jl.

I am on 2.7 . After checking now, it seems that python2.7 also has problems importing machinery. Not quite sure why since it's included in site-packages under importlib .

from pycall.jl.

Did you try the import I used in https://github.com/dhoegh/Hawk.jl/blob/master/src/Hawk.jl#L7-13

from pycall.jl.

Yes -- it works just fine, thanks!

However, now I have a new problem. When the local module imports other local modules, the import statement fails.

import-test.jl :

using PyCall

@pyimport imp

filename = abspath(joinpath(dirname(@__FILE__),"callmefirst.py"))
(path,name) = dirname(filename), basename(filename)
(name,ext) = rsplit(name, '.', 2)
(file, filename, data) = imp.find_module(name,[path])
module = imp.load_module(name,file,filename,data)
print("IMPORTED PYTHON")

callmefirst.py:

print "Importing dependencies"
import callme

callme.py:

print "NESTED MODULE SUCCESSFULLY IMPORTED"

running julia import-test.jl returns

Importing dependencies
ERROR: PyError (:PyObject_Call) <type 'exceptions.ImportError'>
ImportError('No module named callme',)

from pycall.jl.

Update: This is fixed when I update callmefirst.py to the following:

import imp
foo = imp.load_source('callme', '/home/shrif/Hyperloop/src/hyperloop/callme.py')
print(foo)

However, this is pretty inconvenient since I'd need to change all my import statements (especially if I'm importing a python framework which relies on calling local modules as such). Any advice on how to get around this?

from pycall.jl.

@sherifnada, does it work to just the current directory to the path, via unshift!(PyVector(pyimport("sys")["path"]), "")?

from pycall.jl.

Worked like a charm. Thanks!!

from pycall.jl.

@dhoegh Thanks for your workaround.

This feature would be great for PyCall.

from pycall.jl.

Is there any easy way to do the same thing as unshift!(PyVector(pyimport("sys")["path"]), ""), but looking in the Conda.jl package installation directory? It'd be nice if PyCall could find packages installed with Conda.jl even if users aren't running the conda python binary.

from pycall.jl.

@rofinn, that's a separate issue. You can certainly unshift! any directory you want, in principle. But it is usually not a good idea to "mix" python distros, i.e. to include in your python path the packages from two separate distros. If you want to use the Conda.jl packages in PyCall, you should configure PyCall to use Conda's python.

from pycall.jl.