Struggling with ray import about pyinstaller-hooks-contrib HOT 20 CLOSED

Thank you for the reply. It makes much more sense now. You are totally right about spawning subprocesses where the correct entry point has already been lost due to the freezing approach. However, news is good!

I was thinking whether the spawning subprocesses is absolutely required or not, and started to explore the source code. It turned out that subprocess spawning points can be skipped if an ip address of a ray runtime (which was simply auto for automatic detection in my case as I'm running in a single node) is provided and there is a runtime that has already been started in that address via CLI command ray start --head. This means that the subprocess spawning lines are called from the library itself, if a copy of ray library is kept outside of the binary so the main problem we're talking here is not a concern anymore. I've added a single subprocess.call(bla, bla) line before ray.init('auto') line to initiate the processes through CLI and this workaround worked like a charm.

I'm writing this as there is no information about using ray with pyinstaller on the entire Internet as far as I saw. Thank you for your attention and fast replies!

from pyinstaller-hooks-contrib.

No. Unless ray changes their offending DIY multiprocessing implementation, PyInstaller will never be able to work with ray. There's nothing we can do our end. As for alternatives, I'm afraid all other freezing applications (Nuitka and cxfreeze) will have the same issue - you need a real Python installation to run it. For Windows, you should be able to install your project into one of the embedable builds from python.org and ship that, for Linux install your project into a docker image, for macOS I don't know. If your user base is a bit more technical then realistically I think the best cross way to ship Python projects is to make them into setuptools distributions and put it somewhere public then all your users need to do is install Python then run pip install https://github.com/your-username/your-project.git and it'll do the rest.

from pyinstaller-hooks-contrib.

Hey @mishashevelinn,
Let me put the exact call I've ended up here (note that I've changed the use of subprocess e.g. subprocess.call() to subprocess.Popen()):

init_ray_process = subprocess.Popen(
    [
        'ray',
        'start',
        '--head',
        '--port=6379',
        '--include-dashboard=False',
        '--num-gpus=1',
        '--num-cpus={}'.format(resources['num_threads']),
        '--object-store-memory={}'.format(resources['object_store_memory']),
        '--memory={}'.format(resources['ray_memory']),
        '--redis-max-memory={}'.format(resources['redis_max_memory'])
    ],
    stderr=subprocess.PIPE,
    universal_newlines=True
)

_, stderr = init_ray_process.communicate()

I hope this helps you, at least gives you a general idea. Let me elaborate what I mean by "keeping ray outside". It is not related to the folder structure. First thing's first; ray had already been added to the .spec file to collect its data files during the compiling process via PyInstaller. However, the compiled executable did not work as ray cannot be used in such a way (see the discussion above). The only way to make my executable use ray is having it installed outside. Therefore, while dockerizing the working environment with the executable, I just installed ray as a python package that can be accessible outside the executable itself. This is why I'm calling it via subprocess. I hope this little elaboration makes it a bit clearer.

from pyinstaller-hooks-contrib.

@bcaldir we also have Pyinstaller app as well and got ray packaged with it. When you run your subprocess call is the ray library still package within the exe or is it in a separate location?

Thanks for all the interesting discussions. I came across exactly a same scenario as you mentioned where I have to package the exe and share it with the customer, hence cannot use the other suggestion in the thread where a different subprocess is opened.

@quaesitor-scientiam I am curious to understand what files (2 exe referred in your comment) you added and where did you keep it. Also, did you happen to use add-data in pyinstaller to add it in your exe package? It would be great if you can kindly help share these details..
I am using ray=2.2.0 and I see only one executable in my Python virtual env root, inside Scripts folder as ray.exe. Please share if you are referring to something else

from pyinstaller-hooks-contrib.

Hmm, this looks like it will get ugly.

https://github.com/ray-project/ray/blob/e5caaa7d1f6456cde31628ff048d68928bfe9337/python/ray/_private/services.py#L987-L989

Those assertions will fail in PyInstaller because modules are not files - rather they're zipped up into an archive embedded in the EXE. I'll have a go at bullying my way through with collect_data_files(..., py_files=True) but I doubt it'll work...

from pyinstaller-hooks-contrib.

Setting datas=collect_data_files("ray", include_py_files=True) in the spec file fixes the assertion but causes a huge infinite loop which crashed my terminal. Looking at my process manager I see it's firing off all kinds of sub processes so this loop will be hard to trace.

from pyinstaller-hooks-contrib.

Looking back at this is see this line which means that the subprocesses being launched is another instance of pyinstaller/pyinstaller#5902 (comment) which we can't solve.

from pyinstaller-hooks-contrib.

@yolking @bwoodsend Is there any update on this one? I have to build my application using ray into an executable and I've been using PyInstaller all the time. I'm struggling with this problem nowadays and couldn't fix it. If this is absolutely impossible building it with PyInstaller in any ways, what do you recommend?

from pyinstaller-hooks-contrib.

I've switched to aiomultiprocess for my applications. It's much more specific though, but works with PyInstaller . I still use ray for other tasks, but they don't require .exe version. Maybe you should look at other multiprocess libraries too if it's possible. I haven't tried committing new issue on ray github - got the feeling they wouldn't care about this problem. Maybe you should.

from pyinstaller-hooks-contrib.

Thank you both of you for your fast replies. Those are very helpful! However, I have to use ray for multiple reasons and it's not an option replacing it to a multiprocess library at least for now.

One last question. I was thinking of the following, what would you think about such a work around?

1. Excluding the ray library by setting excludes=['ray'] during Analysis object instantiation in the spec file (not sure if this would be necessary or change anything in this context)
2. Installing the library via pip (this could be done into a separate venv as well, again not sure another env would change anything)
3. Specifying the path of the site-packages folder under env (or venv) by setting pathex=['path/to/site-packages'] during Analysis object instantiation in the spec file.

It sounds a correct way of making the executable to see a folder of libraries that are not embedded into itself. What would you say about such an approach?

from pyinstaller-hooks-contrib.

It sounds a correct way of making the executable to see a folder of libraries that are not embedded into itself. What would you say about such an approach?

excludes and pathex influence only the build process (i.e., the analysis), it has no impact on the search paths in the actual executable. That said, your program could extend the sys.path to include external paths with installed python packages.

But I don't think this would help in ray's case, because the problem is that spawns its sub-processes using sys.executable, which in frozen application is not python interpreter, but frozen application's executable. So the entry point script would need to somehow identify that its subsequent (re)launches are part of ray's attempts to spawn sub-processes, and divert the program flow accordingly (which is what we do for multiprocessing module, by overriding its freeze_support function in our runtime hook).

from pyinstaller-hooks-contrib.

Hi @bcaldir,

Could you elaborate on your solution please? I'm facing the same issue. I'm wondering what exactly I should add before initializing ray. Thanks!

from pyinstaller-hooks-contrib.

Hi @stestoni,

I actually explained the solution in the previous post. If you still have problem, could you specify it?

from pyinstaller-hooks-contrib.

Nevermind @bcaldir, I solved my ray problem simply bypassing the use of pyinstaller.

For the record, though, it just wasn't clear to me what goes into subprocess.call() in your solution.

from pyinstaller-hooks-contrib.

@bcaldir we also have Pyinstaller app as well and got ray packaged with it. When you run your subprocess call is the ray library still package within the exe or is it in a separate location?

from pyinstaller-hooks-contrib.

Hey @quaesitor-scientiam,

ray is both packaged within the resulting exe and installed separately in our case. To call the ray CLI via subprocess you already need a separate copy of it outside of the exe itself. I'm not sure though it should still be packaged or not as it is already available outside of the exe but we'd had enough with this PyInstaller and ray issues that day and just worked around as explained.

from pyinstaller-hooks-contrib.

Thanks, in our case since it is going to customers so having ray separate is not an option. Thanks for the quick reply.

from pyinstaller-hooks-contrib.

You're welcome. I hope you can find another solution as the explained one is basically useless for single executable deployments.

from pyinstaller-hooks-contrib.

@bcaldir got it working via PyInstaller by including the ray executables (2 files) and include them in a local directory of the app. That way the app is able to spin up the Ray head node and process normally. You have to issue the ray stop to cleanup as well.

from pyinstaller-hooks-contrib.

Thank you for the reply. It makes much more sense now. You are totally right about spawning subprocesses where the correct entry point has already been lost due to the freezing approach. However, news is good!

I was thinking whether the spawning subprocesses is absolutely required or not, and started to explore the source code. It turned out that subprocess spawning points can be skipped if an ip address of a ray runtime (which was simply auto for automatic detection in my case as I'm running in a single node) is provided and there is a runtime that has already been started in that address via CLI command ray start --head. This means that the subprocess spawning lines are called from the library itself, if a copy of ray library is kept outside of the binary so the main problem we're talking here is not a concern anymore. I've added a single subprocess.call(bla, bla) line before ray.init('auto') line to initiate the processes through CLI and this workaround worked like a charm.

I'm writing this as there is no information about using ray with pyinstaller on the entire Internet as far as I saw. Thank you for your attention and fast replies!

@bcaldir Thank you for providing a workaround.
As it was mentioned here, It's not clear what to pass to subprocess.call(bla, bla)
In addition - can you elaborate on the "ray kept outside"? What folder structure is possible?
Thanks a lot!

from pyinstaller-hooks-contrib.