modin-project / unidist Goto Github PK
View Code? Open in Web Editor NEWUnified Distributed Execution
Home Page: https://unidist.readthedocs.io
License: Apache License 2.0
Unified Distributed Execution
Home Page: https://unidist.readthedocs.io
License: Apache License 2.0
Parameters support should be added to unidist.init
method to give more flexibility in setup framework.
Parameters are the next:
Traceback (most recent call last):
File "/localdisk/aprutsko/unidist/unidist/core/backends/mpi/core/worker.py", line 749, in <module>
event_loop()
File "/localdisk/aprutsko/unidist/unidist/core/backends/mpi/core/worker.py", line 675, in event_loop
request = communication.recv_complex_operation(comm, source_rank)
File "/localdisk/aprutsko/unidist/unidist/core/backends/mpi/core/communication.py", line 384, in recv_complex_operation
return recv_complex_data(comm, source_rank)
File "/localdisk/aprutsko/unidist/unidist/core/backends/mpi/core/communication.py", line 291, in recv_complex_data
raw_buffers = [None] * raw_buffers_size
TypeError: can't multiply sequence by non-int of type 'list'
Deps should be added to existing env files.
Unidist can't start mpi
backend on Windows with the next exception:
C:\Users\aprutsko\repositories\unidist>python unidist\cli -m pytest unidist\test\test_general.py -b mpi
Windows fatal exception: access violation
Current thread 0x00005c44 (most recent call first):
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 1174 in exec_module
File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1042 in _handle_fromlist
File "C:\Users\aprutsko\repositories\unidist\unidist\core\backends\mpi\core\communication.py", line 18 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 843 in exec_module
File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "C:\Users\aprutsko\repositories\unidist\unidist\core\backends\mpi\core\executor.py", line 13 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 843 in exec_module
File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "C:\Users\aprutsko\repositories\unidist\unidist\core\backends\mpi\core\__init__.py", line 5 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 843 in exec_module
File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "C:\Users\aprutsko\repositories\unidist\unidist\core\backends\mpi\backend.py", line 10 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 843 in exec_module
File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "C:\Users\aprutsko\repositories\unidist\unidist\core\base\utils.py", line 88 in get_backend_proxy
File "C:\Users\aprutsko\repositories\unidist\unidist\api.py", line 34 in make_decorator
File "C:\Users\aprutsko\repositories\unidist\unidist\api.py", line 118 in remote
File "C:\Users\aprutsko\repositories\unidist\unidist\test\utils.py", line 40 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 843 in exec_module
File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "C:\Users\aprutsko\repositories\unidist\unidist\test\test_general.py", line 10 in <module>
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\assertion\rewrite.py", line 170 in exec_module
File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "<frozen importlib._bootstrap>", line 1014 in _gcd_import
File "C:\Users\aprutsko\.conda\envs\unidist\lib\importlib\__init__.py", line 127 in import_module
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\pathlib.py", line 524 in import_path
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\python.py", line 578 in _importtestmodule
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\python.py", line 500 in _getobj
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\python.py", line 291 in obj
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\python.py", line 516 in _inject_setup_module_fixture
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\python.py", line 503 in collect
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\runner.py", line 341 in <lambda>
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\runner.py", line 311 in from_call
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\runner.py", line 341 in pytest_make_collect_report
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\runner.py", line 458 in collect_one_node
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\main.py", line 808 in genitems
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\main.py", line 634 in perform_collect
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\main.py", line 333 in pytest_collection
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\main.py", line 322 in _main
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\main.py", line 269 in wrap_session
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\main.py", line 316 in pytest_cmdline_main
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\config\__init__.py", line 162 in main
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\config\__init__.py", line 185 in console_main
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pytest\__main__.py", line 5 in <module>
File "C:\Users\aprutsko\.conda\envs\unidist\lib\runpy.py", line 87 in _run_code
File "C:\Users\aprutsko\.conda\envs\unidist\lib\runpy.py", line 194 in _run_module_as_main
===================================================================================
We should provide some stuff like setting env vars for unidist
behavior tuning.
List of functions:
ray.cluster_resources()
ray.nodes()
The APIs may be either the same as above or similar and provide all required info.
Testing of actor for multiprocessing
backend on Windows throwed the next errors:
@pytest.mark.parametrize("is_use_options", [True, False])
def test_address_space(is_use_options):
> actor0 = TestActor.options().remote(0) if is_use_options else TestActor.remote(0)
unidist\test\test_actor.py:34:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
unidist\core\base\actor.py:189: in remote
actor_cls = self._actor_cls(*self._actor_cls_args)._remote(*args, **kwargs)
unidist\core\backends\multiprocessing\actor.py:132: in _remote
self._actor_handle = mp.Actor(self._cls, *args, **kwargs)
unidist\core\backends\multiprocessing\core\actor.py:107: in __init__
manager.start()
..\..\.conda\envs\unidist\lib\multiprocessing\managers.py:579: in start
self._process.start()
..\..\.conda\envs\unidist\lib\multiprocessing\process.py:121: in start
self._popen = self._Popen(self)
..\..\.conda\envs\unidist\lib\multiprocessing\context.py:327: in _Popen
return Popen(process_obj)
..\..\.conda\envs\unidist\lib\multiprocessing\popen_spawn_win32.py:93: in __init__
reduction.dump(process_obj, to_child)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
obj = <SpawnProcess name='BaseManager-15' parent=22516 initial>, file = <_io.BufferedWriter name=14>, protocol = None
def dump(obj, file, protocol=None):
'''Replacement for pickle.dump() using ForkingPickler.'''
> ForkingPickler(file, protocol).dump(obj)
E _pickle.PicklingError: Can't pickle <class 'unidist.test.utils.TestActor'>: it's not the same object as unidist.test.utils.TestActor
..\..\.conda\envs\unidist\lib\multiprocessing\reduction.py:60: PicklingError
Current set is present as requirements.txt
, it will be good to use deps from this file in conda env creating and setup.py
without creating separate files to have one place with deps.
We should investigate hotspots for MPI backend on benchmarks (plasticc, cencus, taxi) and try fixing them.
A list of fixed performance issues/PRs:
This package includes MPICH binaries (an implementation of MPI) and does not require installing MPI before mpi4py.
Need to create a way to use all unidist
backends by the same way. Also, this is needed for:
unidist
MPI scriptsThe current default redis password (RayRedisPassword) is incompatible with the default value in Ray. It would be better to use default value from Ray:
from ray import ray_constants
ray_constants.REDIS_DEFAULT_PASSWORD
This makes possible to connect to the default Ray cluster without set `MODIN_REDIS_PASSWORD`
Property Parameter.choices
isn't used. Need to add processing of this property.
This is a subtask of #9.
Main readme.md should be updated and extended to provide more info on unidist
.
Should be similar to the contributing page in Modin - https://modin.readthedocs.io/en/latest/contributing.html.
Might not be limited by this.
Tests in test_general.py
hang in case of using a small number of workers.
unidist/unidist/test/test_general.py
Lines 15 to 24 in bd45958
Possible reason is in incorrect finishing worker-process in multiprocessing
backend.
To reproduce need to set CpuCount = 2
.
List of needed configs:
IsDaskCluster
DaskSchedulerAddress
Update the repo with new logos
As initial step for CI linter checks flake8
, black
and commit-message check should be included.
Issue is a part of #9
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.