baguasys / bagua Goto Github PK

View Code? Open in Web Editor NEW

865.0 16.0 84.0 7.13 MB

Bagua Speeds up PyTorch

Home Page: https://tutorials.baguasys.com/

License: MIT License

Python 64.49% Makefile 0.11% Shell 1.40% Rust 26.83% C 1.13% C++ 3.89% Cuda 2.15%

distributed pytorch rust-lang deep-learning distributed-systems distributed-computing machine-learning bagua

bagua's People

Contributors

Stargazers

Watchers

Forkers

nobles5e tengxu-sun ganshaoduo fossabot jphgxq distributed-deep-learning qoboty snailtolight tonywork fpzh2011 ilibx dd-guo zhihao-chen lycing org-mars myaniu raceli nvsnvyu schragf big-data-ai mr-nineteen jie311 abangdd benmfeng jliu87 kelenlv trendingtechnology gluver huangtiansheng stjordanis allwefantasy fengyindiehun lwl-wss weblfe machinelearningsystem lonsdale8734 yanxing-shi luckyplusten youhe-jiang netmosquito lipovsek vungocastis persiaml vincentwei2021 hmidani-abdelilah lheimbach1 jcgit2018 zhangkg zhjc huabeixiaobai lzy-v gnap woqidaideshi fazziekey arebs23 jerrylead mmathys xiaoqingwang samuelbohl hongshibao pjolles warshon wisdom0530 ymlei harryoung lujiajun boneyag revathyvenugopal162 pmoecke openssl-sg-insights gg-big-org ajunlonglive styler00dollar markhng525 szkang1990 liayan ledinhduy281 gaoxiaojun arpitjain799 ymjs-irfan iq-scm datalayout zubair527 malcolmgreaves

bagua's Issues

consider the case where this is a scalar

bagua/bagua/torch_api/distributed.py

Lines 90 to 95 in 96cb6fe

 ): # TODO: consider the case where this is a scalar 

 broadcast(inner_state, root=0) 

 def _bagua_autotune_step(self): 

 CYCLE_STEP = 100 

 start_time = time.time()

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

@shjwudp add support for reporting tensor completion order

bagua/bagua/torch_api/distributed.py

Lines 124 to 129 in 1bb1ad1

 # TODO: @shjwudp add support for reporting tensor completion order 

 # so that the autotune service does not rely on tensor registration 

 # order 

 rsp = self._bagua_autotune_client.report_metrics( 

 rank=get_rank(), 

 unix_timestamp=time.time(),

This issue was generated by todo based on a `TODO` comment in `1bb1ad1` when #86 was merged. cc @shjwudp.

`baguarun` cannot deal with `--no_python`

but we should support --no_python

@shjwudp merge with service module

bagua/bagua/autotune/__init__.py

Lines 1 to 4 in 96cb6fe

 # TODO: @shjwudp merge with service module 

 import copy 

 import collections 

 import logging

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

@ganshaoduo check if this should be used

bagua/bagua/torch_api/algorithms/onebit_adam.py

Lines 192 to 197 in 96cb6fe

 # weight_decay = group["weight_decay"] # TODO: @ganshaoduo check if this should be used 

 beta1, beta2 = group["betas"] 

 eps = group["eps"] 

 for param_id, param in enumerate(group["params"]): 

 state = self.state[param]

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

@shjwudp add support for reporting tensor completion order so that the autotune ...

bagua/bagua/torch_api/distributed.py

Lines 125 to 130 in 96cb6fe

 # TODO: @shjwudp add support for reporting tensor completion order so that the autotune service does not 

 # rely on tensor registration order 

 from bagua.torch_api.communication import get_bagua_hyperparameters 

 self._bagua_autotune_client.report_metrics( 

 rank=get_rank(),

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

do we need to check whether optimizers and model parameters are the same?

bagua/bagua/torch_api/distributed.py

Lines 182 to 187 in 96cb6fe

 # TODO: do we need to check whether optimizers and model parameters are the same? 

 self.bagua_optimizers = ( 

 optimizers #: the optimizers passed in by ``with_bagua(...)`` 

 ) 

 self.bagua_algorithm = ( 

 algorithm #: the algorithm passed in by ``with_bagua(...)``

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

Support error compensated algorithms

And Cite Bagua section on readme and tutorial

Add complete example code link to https://baguasys.github.io/tutorials/how-to-create-a-new-algorithm/index.html

Use tmpfile to create autotune log to avoid filename collision

Also, we need the option to disable creating a log file.

Some scripts in `tests` are obsolete, and cannot pass mypy

upgrade to `bagua-core` 0.3

@shjwudp add support for reporting tensor completion order

bagua/bagua/torch_api/distributed.py

Lines 124 to 129 in e4ee5ee

 # TODO: @shjwudp add support for reporting tensor completion order 

 # so that the autotune service does not rely on tensor registration 

 # order 

 rsp = self._bagua_autotune_client.report_metrics( 

 rank=get_rank(), 

 unix_timestamp=time.time(),

This issue was generated by todo based on a `TODO` comment in `e4ee5ee`. It's been assigned to @shjwudp because they committed the code.

@shjwudp rename this to is autotune completed

bagua/bagua/torch_api/distributed.py

Lines 296 to 301 in 96cb6fe

 ] # TODO: @shjwudp rename this to is autotune completed 

 recommended_buckets = map( 

 lambda x: list(map(lambda y: self._bagua_tensor_map[y["name"]], x)), 

 response["recommended_hyperparameters"]["buckets"], 

 ) 

 return list(recommended_buckets)

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

python refactor tracking issue

Test script https://github.com/BaguaSys/examples/tree/main/benchmark

autotune service: error handling

All rsp.json() should handle network failure

Implement decentralized algorithm on open source version

broadcast parameters on every algorithm reset

add CONTRIBUTING guidelines

For better collaboration.

remove parameter group logic

bagua/bagua/torch_api/distributed.py

Lines 276 to 281 in 96cb6fe

 ] # TODO: remove parameter group logic 

 ) 

 self._bagua_autotune_client.register_models( # TODO: @shjwudp rename to register tensors 

 autotune_tensor_list, bagua_tensor_group_info 

 ).json() # TODO: @shjwudp error check

With bagua-core 0.3, we no longer need to flatten all at once. Now we can remove parameter groups.

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

@shjwudp

bagua/bagua/service/autotune_service.py

Lines 221 to 224 in 96cb6fe

 req: dict = request.get_json(force=True) # TODO: @shjwudp 

 tensor_ready_order: list = req["tensor_ready_order"] 

 communication_time_ms: float = req["communication_time_ms"] 

 hyperparameters: dict = req["hyperparameters"]

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

remove some log info

remove some log info in bagua.

INTERNAL:EXEMPT_ISSUE_CHECKER

@shjwudp rename to register tensors

bagua/bagua/torch_api/distributed.py

Lines 279 to 284 in 96cb6fe

 self._bagua_autotune_client.register_models( # TODO: @shjwudp rename to register tensors 

 autotune_tensor_list, bagua_tensor_group_info 

 ).json() # TODO: @shjwudp error check 

 def _bagua_autotune_get_buckets(self): 

 response = self._bagua_autotune_client.ask_hyperparameters(

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

fix FusedOptimizer

We need to solve the optimizer OOM when param storage changed

Add benchmark CI to github repo

autotune service: add support for reporting tensor completion order

Currently autotune service rely on the order of tensor registration to do bucketing.

We need to add support for reporting tensor completion order so that the autotune service does not rely on tensor registration order

add 1bit adam doc string

Implement low precision decentralized algorithm in open source version

add pytype CI checking

For better bug discovery.

Fix service development warning

2021-06-15T02:42:53.000561022Z stdout  * Environment: production
2021-06-15T02:42:53.000607172Z stdout    WARNING: This is a development server. Do not use it in a production deployment.
2021-06-15T02:42:53.00061242Z stdout    Use a production WSGI server instead.

autotune service: fix typing

We need to fix some type mismatch in the current implementation:

autotune_system_hyperparameters: Function IntParam.__init__ was called with the wrong arguments [wrong-arg-types]
         Expected: (self, val, space_dimension: Tuple[int, int])
  Actually passed: (self, val, space_dimension: List[int])
File "/github/workspace/bagua/autotune/__init__.py", line 173, in autotune_system_hyperparameters: Function IntParam.__init__ was called with the wrong arguments [wrong-arg-types]
         Expected: (self, val, space_dimension: Tuple[int, int])
  Actually passed: (self, val, space_dimension: List[int])
File "/github/workspace/bagua/autotune/__init__.py", line 180, in autotune_system_hyperparameters: Function IntParam.__init__ was called with the wrong arguments [wrong-arg-types]
         Expected: (self, val, space_dimension: Tuple[int, int])
  Actually passed: (self, val, space_dimension: List[int])
File "/github/workspace/bagua/autotune/__init__.py", line 187, in autotune_system_hyperparameters: Function IntParam.__init__ was called with the wrong arguments [wrong-arg-types]
         Expected: (self, val, space_dimension: Tuple[int, int])
  Actually passed: (self, val, space_dimension: List[int])

Add all communication primitives (such as send recv alltoall) to communication module

Currently there are only broadcast and allreduce: https://bagua.readthedocs.io/en/latest/autoapi/bagua/torch_api/communication/index.html

We need a python test script to test all primitive operations have the same result as torch.distriubuted implementation

@shjwudp do we need global hyperparameters?

bagua/bagua/torch_api/distributed.py

Lines 290 to 295 in 96cb6fe

 get_bagua_hyperparameters().update( # TODO: @shjwudp do we need global hyperparameters? 

 response["recommended_hyperparameters"] 

 ) 

 self._bagua_autotune_completed = not response[ 

 "is_autotune_processing"

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

Add tests and examples for load balance loader and fused optimizer

@shjwudp add support for reporting tensor completion order so that the autotune ...

bagua/bagua/torch_api/distributed.py

Lines 125 to 130 in 6f62a84

 # TODO: @shjwudp add support for reporting tensor completion order so that the autotune service does not 

 # rely on tensor registration order 

 from bagua.torch_api.communication import get_bagua_hyperparameters 

 self._bagua_autotune_client.report_metrics( 

 rank=get_rank(),

This issue was generated by todo based on a `TODO` comment in `6f62a84`. It's been assigned to @NOBLES5E because they committed the code.

Implement OneBitAdam on open source version

autotune service: naming

autotune_client.register_models should be autotune_client.register_tensors
parameter_groups should be tensor_groups
response["is_autotune_processing"] rename this to is_autotune_completed

Fix API doc `communication` module

@shjwudp error check

bagua/bagua/torch_api/distributed.py

Lines 281 to 286 in 96cb6fe

 ).json() # TODO: @shjwudp error check 

 def _bagua_autotune_get_buckets(self): 

 response = self._bagua_autotune_client.ask_hyperparameters( 

 rank=get_rank(), train_iter=self.bagua_train_step_counter 

 ).json()

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

Run large model benchmark

For example some models in https://huggingface.co/transformers/pretrained_models.html

@shjwudp check whether these are still needed

bagua/bagua/torch_api/utils.py

Lines 261 to 264 in 96cb6fe

 # score = np.mean(score_list) # TODO: @shjwudp check whether these are still needed 

 # std = np.std(score_list) 

 return np.mean(score_list), np.std(score_list), score_list.tolist()

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

fix documentation configurations for bagua 0.6

choose the best default message_size

We need experiment to confirm whether we should change message_size to 40M.

Implement SyncBN

autotune: create corresponding introduction and usage on tutorial website

Duplicated doc string

bagua/bagua/torch_api/__init__.py

Lines 346 to 347 in 3a89e12

  * `delay_reduce`(_bool_) - Overlap communication with computation. Default: `True`. 

  * `delay_reduce`(_bool_): Delay all communication to the end of the backward pass. This disables

use elastic launch script as `bagua.distributed.launch.py`

In order to be consistent with PyTorch 1.9 launcher.

check duplicated names

bagua/bagua/torch_api/algorithms/__init__.py

Lines 69 to 74 in 96cb6fe

 ) # TODO: check duplicated names 

 bagua_buckets.append(bagua_bucket) 

 return bagua_buckets 

 def init_forward_pre_hook(self, bagua_module: BaguaModule): 

 """Given a `BaguaModule`, return a hook function that will be executed before the

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

remove

bagua/bagua/torch_api/communication.py

Lines 243 to 246 in 96cb6fe

 # TODO: remove 

 torch.cuda.synchronize()

	): # TODO: consider the case where this is a scalar
	broadcast(inner_state, root=0)

	def _bagua_autotune_step(self):
	CYCLE_STEP = 100
	start_time = time.time()

	# TODO: @shjwudp add support for reporting tensor completion order
	# so that the autotune service does not rely on tensor registration
	# order
	rsp = self._bagua_autotune_client.report_metrics(
	rank=get_rank(),
	unix_timestamp=time.time(),

	# TODO: @shjwudp merge with service module
	import copy
	import collections
	import logging

	# weight_decay = group["weight_decay"] # TODO: @ganshaoduo check if this should be used
	beta1, beta2 = group["betas"]
	eps = group["eps"]

	for param_id, param in enumerate(group["params"]):
	state = self.state[param]

	# TODO: @shjwudp add support for reporting tensor completion order so that the autotune service does not
	# rely on tensor registration order
	from bagua.torch_api.communication import get_bagua_hyperparameters

	self._bagua_autotune_client.report_metrics(
	rank=get_rank(),

	# TODO: do we need to check whether optimizers and model parameters are the same?
	self.bagua_optimizers = (
	optimizers #: the optimizers passed in by ``with_bagua(...)``
	)
	self.bagua_algorithm = (
	algorithm #: the algorithm passed in by ``with_bagua(...)``

	] # TODO: @shjwudp rename this to is autotune completed
	recommended_buckets = map(
	lambda x: list(map(lambda y: self._bagua_tensor_map[y["name"]], x)),
	response["recommended_hyperparameters"]["buckets"],
	)
	return list(recommended_buckets)

	] # TODO: remove parameter group logic
	)

	self._bagua_autotune_client.register_models( # TODO: @shjwudp rename to register tensors
	autotune_tensor_list, bagua_tensor_group_info
	).json() # TODO: @shjwudp error check

	req: dict = request.get_json(force=True) # TODO: @shjwudp
	tensor_ready_order: list = req["tensor_ready_order"]
	communication_time_ms: float = req["communication_time_ms"]
	hyperparameters: dict = req["hyperparameters"]

	get_bagua_hyperparameters().update( # TODO: @shjwudp do we need global hyperparameters?
	response["recommended_hyperparameters"]
	)

	self._bagua_autotune_completed = not response[
	"is_autotune_processing"

	).json() # TODO: @shjwudp error check

	def _bagua_autotune_get_buckets(self):
	response = self._bagua_autotune_client.ask_hyperparameters(
	rank=get_rank(), train_iter=self.bagua_train_step_counter
	).json()

	# score = np.mean(score_list) # TODO: @shjwudp check whether these are still needed
	# std = np.std(score_list)

	return np.mean(score_list), np.std(score_list), score_list.tolist()

	* `delay_reduce`(_bool_) - Overlap communication with computation. Default: `True`.
	* `delay_reduce`(_bool_): Delay all communication to the end of the backward pass. This disables

	) # TODO: check duplicated names
	bagua_buckets.append(bagua_bucket)
	return bagua_buckets

	def init_forward_pre_hook(self, bagua_module: BaguaModule):
	"""Given a `BaguaModule`, return a hook function that will be executed before the

baguasys / bagua Goto Github PK

bagua's People

Contributors

Stargazers

Watchers

Forkers

bagua's Issues

This issue was generated by todo based on a TODO comment in 96cb6fe when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a TODO comment in 1bb1ad1 when #86 was merged. cc @shjwudp.

This issue was generated by todo based on a TODO comment in 96cb6fe when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a TODO comment in 96cb6fe when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a TODO comment in 96cb6fe when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a TODO comment in 96cb6fe when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a TODO comment in e4ee5ee. It's been assigned to @shjwudp because they committed the code.

This issue was generated by todo based on a TODO comment in 96cb6fe when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a TODO comment in 96cb6fe when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a TODO comment in 96cb6fe when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a TODO comment in 96cb6fe when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a TODO comment in 96cb6fe when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a TODO comment in 6f62a84. It's been assigned to @NOBLES5E because they committed the code.

This issue was generated by todo based on a TODO comment in 96cb6fe when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a TODO comment in 96cb6fe when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a TODO comment in 96cb6fe when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a TODO comment in 96cb6fe when #24 was merged. cc @BaguaSys.

Recommend Projects

Recommend Topics

Recommend Org

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a `TODO` comment in `1bb1ad1` when #86 was merged. cc @shjwudp.

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a `TODO` comment in `e4ee5ee`. It's been assigned to @shjwudp because they committed the code.

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a `TODO` comment in `6f62a84`. It's been assigned to @NOBLES5E because they committed the code.

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.

This issue was generated by todo based on a `TODO` comment in `96cb6fe` when #24 was merged. cc @BaguaSys.