I've tried a single and two gpu run. I have observed training acceleration for two gpu

maybe you can set num_workers to <code class="notrans

Does single node muti gpu accelerate training? about freevc HOT 4 CLOSED

olawod commented on July 23, 2024

Does single node muti gpu accelerate training?

from freevc.

Comments (4)

OlaWod commented on July 23, 2024 1

tried 2 gpu on our machine and did not encounter this error. I guess it might be caused by lack of system resources (e.g. number of processes or threads).

from freevc.

OlaWod commented on July 23, 2024

yes theoretically multi gpu can accelerate training

from freevc.

skol101 commented on July 23, 2024

I see.

When training with multi GPU setup I get the error during eval step, but the training continues

INFO:freevc:[2.674854040145874, 2.1956264972686768, 4.762864112854004, 17.808719635009766, 1.9186915159225464, 40000, 0.00019515923892096098]
terminate called without an active exception
Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x7f71bd0233a0>
Traceback (most recent call last):
  File "/home/sk/anaconda3/envs/freevc/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1466, in __del__
    self._shutdown_workers()
  File "/home/sk/anaconda3/envs/freevc/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1430, in _shutdown_workers
    w.join(timeout=_utils.MP_STATUS_CHECK_INTERVAL)
  File "/home/sk/anaconda3/envs/freevc/lib/python3.8/multiprocessing/process.py", line 149, in join
    res = self._popen.wait(timeout)
  File "/home/sk/anaconda3/envs/freevc/lib/python3.8/multiprocessing/popen_fork.py", line 44, in wait
    if not wait([self.sentinel], timeout):
  File "/home/sk/anaconda3/envs/freevc/lib/python3.8/multiprocessing/connection.py", line 931, in wait
    ready = selector.select(timeout)
  File "/home/sk/anaconda3/envs/freevc/lib/python3.8/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
  File "/home/sk/anaconda3/envs/freevc/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 377049) is killed by signal: Aborted. 
INFO:freevc:Saving model and optimizer state at iteration 197 to ./logs/freevc/G_40000.pth
INFO:freevc:Saving model and optimizer state at iteration 197 to ./logs/freevc/D_40000.pth

from freevc.