Multiple GPU training and changing to librosa mel spec? about auxiliaryasr HOT 2 CLOSED

from logging import StreamHandler
import colossalai

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
handler = StreamHandler()
handler.setLevel(logging.DEBUG)
logger.addHandler(handler)

torch.backends.cudnn.benchmark = True

@click.command()
@click.option('-p', '--config_path', default='Configs/config.yml', type=str)
def main(config_path):
    colossalai.launch_from_torch(config="Configs/config.py")
    config = yaml.safe_load(open(config_path))
    log_dir = config['log_dir']
    if not osp.exists(log_dir): os.mkdir(log_dir)
    shutil.copy(config_path, osp.join(log_dir, osp.basename(config_path)))
    ...

from colossalai.amp import AMP_TYPE

fp16 = dict(
  mode=AMP_TYPE.TORCH
  # below are default values for grad scaler
)

parallel = dict(
    tensor=dict(size=2, mode='1d')
)

gradient_accumulation = 4
clip_grad_norm = 1.0

rank=0
world_size=1
host="localhost"
port=29500

/bin/bash: /opt/conda/lib/libtinfo.so.6: no version information available (required by /bin/bash)
[02/22/23 04:27:13] INFO     colossalai - colossalai - INFO:                    
                             /opt/conda/lib/python3.7/site-packages/colossalai/c
                             ontext/parallel_context.py:521 set_device          
[02/22/23 04:27:13] INFO     colossalai - colossalai - INFO:                    
                             /opt/conda/lib/python3.7/site-packages/colossalai/c
                             ontext/parallel_context.py:521 set_device          
                    INFO     colossalai - colossalai - INFO: process rank 1 is  
                             bound to device 1                                  
                    INFO     colossalai - colossalai - INFO: process rank 0 is  
                             bound to device 0                                  
[02/22/23 04:27:16] INFO     colossalai - colossalai - INFO:                    
                             /opt/conda/lib/python3.7/site-packages/colossalai/c
                             ontext/parallel_context.py:557 set_seed            
[02/22/23 04:27:16] INFO     colossalai - colossalai - INFO:                    
                             /opt/conda/lib/python3.7/site-packages/colossalai/c
                             ontext/parallel_context.py:557 set_seed            
                    INFO     colossalai - colossalai - INFO: initialized seed on
                             rank 1, numpy: 1024, python random: 1024,          
                             ParallelMode.DATA: 1024, ParallelMode.TENSOR:      
                             1025,the default parallel seed is                  
                             ParallelMode.DATA.                                 
                    INFO     colossalai - colossalai - INFO: initialized seed on
                             rank 0, numpy: 1024, python random: 1024,          
                             ParallelMode.DATA: 1024, ParallelMode.TENSOR:      
                             1024,the default parallel seed is                  
                             ParallelMode.DATA.                                 
                    INFO     colossalai - colossalai - INFO:                    
                             /opt/conda/lib/python3.7/site-packages/colossalai/i
                             nitialize.py:120 launch                            
                    INFO     colossalai - colossalai - INFO: Distributed        
                             environment is initialized, data parallel size: 1, 
                             pipeline parallel size: 1, tensor parallel size: 2 
/bin/bash: /opt/conda/lib/libtinfo.so.6: no version information available (required by /bin/bash)
[02/22/23 04:27:13] INFO     colossalai - colossalai - INFO:                    
                             /opt/conda/lib/python3.7/site-packages/colossalai/c
                             ontext/parallel_context.py:521 set_device          
[02/22/23 04:27:13] INFO     colossalai - colossalai - INFO:                    
                             /opt/conda/lib/python3.7/site-packages/colossalai/c
                             ontext/parallel_context.py:521 set_device          
                    INFO     colossalai - colossalai - INFO: process rank 1 is  
                             bound to device 1                                  
                    INFO     colossalai - colossalai - INFO: process rank 0 is  
                             bound to device 0                                  
[02/22/23 04:27:16] INFO     colossalai - colossalai - INFO:                    
                             /opt/conda/lib/python3.7/site-packages/colossalai/c
                             ontext/parallel_context.py:557 set_seed            
[02/22/23 04:27:16] INFO     colossalai - colossalai - INFO:                    
                             /opt/conda/lib/python3.7/site-packages/colossalai/c
                             ontext/parallel_context.py:557 set_seed            
                    INFO     colossalai - colossalai - INFO: initialized seed on
                             rank 1, numpy: 1024, python random: 1024,          
                             ParallelMode.DATA: 1024, ParallelMode.TENSOR:      
                             1025,the default parallel seed is                  
                             ParallelMode.DATA.                                 
                    INFO     colossalai - colossalai - INFO: initialized seed on
                             rank 0, numpy: 1024, python random: 1024,          
                             ParallelMode.DATA: 1024, ParallelMode.TENSOR:      
                             1024,the default parallel seed is                  
                             ParallelMode.DATA.                                 
                    INFO     colossalai - colossalai - INFO:                    
                             /opt/conda/lib/python3.7/site-packages/colossalai/i
                             nitialize.py:120 launch                            
                    INFO     colossalai - colossalai - INFO: Distributed        
                             environment is initialized, data parallel size: 1, 
                             pipeline parallel size: 1, tensor parallel size: 2 
/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py:560: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  cpuset_checked))
/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py:560: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  cpuset_checked))
{'max_lr': 0.0001, 'pct_start': 0.0, 'epochs': 200, 'steps_per_epoch': 15502}
[train]:   0%|                                        | 0/15502 [00:00<?, ?it/s]{'max_lr': 0.0001, 'pct_start': 0.0, 'epochs': 200, 'steps_per_epoch': 15502}
[train]:   0%|                                        | 0/15502 [00:00<?, ?it/s]'
'
'
'
''

'
'
'
'
'
'
'
'
'
'
/kaggle/AuxiliaryASR/AuxiliaryASR/trainer.py:158: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  mel_input_length = mel_input_length // (2 ** self.model.n_down)
/kaggle/AuxiliaryASR/AuxiliaryASR/trainer.py:158: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  mel_input_length = mel_input_length // (2 ** self.model.n_down)
'
'
'
'
'
'
[train]:   0%|                             | 1/15502 [00:13<56:25:46, 13.11s/it]'
'
'
'
'
'
''

[train]:   0%|                             | 4/15502 [00:18<14:02:20,  3.26s/it]'
'
[train]:   0%|                             | 4/15502 [00:18<14:09:01,  3.29s/it]'
'
[train]:   0%|                             | 5/15502 [00:20<12:39:14,  2.94s/it]'
'
'
'
'
'