git-cloner / llama2-chinese Goto Github PK

Training Epoch0: 0it [00:00, ?it/s]
Training Epoch0: 0it [00:00, ?it/s]
Traceback (most recent call last):
  File "/data-ai/usr/weiling/Llama2-chinese/llama_finetuning.py", line 238, in <module>
    fire.Fire(main)
  File "/opt/conda/envs/llama-recipes/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/conda/envs/llama-recipes/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/opt/conda/envs/llama-recipes/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/data-ai/usr/weiling/Llama2-chinese/llama_finetuning.py", line 221, in main
    results = train(
  File "/data-ai/usr/weiling/Llama2-chinese/utils/train_utils.py", line 112, in train
    train_epoch_loss = total_loss / data_set_len
ZeroDivisionError: float division by zero
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 12944) of binary: /opt/conda/envs/llama-recipes/bin/python3.9
Traceback (most recent call last):
  File "/opt/conda/envs/llama-recipes/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/envs/llama-recipes/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/opt/conda/envs/llama-recipes/lib/python3.9/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/opt/conda/envs/llama-recipes/lib/python3.9/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/opt/conda/envs/llama-recipes/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/opt/conda/envs/llama-recipes/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: