cannot import name 'ShardedDDPOption' from 'transformers.trainer' about transformers HOT 6 OPEN

cc @muellerzr and @SunMarc

from transformers.

Hey @nishitanand, thanks for reporting ! Could you share your traceback ? This shouldn't happen as with your current version of transformers (4.43.4), ShardedDDPOption no longer exist. Mayve try to uninstall transformers then install it again.

from transformers.

Hi, I uninstalled and installed transformers again. I have tried with transformers version 4.44.2 as well. Same error.
I think the problem is that the code uses/requires sharded DDP and I think sharded DDP is removed after transformers v4.34.0 i.e. v4.35.0 onwards. Earlier I used vicunav1.5 and the older version of tranformers worked fine, but I have upgraded Vicuna-v1.5 to Llama 3.1 and Llama 3.1 requires newer version of transformers, which sadly doesn't have sharded DDP.

Here is the traceback:

Traceback (most recent call last):
File "/fs/nexus-scratch/nishit/gamma/encoderfree/EVE/train_mem.py", line 14, in
from eve.train.train import train
File "/fs/nexus-scratch/nishit/gamma/encoderfree/EVE/eve/train/train.py", line 43, in
from eve.train.eve_trainer import EVETrainer
File "/fs/nexus-scratch/nishit/gamma/encoderfree/EVE/eve/train/eve_trainer.py", line 8, in
from transformers.trainer import (ALL_LAYERNORM_LAYERS, ShardedDDPOption,
ImportError: cannot import name 'ShardedDDPOption' from 'transformers.trainer' (/fs/nexus-scratch/nishit/miniconda3/envs/eve_llama3/lib/python3.10/site-packages/transformers/trainer.py)
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 671982) of binary: /fs/nexus-scratch/nishit/miniconda3/envs/eve_llama3/bin/python
Traceback (most recent call last):
File "/fs/nexus-scratch/nishit/miniconda3/envs/eve_llama3/bin/torchrun", line 33, in
sys.exit(load_entry_point('torch==2.0.1', 'console_scripts', 'torchrun')())
File "/fs/nexus-scratch/nishit/miniconda3/envs/eve_llama3/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, **kwargs)
File "/fs/nexus-scratch/nishit/miniconda3/envs/eve_llama3/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/fs/nexus-scratch/nishit/miniconda3/envs/eve_llama3/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/fs/nexus-scratch/nishit/miniconda3/envs/eve_llama3/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/fs/nexus-scratch/nishit/miniconda3/envs/eve_llama3/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
train_mem.py FAILED
Failures:
<NO_OTHER_FAILURES>

from transformers.

That's indeed the case. It looks like the code in their repo needs to be updated to work with the current Trainer. Sorry for the breaking change. Do you know what replaced SharedDDP @muellerzr, so that @nishitanand can fix the trainer subclass ?

from transformers.

Hi @muellerzr, I'd really appreciate it if you could throw some light on the issue. I'm working on a priority project.

from transformers.

Hi @SunMarc, any pointers on how to solve the issue?

from transformers.