Comments (6)
cc @muellerzr and @SunMarc
from transformers.
Hey @nishitanand, thanks for reporting ! Could you share your traceback ? This shouldn't happen as with your current version of transformers (4.43.4), ShardedDDPOption
no longer exist. Mayve try to uninstall transformers then install it again.
from transformers.
Hi, I uninstalled and installed transformers again. I have tried with transformers version 4.44.2 as well. Same error.
I think the problem is that the code uses/requires sharded DDP and I think sharded DDP is removed after transformers v4.34.0 i.e. v4.35.0 onwards. Earlier I used vicunav1.5 and the older version of tranformers worked fine, but I have upgraded Vicuna-v1.5 to Llama 3.1 and Llama 3.1 requires newer version of transformers, which sadly doesn't have sharded DDP.
Here is the traceback:
Traceback (most recent call last):
File "/fs/nexus-scratch/nishit/gamma/encoderfree/EVE/train_mem.py", line 14, in
from eve.train.train import train
File "/fs/nexus-scratch/nishit/gamma/encoderfree/EVE/eve/train/train.py", line 43, in
from eve.train.eve_trainer import EVETrainer
File "/fs/nexus-scratch/nishit/gamma/encoderfree/EVE/eve/train/eve_trainer.py", line 8, in
from transformers.trainer import (ALL_LAYERNORM_LAYERS, ShardedDDPOption,
ImportError: cannot import name 'ShardedDDPOption' from 'transformers.trainer' (/fs/nexus-scratch/nishit/miniconda3/envs/eve_llama3/lib/python3.10/site-packages/transformers/trainer.py)
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 671982) of binary: /fs/nexus-scratch/nishit/miniconda3/envs/eve_llama3/bin/python
Traceback (most recent call last):
File "/fs/nexus-scratch/nishit/miniconda3/envs/eve_llama3/bin/torchrun", line 33, in
sys.exit(load_entry_point('torch==2.0.1', 'console_scripts', 'torchrun')())
File "/fs/nexus-scratch/nishit/miniconda3/envs/eve_llama3/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, **kwargs)
File "/fs/nexus-scratch/nishit/miniconda3/envs/eve_llama3/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/fs/nexus-scratch/nishit/miniconda3/envs/eve_llama3/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/fs/nexus-scratch/nishit/miniconda3/envs/eve_llama3/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/fs/nexus-scratch/nishit/miniconda3/envs/eve_llama3/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
train_mem.py FAILED
Failures:
<NO_OTHER_FAILURES>
from transformers.
That's indeed the case. It looks like the code in their repo needs to be updated to work with the current Trainer. Sorry for the breaking change. Do you know what replaced SharedDDP
@muellerzr, so that @nishitanand can fix the trainer subclass ?
from transformers.
Hi @muellerzr, I'd really appreciate it if you could throw some light on the issue. I'm working on a priority project.
from transformers.
Hi @SunMarc, any pointers on how to solve the issue?
from transformers.
Related Issues (20)
- I am sure I've made a mistake, but confused about the behavior of ``attention_mask'' of the roformer. HOT 2
- can't resume lora training due to wandb logging num params
- Phi-3's tokenizer `.convert_ids_to_tokens` returns bytes instead of a string HOT 1
- The forced adoption of low_cpu_mem_usage behavior ruins models that require absolute position embedding. HOT 2
- Phi-3 maximum recursion depth exceeded when execute AutoModelForCausalLM.from_pretrained HOT 2
- for LlaVA output of image_hidden_states not possible? HOT 1
- Bug with finetuning Gemma 2 models HOT 3
- DINOV2 generate different outputs for the same inputs HOT 6
- Can't run docker because of flash attention 2? HOT 1
- Training Resumes with Increased Loss Despite Checkpoint Loading HOT 1
- How do LLMs identify generation start point during fine-tuning? HOT 1
- [Nougat] image after NougatProcessor leads to Traceback
- Add "EAT: Self-Supervised Pre-Training with Efficient Audio Transformer"
- How to install transformers==4.45, two or three days I can install successfully, but today cannot. HOT 2
- Accelerate x Trainer issue tracker: HOT 6
- ImportError: attempted relative import with no known parent package HOT 2
- List of Typos in the `transformers[en]` Docs
- bus error on version 4.43.0 with pretrained community CLIP model - MacOS HOT 4
- Can't save quantized models HOT 7
- [Docs] How to build offline HTML or Docset files for other documentation viewers? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from transformers.