Comments (9)
Thanks for your watch on our project. We now support deepspeed ZeRO-3 when using BAdam, please see our detailed instruction: https://github.com/Ledzy/BAdam?tab=readme-ov-file#partition-by-module-model-parallel.
from badam.
Could you provide a bit more detail, for example on how to use BAdamCallback? The examples in the Readme are not complete, are they?
from badam.
Could you provide a bit more detail, for example on how to use BAdamCallback? The examples in the Readme are not complete, are they?
Hi @paulcx , please follow the provided example:
from badam.utils import BAdamCallback
callbacks = original_callbacks.append(BAdamCallback) # add the callback
trainer = YourTrainerClass(
...,
callbacks=callbacks
)
If there is no original_callbacks
, just pass callbacks=[BAdamCallback]
when initializing your Trainer.
from badam.
Could you provide a bit more detail, for example on how to use BAdamCallback? The examples in the Readme are not complete, are they?
Hi @paulcx , please follow the provided example:
from badam.utils import BAdamCallback callbacks = original_callbacks.append(BAdamCallback) # add the callback trainer = YourTrainerClass( ..., callbacks=callbacks )If there is no
original_callbacks
, just passcallbacks=[BAdamCallback]
when initializing your Trainer.
My understanding is that BAdam doesn’t need to be initialized? How about the zero3 config. Is anything need to be done except these codes above?
from badam.
To use ZeRO-3, you should create a configuration file which specifies ZeRO stage and other configurations; see https://github.com/Ledzy/BAdam/blob/master/sample_ds_zero3.json for an example. You script should be launched using deepspeed
or accelerate launch
or other distributed launching command, depending on your circumstance.
The aforementioned part is the setup of Deepspeed and is decouple with the optimizers, e.g. BAdam. Using BAdam under ZeRO-3 is almost the same as using it in single GPU mode, except that one needs to
- pass
ds_zero3_enabled=True
when creatingBlockOptimizer
, and - create a reference to deepspeed optimizer, which can be achieved by
- approach 1: simply set
block_optim.ds_optimizer = ds_optimizer
afterdeepspeed.initialize
, or - approach 2: add an additional
BAdamCallback
when instancing your Trainer, which actually does the same thing as approach 1; this is cleaner when using huggingface Trainer.
Hope my response is helpful for you. Feel free to reach us if you have any other questions.
from badam.
In approach 2, step 1 (pass ds_zero3_enabled=True when creating BlockOptimizer) is not required?
from badam.
It's required for the current version.
from badam.
In deepspeed+huggingface Trainer approach, the optimizer parameters are saved in ds_config.json. For example,
"optimizer": { "type": "AdamW", "params": { "lr": "auto", "betas": [ 0.9, 0.999 ], "eps": 1e-8, "weight_decay": "auto" }
Is an example of BlockOptimizer?
from badam.
BlockOptimizer inherit configurations in ds_config.json
from badam.
Related Issues (11)
- 支持多卡吗? HOT 3
- RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn HOT 3
- What is the difference from existing HiFT "A Hierarchical Full Parameter Fine-Tuning Strategy"? HOT 7
- type object 'PPODecorators' has no attribute 'empty_cuda_cache'. Did you mean: 'empty_device_cache'? HOT 11
- How to realize the loss curves of badam and lora in the paper HOT 3
- Error when using BlockOptimizerRatio HOT 4
- Floating point exception when running sft with lora HOT 3
- 同样数据,BAdam速度比Lora慢很多 HOT 9
- badam 预训练时间太长 比全参数预训练都长
- Can BlockOptimizer switch learning rates? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from badam.