Comments (2)
Additionally, the total size only takes into account the first n-1
shards.
from transformers.
Thanks for the report @xenova ! The easiest solution would be to update the get_tensor_size
function in huggingface_hub library as it doesn't "work" with meta
tensor:
def get_tensor_size(tensor: "torch.Tensor") -> int:
return tensor.numel() * tensor.element_size()
In accelerate, we have the following for example:
def id_tensor_storage(tensor: torch.Tensor) -> Tuple[torch.device, int, int]:
"""
Unique identifier to a tensor storage. Multiple different tensors can share the same underlying storage. For
example, "meta" tensors all share the same storage, and thus their identifier will all be equal. This identifier is
guaranteed to be unique and constant for this tensor's storage during its lifetime. Two tensor storages with
non-overlapping lifetimes may have the same id.
"""
_SIZE = {
torch.int64: 8,
torch.float32: 4,
torch.int32: 4,
torch.bfloat16: 2,
torch.float16: 2,
torch.int16: 2,
torch.uint8: 1,
torch.int8: 1,
torch.bool: 1,
torch.float64: 8,
}
try:
storage_ptr = tensor.untyped_storage().data_ptr()
storage_size = tensor.untyped_storage().nbytes()
except Exception:
# Fallback for torch==1.10
try:
storage_ptr = tensor.storage().data_ptr()
storage_size = tensor.storage().size() * _SIZE[tensor.dtype]
except NotImplementedError:
# Fallback for meta storage
storage_ptr = 0
# On torch >=2.0 this is the tensor size
storage_size = tensor.nelement() * _SIZE[tensor.dtype]
return tensor.device, storage_ptr, storage_size
This way, we will have the state dict properly splitted with the right tensor size. Note that the state_dict will contain meta tensors. But, we update the state dict afterwards using get_state_dict_from_offload
(we can't do that before as the might not have enough storage on gpus+cpu because some layers are stored in the disk). LMK if this works for you @Wauplin !
from transformers.
Related Issues (20)
- How to use hugginface for training: google-t5/t5-base
- The llava processor does not appear to support batch process. HOT 6
- [Falcon Mamba] Unexpected model output with use_cache=False and model.train() HOT 5
- 'CohereModel' object has no attribute '_prune_heads' HOT 3
- When will Transformers version 4.45.0 be released? HOT 1
- RobertaTokenizer has incorrect character offsets HOT 3
- M2M100Tokenizer vocabulary size is not equal to the m2m embedding_size for the "facebook/m2m100_418M" model. HOT 1
- when l use the model generate method within @tf.function, it encounterd a mistake HOT 1
- cannot import name 'ShardedDDPOption' from 'transformers.trainer' HOT 6
- Supprot for qwen2moe gguf models HOT 4
- ValueError: Cannot use apply_chat_template() because tokenizer.chat_template is not set HOT 4
- DETR documentation typo: repeated word `be be`
- Community contribution: Adding GGUF support for more architectures HOT 7
- LlavaNextProcessor bug in `_get_unpadded_features` HOT 1
- latest 44.4.2 doesn't support falcon_mamba HOT 1
- Add Log-Softmax Temperature Option to transformers.Seq2SeqTrainingArguments for CE Loss HOT 1
- prepare_fa2_from_position_ids error in training with batch_size > 1
- Static KV cache status: How to use it? Does it work for all models? HOT 2
- Custom pipeline in remote repo cannot load custom model from remote repo. HOT 5
- Qwen2-VL from_config broken? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from transformers.