Comments (3)
Does the following fix not work: daf0006?
In particular, these lines: daf0006#diff-703512d9cce575fe32a776ec738162312b6276de08ac4846a50f07e3903cfdacR239-R245.
from megatron-lm.
It works. Problem is I've been using Megatron released on Jan without separate bucket for shared embedding. Just switched to latest master and solved it. Thanks!
from megatron-lm.
Awesome, great to hear!
from megatron-lm.
Related Issues (20)
- [ENHANCEMENT]How to specify the number of layers in each pipeline stage in my mind?
- [BUG] Mismatch Between Docstring and Behavior in core.tensor_parallel.random.model_parallel_cuda_manual_seed
- Projeto liliti stk 3.6.9 acabou
- When H800 is trained with FP8, the performance is not significantly improved compared to FP16, and is even worse than FP16. HOT 3
- [QUESTION]Where does the attention_mask come from when the gpt_model is not the first or last pipeline stage?
- [QUESTION] Loss increased by 10 times at second step (after one step of backward). HOT 17
- [QUESTION]Mamba-2-hybrid Weights HOT 4
- [QUESTION] why the _p2p_ops functions has the condition branches for get_pipeline_model_parallel_rank()
- [BUG] the argument of parser.add_argument is wrong in tools/checkpoint/convert.py
- [ENHANCEMENT]Can we pass a tuple that includs all the tensors I'd like to pass between pipeline's different stages?
- [QUESTION] Using segformer segmentation models
- [QUESTION] Question about Mixtral compatibility with Megatron-LM core0.7.0
- [BUG] megatron.training not found HOT 3
- [QUESTION] How to time the code
- [BUG] pipeline_paralle is not available when pp_size > 2
- [BUG] RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead. HOT 1
- [QUESTION]when pretraining bert,meet bug:cuBLAS Error: the requested functionality is not supported
- [QUESTION] Gloo connectFullMesh failed when the number of nodes setting "export GLOO_SOCKET_IFNAME=bond4" exceeds 60
- [QUESTION] OSError: [Errno 28] No space left on device HOT 4
- [QUESTION] --overlap-grad-allreduce failing as gradients coming through as None in param hook HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from megatron-lm.