Comments (3)
srdfjy
commented on June 11, 2024
I have temporarily solved this issue by changing num_workers=4 and prefetch=250 to num_workers=2 and prefetch=125. However, I'm not sure why setting a higher number of num_workers would lead to this error. It seems that different numbers of GPU cards need to match the appropriate number of num_workers.
from wenet.
Mddct
commented on June 11, 2024
may some oom occurs in training，
num workers * gpus <= cpus coresfrom wenet.
srdfjy
commented on June 11, 2024
may some oom occurs in training，
num workers * gpus <= cpus cores
I am using a total of 4 machines, each equipped with 4 V100 GPUs with 16GB of memory and 100 cores of dedicated CPU.
from wenet.
Related Issues (20)
- Does wenet support saving best model during training? Or having early stopping scheme HOT 1
- Multi-Query Attention failed to export onnx model HOT 2
- arm平台运行onnxruntime报错 HOT 1
- 降噪之后的音频推理准确度下降 HOT 2
- Whisper finetuning support for other languages HOT 1
- Triton Server - support of Unified Conformer model fails HOT 6
- MOE 模型是否可以支持导出onnx HOT 1
- Can paraformer be inferenced with gpu and runtime ?
- NotImplemented: Subclasses of Dataset should imlement_getitem_. HOT 1
- 部署到安卓，错误率很高 HOT 1
- update torch to 2.3.0+cu121, torchaudio fail in func tar_file_and_group of wenet/dataset/datapipes.py HOT 1
- How can I set beam_size in Android runtime?
- 在两人对话的场景中，未来有计划增加区分对话人的功能特性吗
- 使用命令时报错 HOT 1
- 训练的模型后验为空的问题 HOT 1
- 跑paraformer执行转换格式命令时报错
- paraformer模型训练报错 HOT 1
- Does the wenetruntime package still exist? HOT 2
- jit export transducer HOT 2
- Rank 1 failed to pass monitoredBarrier in 1200000 ms HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wenet.