Comments (15)
@XWwwwww You are right. I run the interactive scripts on a single GPU cloud machine, and eva_finetune script on a multi-gpu cloud machine in distributed mode. Both with the docker provided. The interactive script works well to build the utils.so, but finetune script fails with ninja: fatal: waitpid(113): No child processes.
Anyway, I just tried to copy my utils.so generated by the interactive script to my multi-gpu machine, and it works well now. Thanks!
from eva.
what's the version of pytorch in your environment
from eva.
what's the version of pytorch in your environment
@XWwwwwwprint(torch.version)
1.10.1+cu102
from eva.
Thanks for reporting this. We will check the environment.
from eva.
I encountered the same error. Here's my environment
torch 1.10.1 + cu111 + gcc 9.3
from eva.
Hi. I upgrade the torch to 1.10.2, and cuda to cu113. That solves the problem.
from eva.
Hi. I upgrade the torch to 1.10.2, and cuda to cu113. That solves the problem.
@Hermes777 Hi were you able to get the path /torch_extensions/py38_cu102/utils/utils.so after you upgrade the torch and cuda?
from eva.
From the term "py38_cu102", you are obviously using cuda 102, which might not compatible.
from eva.
I checked the directory utils/, it contains utils.so
Never forget to install the apex once again, after you upgrade the torch and cuda.
from eva.
I checked the directory utils/, it contains utils.so
Never forget to install the apex once again, after you upgrade the torch and cuda.
Thanks so much. I will try later
from eva.
@t1101675 @XWwwwww Sorry, I'm still having this issue. I don't think it is the problem of cuda version, as I'm using cuda10.2 which is suggested in your readme. I'm running with the docker provided and I was able to run the interactive scripts. But the eva_finetune still gives me the aforementioned error. I checked the log and spotted that while building the extension module utils, there was
Building extension module utils...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: fatal: waitpid(113): No child processes
So the building of utils might fail? Please help and give me some advices, thanks!
from eva.
@Vincentwei1021 We upload the missing file in src/ds_fix/utils.so
. Let me know if it works.
from eva.
@XWwwwww Hi thanks for your reply. I have tried to use the file, and it gives the following error:
ImportError: /.cache/torch_extensions/py38_cu102/utils/utils.so: undefined symbol: _ZNK2at6Tensor6narrowElll
from eva.
@t1101675 @XWwwwww Sorry, I'm still having this issue. I don't think it is the problem of cuda version, as I'm using cuda10.2 which is suggested in your readme. I'm running with the docker provided and I was able to run the interactive scripts. But the eva_finetune still gives me the aforementioned error. I checked the log and spotted that while building the extension module utils, there was
Building extension module utils... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: fatal: waitpid(113): No child processes
So the building of utils might fail? Please help and give me some advices, thanks!
I just notice that you have already been able to run the interactive scripts, which means the utils.so
file has already been automatically compiled and saved in your cache path, right?
And the current error message is ninja: fatal: waitpid(113): No child processes
instead of cache/torch_extensions/py38_cu102/utils/utils.so: cannot open shared object file: No such file or directory
?
from eva.
spotted
@XWwwwww You are right. I run the interactive scripts on a single GPU cloud machine, and eva_finetune script on a multi-gpu cloud machine in distributed mode. Both with the docker provided. The interactive script works well to build the utils.so, but finetune script fails with ninja: fatal: waitpid(113): No child processes.
Anyway, I just tried to copy my utils.so generated by the interactive script to my multi-gpu machine, and it works well now. Thanks!
i met this situation :
Building extension module utils...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: fatal: waitpid(97011): No child processes
and occured this error:
....
File "/home/zhangwenjuan/anaconda3/envs/evacuda11/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1079, in load
return _jit_compile(
File "/home/zhangwenjuan/anaconda3/envs/evacuda11/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1317, in _jit_compile
return _import_module_from_library(name, build_directory, is_python_module)
File "/home/zhangwenjuan/anaconda3/envs/evacuda11/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1699, in _import_module_from_library
file, path, description = imp.find_module(module_name, [path])
File "/home/zhangwenjuan/anaconda3/envs/evacuda11/lib/python3.8/imp.py", line 296, in find_module
raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'utils'
could you give me some suggestions? thank you!
from eva.
Related Issues (20)
- torch multiprocessing api failed HOT 6
- 关于huggingface的finetune前的结果出入 HOT 15
- 请问这些输出是哪里代码打印的? HOT 1
- 请问,我在运行eva_finetune的时候出现了ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 58058) of binary: HOT 6
- finetune的输入长度 HOT 1
- 这个效果是认真的吗?is this result true? HOT 1
- example.py 那里的文件路径放的都是bin的模型吗? HOT 2
- 请问role embeds参数的初始化问题 HOT 6
- error occurred when i run eva_finetune.py HOT 4
- 我停止了思考 HOT 7
- huggingface的 tokenizer报错 HOT 11
- 微调错误RuntimeError: apex.optimizers.FusedAdam requires cuda extensions HOT 4
- huggingface分支代码运行报错问题及解决方法
- 运行交互式评测报错 HOT 1
- 微调报CUDA out of memory错误 HOT 6
- RuntimeError: a leaf Variable... HOT 2
- EVA2.0-xLarge 模型? HOT 2
- 确定现在给出的这个xlarge模型没有问题吗?为什么效果这么差呀,回答都混乱了 HOT 8
- EVA3 HOT 1
- 关于数据过滤 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from eva.