Comments (9)
感谢你的报告,我们后续会改进这里的体验。我们正在考虑优化一下小模型的切分方案,或者从根本上启动一个维护模型的进程,不用每个task都进行多次加载,释放。
from opencompass.
We recommend using English or English & Chinese for issues so that we could have broader discussion.
from opencompass.
好的 期待你们新的版本
from opencompass.
那再请问一下 我有什么设置可以加速这个模型的解码吗 目前看来八块a100要评测完一个小数据集合 还是会耗费大量的时间
from opencompass.
那再请问一下 我有什么设置可以加速这个模型的解码吗 目前看来八块a100要评测完一个小数据集合 还是会耗费大量的时间
这个一般都是模型本身以及huggingface需要考虑的问题,我们主要专注于评测部分
但是根据我们的经验,原生的llama会快于使用huggingface接口。
from opencompass.
我对huggingface解码不是特别了解 请问现在是八块卡分别加载完整的模型然后分隔数据集解码吗? 是什么导致了模型频繁开辟申请呢 batchsize的设定是否生效 因为我观测到的解码速度和我之前单卡测llama-7b的速度相差很大
from opencompass.
请问现在是八块卡分别加载完整的模型然后分隔数据集解码吗?
是的,
是什么导致了模型频繁开辟申请呢 batchsize的设定是否生效
默认的任务切分参数主要考虑的 100+B模型的推理,就是task的分片太小,导致频繁实例化,加载,推理,释放。具体可参考size 调整一下分片大小。
batchsize 是生效的,但生成慢是因为 1. GPT 这类模型,预测 N 个次,就是要推理N次,生成任务就会慢。2. huggingface 的接口会慢于原生llama,这个需要查huggingface的代码看原因。
也有可能其他的问题导致,如果有条件,可以测试一下不用opencompass纯推理同样问题所需要的开销,进行一下对比。
如果你实验的有任何改进,希望反馈给我们,我们会进一步改进opencompass的用户体验。
from opencompass.
好的感谢
from opencompass.
Feel free to re-open this issue if needed.
from opencompass.
Related Issues (20)
- [Bug] 不支持python3.10以上安装
- [Bug] ValueError: not enough values to unpack HOT 1
- [Feature] Difficulty in Evaluating Custom Models with OpenCompass HOT 1
- meta-llama/Meta-Llama-3-8B-Instruct evaluated results is not consistent with hugging face's official results HOT 2
- [Bug] 增加数据集时失败 HOT 1
- [Feature] Add WildBench HOT 1
- [Bug] 大佬们,这个函数好像写的有问题,只能解析出来[BEGIN]到[DONE]中间的代码,然而基座模型最先输出的代码不是以[BEGIN]开头的。 HOT 1
- [Bug] llama3 8b 基座模型在ARC-C PPL数据集上的评估,accuracy只有41,不正常
- [Feature] Cached Dataset load
- [Bug] Find `scikit-learn` version conflict in `requirements/runtime.txt` and `requirements/extra.txt` HOT 1
- [Feature] 没有找到“subjective”中的compassarena、compassbench、creationbench等数据 HOT 1
- [Feature] 为啥我开始评测一直卡在这里 HOT 1
- [Bug] When I attempted to perform the agent evaluation, the console returned an error: "AttributeError: 'OpenAI' object has no attribute 'chat'". HOT 1
- [Bug] 在eval_qwen_7b这个base模型评估中,为何出现leaderboard.qwen中的gen的数据集,基础模型不应该是ppl的评估方式吗 HOT 3
- 有人配置过mmlu_pro数据集么?求分享代码~
- [Feature] Improve the Documentation for Subjective Evaluation
- [Bug] 按文档使用gpt3.5 测试数据集报错 HOT 1
- [Bug] When evaluation, {prediction} in origin_prompt is not replaced with model's response? HOT 2
- [Bug] qwen1.5-7B base 版本 在math测试集下得分仅有2.6分左右 远低于 官方评测给出的结果
- [Bug] flames的flames-scorer无法正确加载 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from opencompass.