huggingface数据集离线加载 about opencompass HOT 4 CLOSED

First, manually download all the huggingface dependency to where it should be (e.g. ~/.cache/huggingface). Then you may set the following env variables and start OpenCompass scripts.

export HF_DATASETS_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export HF_EVALUATE_OFFLINE=1

from opencompass.

To my knowledge, once load_dataset is executed, the dataset has already been cached to the disk. There is no need to save_to_disk again. Still, you can download the files in one machine and copy them to another machine in any way possible (e.g. in-cluster networks / USB flask disk / mount disk). And that is all I know.
I am not sure whether this post would be helpful to you.

from opencompass.

First, manually download all the huggingface dependency to where it should be (e.g. ~/.cache/huggingface). Then you may set the following env variables and start OpenCompass scripts.

export HF_DATASETS_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export HF_EVALUATE_OFFLINE=1

手工下载huggingface的数据集的方式有哪些，目前我知道的一种方法是：采用能联网的电脑，使用load_dataset 函数，从网络上load，然后再save_to_disk(my_dataset_dir)，磁盘上应该就会有了；
还有其他方式吗？感谢~

from opencompass.

To my knowledge, once load_dataset is executed, the dataset has already been cached to the disk. There is no need to save_to_disk again. Still, you can download the files in one machine and copy them to another machine in any way possible (e.g. in-cluster networks / USB flask disk / mount disk). And that is all I know. I am not sure whether this post would be helpful to you.

赞，多谢

from opencompass.