lhrlab / chatkbqa Goto Github PK

Your work is really good and has given me a lot of inspiration, but when running the following command, the following situation occurs.

CUDA_VISIBLE_DEVICES=1 nohup python -u eval_final.py --dataset WebQSP --pred_file Reading/LLaMA2-7b/WebQSP_Freebase_NQ_lora_epoch100/evaluation_beam/beam_test_top_k_predictions.json >> predfinal_LLaMA2-7b_WebQSP_Freebase_NQ_lora_epoch100.txt 2>&1 &

I don't know what's wrong, can you give me some suggestions? Thanks

Hello,

I am attempting to reproduce the results of ChatKBQA on the WebQSP dataset, and I have some confusion regarding the metrics used. Specifically, I am trying to determine which of the provided metrics in the repository corresponds to the "F1 Hits@1 Acc 79.8 83.2 73.8" reported in the paper's results.

In the repository, the following metrics are provided for the WebQSP dataset:
total:1639, ex_cnt:1026, ex_rate:0.6259914582062233, real_ex_rate:0.6424546023794615, contains_ex_cnt:1227, contains_ex_rate:0.74862721171446 real_contains_ex_rate:0.7683155917345021

I would appreciate it if you could help me understand which of these metrics corresponds to the "F1 Hits@1 Acc" reported in the paper. This clarification will greatly assist me in accurately reproducing the results.

Thank you for your assistance.

Best regards,

I encountered this error during the training of Baichuan2-7b, and after searching for relevant solutions, I found that upgrading torch to version 2.0 was suggested. However, I am curious if there are any alternative solutions without upgrading torch.

TypeError: sdp_kernel() got an unexpected keyword argument 'enable_mem_efficient'

Traceback (most recent call last):
File "LLMs/LLaMA/src/train_bash.py", line 16, in
main()
File "LLMs/LLaMA/src/train_bash.py", line 7, in main
run_exp()
File "C:\Code\ChatKBQA-main\LLMs\LLaMA\src\llmtuner\tuner\tune.py", line 26, in run_exp
run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
File "C:\Code\ChatKBQA-main\LLMs\LLaMA\src\llmtuner\tuner\sft\workflow.py", line 28, in run_sft
model, tokenizer = load_model_and_tokenizer(model_args, finetuning_args, training_args.do_train, stage="sft")
File "C:\Code\ChatKBQA-main\LLMs\LLaMA\src\llmtuner\tuner\core\loader.py", line 171, in load_model_and_tokenizer
model = AutoModelForCausalLM.from_pretrained(
File "C:\ProgramData\anaconda3\envs\ChatKBQA\lib\site-packages\transformers\models\auto\auto_factory.py", line 556, in from_pretrained
return model_class.from_pretrained(
File "C:\ProgramData\anaconda3\envs\ChatKBQA\lib\site-packages\transformers\modeling_utils.py", line 3375, in from_pretrained
model = cls(config, *model_args, **model_kwargs)
File "C:\Users\DW.cache\huggingface\modules\transformers_modules\chatglm2-6b\modeling_chatglm.py", line 856, in init
self.transformer = ChatGLMModel(config, empty_init=empty_init, device=device)
File "C:\Users\DW.cache\huggingface\modules\transformers_modules\chatglm2-6b\modeling_chatglm.py", line 756, in init
self.encoder = init_method(GLMTransformer, config, **init_kwargs)
File "C:\ProgramData\anaconda3\envs\ChatKBQA\lib\site-packages\torch\nn\utils\init.py", line 52, in skip_init
return module_cls(*args, **kwargs).to_empty(device=final_device)
File "C:\ProgramData\anaconda3\envs\ChatKBQA\lib\site-packages\torch\nn\modules\module.py", line 868, in to_empty
return self._apply(lambda t: torch.empty_like(t, device=device))
File "C:\ProgramData\anaconda3\envs\ChatKBQA\lib\site-packages\torch\nn\modules\module.py", line 641, in _apply
module._apply(fn)
File "C:\ProgramData\anaconda3\envs\ChatKBQA\lib\site-packages\torch\nn\modules\module.py", line 641, in _apply
module._apply(fn)
File "C:\ProgramData\anaconda3\envs\ChatKBQA\lib\site-packages\torch\nn\modules\module.py", line 641, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "C:\ProgramData\anaconda3\envs\ChatKBQA\lib\site-packages\torch\nn\modules\module.py", line 664, in _apply
param_applied = fn(param)
File "C:\ProgramData\anaconda3\envs\ChatKBQA\lib\site-packages\torch\nn\modules\module.py", line 868, in
return self._apply(lambda t: torch.empty_like(t, device=device))
RuntimeError: [enforce fail at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\c10\core\impl\alloc_cpu.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 112197632 bytes.

Hello, my friend
During the training of LLAMA2-13b on an A30 GPU equipped with 24GB of video memory, I am facing an error concerning GPU memory allocation. Are there any feasible solutions or code modifications that can resolve this issue?

error：torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 23.50 GiB total capacity; 23.16 GiB already allocated; 2.81 MiB free; 23.16 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Thanks!

作者您好，我是一名研0小白，想要学习您的代码。在查看train_bash.py时from llmtuner import run_exp提示报错，发现在Github上提供的代码llmtuner包中没有run_exp，在beam_output_eva.py也有类似情况。希望作者大大能够帮我解答疑惑，问题如有冒犯请多多谅解。祝作者大大天天开心，paper多多。

Hi,

Thanks for the outstanding work your team has accomplished.

I have a question: Could you kindly explain what "Oracle Entity Linking Annotations" refer to in your work?

Thank you in advance for your time and assistance.

Hi,
After running python parse_sparql_webqsp.py program, SExpr expression will be generated, but there is null in the reproduced result. Is this reasonable?
Thank you for your assistance.
Best regards.

在跑代码的过程中loss刚开始为300多，然后一下跌到零值并且一直是零，请问是什么问题，数据集是按照百度网盘下载并放置。

Hi,
Thanks for sharing the code, I was trying to run it, and get stack in
python -u LLMs/LLaMA/src/beam_output_eva.py --model_name_or_path meta-llama/Llama-2-7b-hf --dataset_dir LLMs/data --dataset WebQSP_Freebase_NQ_test --template llama2 --finetuning_type lora --checkpoint_dir Reading/LLaMA2-7b/WebQSP_Freebase_NQ_lora_epoch100/checkpoint --num_beams 15

I have finish the 100 epoch training on WebQSP and get following error:
AssertionError: Provided path (Reading/LLaMA2-7b/WebQSP_Freebase_NQ_lora_epoch100/checkpoint) does not contain a LoRA weight.

here is the screen shot for the checkpoint I got after training:

Looking forward to your response.

Best regards,
Xiaqiang

I encountered a bug while running the retrieval code. Can you provide any suggestions on how to resolve this issue?

CUDA_VISIBLE_DEVICES=7 python -u eval_final.py --dataset WebQSP --pred_file Reading/LLaMA2-7b/WebQSP_Freebase_NQ_lora_epoch100/evaluation_beam/beam_test_top_k_predictions.json

INFO:simcse.tool:Use `cls_before_pooler` for unsupervised models. If you want to use other pooling policy, specify `pooler` argument.
split:test, topk_file:Reading/LLaMA2-7b/WebQSP_Freebase_NQ_lora_epoch100/evaluation_beam/beam_test_top_k_predictions.json
Reading/LLaMA2-7b/WebQSP_Freebase_NQ_lora_epoch100/evaluation_beam
INFO:entity_retrieval.surface_index_memory:Loading entity vocabulary from disk.
INFO:entity_retrieval.surface_index_memory:Loading surfaces from disk.
INFO:entity_retrieval.surface_index_memory:Done initializing surface index.
Evaluating test:   0%|                                                                                                                                                                             | 0/1639 [00:00<?, ?it/s]( join ( r [ location , country , languages spoken ] ) [ jamaica ] )
(join (r location.country.languages_spoken) m.03_r3)
(join (r location.country.languages_spoken) m.03_r3)
  0%|                                                                                                                                                                                                 | 0/6 [00:00<?, ?it/s]
Evaluating test:   0%|                                                                                                                                                                             | 0/1639 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "eval_final.py", line 586, in <module>
    aggressive_top_k_eval_new(args.split, args.pred_file, args.dataset)
  File "eval_final.py", line 477, in aggressive_top_k_eval_new
    lf, answers = execute_normed_s_expr_from_label_maps_rel(
  File "eval_final.py", line 288, in execute_normed_s_expr_from_label_maps_rel
    query_expr, denotation = try_relation(d)
  File "eval_final.py", line 310, in try_relation
    in_rels, out_rels, _ = get_2hop_relations_with_odbc_wo_filter(ent)
  File "/data/zihengzhang/derongxu/ideas/ChatKBQA/executor/sparql_executor.py", line 527, in get_2hop_relations_with_odbc_wo_filter
    initialize_odbc_connection()
  File "/data/zihengzhang/derongxu/ideas/ChatKBQA/executor/sparql_executor.py", line 27, in initialize_odbc_connection
    odbc_conn = pyodbc.connect(
pyodbc.InterfaceError: ('IM002', '[IM002] [unixODBC][Driver Manager]Data source name not found, and no default driver specified (0) (SQLDriverConnect)')

Hi,
What are the hyper-parameters of rank and alpha?

Thanks for sharing the codes! Could you please upload processed files for training and evaluation (e.g., WebQSP/generation/* and CWQ/generation/*), because I found that label map is empty using my Freebase deployed before. Thanks a lot!

您好！请问一下这个项目可以用在新的知识图谱上吗？微调大模型生成的SExpr怎么转换为其它图谱的Sparql？

训练出来的模型，可以直接在任何新的图谱上使用吗？ 还是说这种方法，必须在新的图谱上进行finetune后，才能使用。

Hi, thank for your great job.

Do you know how to transform a s-expr / sparql_query to a path from source entity to target entity? Rather than only get the final answer with execute_query_with_odbc(sparql_query)

I'm not familirary about the SPARQL system and the usages of pyodbc.

Any assistance you could provide would be greatly appreciated.