pykt-team / pykt-toolkit Goto Github PK
View Code? Open in Web Editor NEWpyKT: A Python Library to Benchmark Deep Learning based Knowledge Tracing Models
Home Page: https://pykt.org
License: MIT License
pyKT: A Python Library to Benchmark Deep Learning based Knowledge Tracing Models
Home Page: https://pykt.org
License: MIT License
Hello, I encountered this error while running the code. How can I resolve it? Looking forward to your reply very much
this issue is for testing
您好,我在用assist2015数据集跑akt、atdkt、simplekt和sparsekt等模型时都报了下述图片上的错误,求助如何
需要在具体代码后加to(device)。
The assistment2017 dataset (anonymized_full_release_competition_dataset.csv) does not include concepts. Why does the data set generated after "data_preprocess.py" have concepts?
求助大佬~
我是把整个包以zip的方式下载然后作为project在pycharm中打开。我按照pykt的quick start中的提示,先将路径定位到我电脑中这个包里面examples文件夹的位置,然后运行python data_preprocess.py --dataset_name=assist2009, 就会出现下述报错:
请问我要跑其他公共数据是都需要自行先把相关的数据集下载好吗~
我尝试从网上下载提到的这份assist2009的数据,但是又出现的另一个错:
我从网上找上述错误的解决方法,我将utf-8 改成 gbk 又报了其他的错:
我看网上解决方案都是加个float,但是加了这个又会报错,然后有评论说编码转成utf-8.
已经彻底晕了,求助热心的大佬!!!
Hi, can I use the multi-skill exercises of myself dataset?
For example:i have a exercises list
whether support my dataset from other source(sush as :codeforces.com,luogu.com.cn)?
Thank you very much for the great contribution of pykt library in the field of knowledge tracing. In the reproduction process, the default value of the parameter "emb_type" is "qid", I would like to know what are the values of the parameter "emb_type"?
Thank you for providing this useful collection of models! DKVMN seems to perform best on my dataset. Yet, I am having trouble to display the knowledge states as described in their paper. Are you aware of any source where the code for creating knowledge states for DKVMN or SKVMN are available?
Thank you very much in advance!
Hi, I'm following the quick start guide, it is unclear how you are meant to access python data_preprocess.py [parameter]
from the previous instructions on the page.
It would be great if this could be clarified
请问一下大佬,我们选用数据时候可以用验证集AUC吗 Epoch: 8, validauc: 0.7784, validacc: 0.7796, best epoch: 8, best auc: 0.7784, train loss: 10.460885047912598, emb_type: qid, model: iekt, save_dir: 还是必须用test呢?
请问dkt_forget模型中,num_rgap、num_sgap、num_pcount三个参数代表什么呢?
How to get other data sets besides ASSIST?
Similarly to my previous issue, it is not clear how I should lay out the files after downloading the various parts of the ednet dataset.
I currently have:
data/ednet/contents
data/ednet/KT1
data/ednet/KT2
data/ednet/KT3
data/ednet/KT4
I am getting a "ValueError, No objects to concatenate" when I run "python data_preprocess.py -d ednet", and am not enjoying reverse engineering the data processing scripts.
Thank you for sharing a masterpiece framework.
I ran various models on the nips_task34 dataset, and the results were well reproduced.
However, on the assistment2009 dataset, I get about 5% better performance on average than reported in the pyKT and QIKT papers, etc. (e.g., DKT and SparseKT-topk achieve about 0.82 and 0.83 val auc, respectively.)
Is there something I'm missing?
您好!
非常感谢您详实的工作,我相信这将对知识追踪领域产生长远的影响。
请问是否有计划支持 multi GPU training,这对更大的batch和更复杂的网络实现而言是必要的。
Hello, you've done a great job in knowledge tracking, but I have some questions about your open-source code. I noticed that there are two files in your code for evaluating models, eval.py and predict.py. As far as I understand, predict.py might not be using the fusion method, which means it could potentially lead to label leakage. Is my understanding correct, or could you please explain the difference between these two files?
FileNotFoundError: [Errno 2] No such file or directory: '../data/ednet\train_valid_sequences.csv'
when running wandb_saint_train.py --dataset_name=ednet
I have already ran python data_preprocess.py --dataset_name=ednet
Hello, sorry to bother you again. May I ask if all models have a training Epoch of 200?
首先感谢项目组的开源,目前我在调试qikt模型时出现些问题,不知道是不是有些包的版本不一样导致的,所以希望项目组可以考虑出一个基础环境安装的包
2023-09-19 00:37:50 - main - said: init_model
model_name:sparsekt
The wrong model name was used...
model is None
Traceback (most recent call last):
File "wandb_sparsekt_train.py", line 39, in
main(params)
File "/root/pykt-toolkit-main/examples/wandb_train.py", line 123, in main
opt = Adam(model.parameters(), learning_rate)
AttributeError: 'NoneType' object has no attribute 'parameters'
def train_one_step(self,data,process=True,return_all=False):
outputs,data_new = self.predict_one_step(data,return_details=True,process=process)
# all
loss_q_all = self.get_loss(outputs['y_question_all'],data_new['rshft'],data_new['sm'])
loss_c_all = self.get_loss(outputs['y_concept_all'],data_new['rshft'],data_new['sm'])
作者你好,我看outputs中的y_question_all和y_concept_all好像是t时刻的预测值,而rshft是t+1时刻的真实值。 为什么要用这两个值做损失?请问是我哪里理解错误了吗
The que_dkt_demo file of the example folder had import the DKTQue method, but I don't find the relation file.
The raw code was " from pykt.models.dkt_que import DKTQue "
I modified the Saint model structure in the saint.py file. However, when I call the same command 'python wandb_saint_train.py --dataset_name=assist2009 --seed=3407 --dropout=0.3 --use_wandb=0 --add_uuid=0' before and after the modification, the accuracy and AUC remain exactly the same. Why is that?
很优秀的工作!谢谢您的贡献,我想知道如果我以问题未输入的话,代码中是有一个参数的设置还是可以直接改输入的呢?例如:c直接改成cq??
感谢pykt为知识追踪领域做出的巨大贡献。
在阅读论文后,关于如何复现Question Level和KC Level仍存有疑问。我运行example中wandb_xxxx_train.py再运行wandb_predict.py后,所示结果包含One-by-One的结果等。请问其他结果中对应的是Question Level还是KC Level的呢?
{'testauc': 0.9188002221943278, 'testacc': 0.8678735316393662, 'window_testauc': 0.9201199409709553, 'window_testacc': 0.868663798838086, 'oriaucconcepts': 0.807274877304899, 'oriauclate_mean': 0.807413805143088, 'oriauclate_vote': 0.8049668720325175, 'oriauclate_all': 0.8058672455891169, 'oriaccconcepts': 0.7851768050790059, 'oriacclate_mean': 0.807144009898327, 'oriacclate_vote': 0.8066383344988972, 'oriacclate_all': 0.8018075205766851, 'windowaucconcepts': 0.8093884777107377, 'windowauclate_mean': 0.8101170671331457, 'windowauclate_vote': 0.8078638216548034, 'windowauclate_all': 0.8085259538001138, 'windowaccconcepts': 0.7864040341303994, 'windowacclate_mean': 0.808367475743734, 'windowacclate_vote': 0.8078753971395257, 'windowacclate_all': 0.8028583348487928}
期待您的回答,感谢
我有个问题 就是我用pykt库里面的 iekt模型跑 然后auc是0.8318。然后我需要atdkt模型数据集里面的 历史正确性这个数据。就把iekt的数据loader换成了atdkt的loader 数据集不变.然后啥也没改 iekt的auc就到了0.923。
修改代码如下:
# curvalid = KTQueDataset(os.path.join(data_config["dpath"], data_config["train_valid_file_quelevel"]),
# input_type=data_config["input_type"], folds={i},
# concept_num=data_config['num_c'], max_concepts=data_config['max_concepts'])
# curtrain = KTQueDataset(os.path.join(data_config["dpath"], data_config["train_valid_file_quelevel"]),
# input_type=data_config["input_type"], folds=all_folds - {i},
# concept_num=data_config['num_c'], max_concepts=data_config['max_concepts'])
elif model_name in ["iekt"]:
curvalid = ATDKTDataset(os.path.join(data_config["dpath"], data_config["train_valid_file"]), data_config["input_type"], {i})
curtrain = ATDKTDataset(os.path.join(data_config["dpath"], data_config["train_valid_file"]), data_config["input_type"], all_folds - {i})
请问哪里出错了啊?
Great work, I managed to run examples and prepare my own "mock" - it works just fine. I have some difficulties though when analyzing the results - I'd like to check individual users' predictions to analyze how it changes when sequences change, go through some examples, etc. Where should I look for stuff like that in the repository?
train_model
in train_model.py
the function doesn't evaluate model with test dataset when save_model==True
however, in some cases we need to save model and evaluate with test dataset
if save_model:
torch.save(model.state_dict(), os.path.join(ckpt_path, model.emb_type+"_model.ckpt"))
max_auc = auc
best_epoch = i
testauc, testacc = -1, -1
window_testauc, window_testacc = -1, -1
if not save_model:
if test_loader != None:
save_test_path = os.path.join(ckpt_path, model.emb_type+"_test_predictions.txt")
testauc, testacc = evaluate(model, test_loader, model.model_name, save_test_path)
if test_window_loader != None:
save_test_path = os.path.join(ckpt_path, model.emb_type+"_test_window_predictions.txt")
window_testauc, window_testacc = evaluate(model, test_window_loader, model.model_name, save_test_path)
validauc, validacc = auc, acc
print(f"Epoch: {i}, validauc: {validauc:.4}, validacc: {validacc:.4}, best epoch: {best_epoch}, best auc: {max_auc:.4}, train loss: {loss_mean}, emb_type: {model.emb_type}, model: {model.model_name}, save_dir: {ckpt_path}")
print(f" testauc: {round(testauc,4)}, testacc: {round(testacc,4)}, window_testauc: {round(window_testauc,4)}, window_testacc: {round(window_testacc,4)}")
I will wait for other opinions.
Hello, I want to ask your opinion on the AKT model with regard to the reason why that model performs best in your delicate framework. (https://arxiv.org/abs/2206.11460)
the image above is the figure of AKT model represented in the paper
qa_embed_diff_data = self.qa_embed_diff(
target) # f_(ct,rt) or #h_rt (qt, rt)差异向量
if self.separate_qa:
qa_embed_data = qa_embed_data + pid_embed_data *
qa_embed_diff_data # uq* f_(ct,rt) + e_(ct,rt)
else:
qa_embed_data = qa_embed_data + pid_embed_data *
(qa_embed_diff_data+q_embed_diff_data) # + uq *(h_rt+d_ct) # (q-response emb diff + question emb diff)
and the code above is what you implemented at pykt/models/akt.py.
I think you followed the right way as the paper's author described.
The point is that I think AKT model has the best performance because it has a chance to know the target answers with "f(c_t, r_t) variation vector" (at the paper), which is "qa_embed_diff_data" (at your code).
As a result, in my opinion, AKT has the best performance because of its already-known target issue.
To resolve the issue, I suggest modifying Architecture forward function as the following code:
else: # dont peek current response
pad_zero = torch.zeros(batch_size, 1, x.size(-1)).to(self.device)
q = x
k = torch.cat([pad_zero, x[:, :-1, :]], dim=1)
v = torch.cat([pad_zero, y[:, :-1, :]], dim=1)
x = block(mask=0, query=q, key=k, values=v, apply_pos=True) # True: +FFN+残差+laynorm 非第一层与0~t-1的的q的attention, 对应图中Knowledge Retriever
# mask=0,不能看到当前的r
flag_first = True
thank you for your attention :)
您好,感谢贵团队对代码进行开源,能够让我学习到有价值的知识。目前项目中没有一题多知识点的数据集嘛。就比如说在qikt模型当中,需要对一个问题对应的多个知识点进行平均嵌入操作。
您好,请问Hyper parameter tunning results给出的每个模型每一折的参数配置,对应了当前模型在该折的最优性能吗
Can you share the benchmark results of your experiments with the ednet dataset?
�Still need to develop the data processing part for ednet experiments?
How long does it takes for GKT training?
Sharing other datasets training time of GKT might be helpful!
Thank you!
In the pykt paper, it seems to evaluate model with KC level.
How can we evalutate model in question level?
It is different with early fusion because, in the peper, DKT/GKT didn't evaluate with early fusion setting.
Do you have any plan to add question level evaluation metrics?
Originally posted by clabra May 7, 2023
Hi,
In the quick start document it is said in the Tuning Protocol section that:
We use the Bayes search method to find the best hyperparameter, it is expensive to run all the hyperparameter combinations. Hence, you can run the get_wandb_new file to check whether to stop the searching. We default to stop the searching if the number of the tuned hyperparameter combinations in each data fold is larger than 300 and there is no AUC improvement on the testing data in the last 100 rounds (output "end!").
But there is not get_wandb_new file in the git repo.
I think quick_start.md might need an update.
Thanks
Excuse me, can the model run only in the GPU environment that supports cuda?
代码修改:y,loss = model.train_one_step(data, device)后续一路添加
Hello, Thanks for the nice framework for knowledge tracing.
I'm also impressed with the delicate experiments you've done on the paper(https://arxiv.org/abs/2206.11460)
Now I'm working on reproducing your Table 2 results shown on the paper, with the assist2015 dataset you've also shared.
And, I got confused about getting the best hyperparameters as you mentioned at A.3 Hyperparameter Search Details of Representative DLKT Baselines.
So, there are two different settings on your code, which is,
Thanks.
我尝试在examples文件夹下执行python -u data_preprocess.py --dataset_name ednet。
最终报错
train+valid original interactions num: 178558352, select num: 0, qs: 12277, cs: 188, seqnum: 588152
Killed
在killed之前我观察到系统内存不断升高,直至killed。
我个人的机器配置有192G内存。请问是处理这个数据集需要更大的内存还是说预处理代码存在内存泄露?
Hi , I am currently working with DKT-F and encountered the following error: "Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!"
作者你好,非常感谢你们提供的非常有用的kt库,但是有个疑问咨询一下,在pykt论文中你们也提到一题多知识点存在信息泄露的问题,通常可以利用知识点聚合方式来避免,但是我看你们实现的代码里面关于模型输出然后loss计算那一块,也还是输入扩展后的知识点序列,然后one-by-one的预测?可能是我对代码的理解偏差,望指正,期待您的回复!
尊敬的作者团队,你们好:
我在尝试参加AAAI竞赛,在使用akt模型预测测试集时报出如下错误:
Namespace(atkt_pad=0, save_dir='saved_model/peiyou_akt_qid_saved_model_3407_0_0.2_256_512_8_4_0.0001_0_1', test_filename='pykt_test.csv', train_ratio=0.5, use_pred=0, use_wandb=0)
Start predicting model: akt, embtype: qid, save_dir: saved_model/peiyou_akt_qid_saved_model_3407_0_0.2_256_512_8_4_0.0001_0_1, dataset_name: peiyou
model_config: {'dropout': 0.2, 'd_model': 256, 'd_ff': 512, 'num_attn_heads': 8, 'n_blocks': 4}
data_config: {'dpath': '../data/peiyou', 'num_q': 7652, 'num_c': 865, 'input_type': ['questions', 'concepts'], 'max_concepts': 6, 'min_seq_len': 3, 'maxlen': 200, 'emb_path': '', 'train_valid_file': 'train_valid_sequences.csv', 'folds': [0, 1, 2, 3, 4]}
Start predict use_pred: False, ratio: 0.5...
Traceback (most recent call last):
File "wandb_eval.py", line 74, in <module>
main(params)
File "wandb_eval.py", line 55, in main
dfinal = evaluate_splitpred_question(model, data_config, testf, model_name, save_test_path, use_pred, ratio, atkt_pad)
File "/media/sda2/yueying/pykt-toolkit/pykt/models/evaluate_model.py", line 611, in evaluate_splitpred_question
qidxs, ctrues, cpreds = predict_each_group2(dtotal, dcur, dforget, curdforget, is_repeat, qidx, uid, idx, model_name, model, t, end, fout, atkt_pad)
File "/media/sda2/yueying/pykt-toolkit/pykt/models/evaluate_model.py", line 1034, in predict_each_group2
y, reg_loss = model(ccc.long(), ccr.long(), ccq.long())
File "/media/sda1/yueying/miniconda3/pkgs/envs/pykt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/media/sda2/yueying/pykt-toolkit/pykt/models/akt.py", line 84, in forward
q_embed_data, qa_embed_data = self.base_emb(q_data, target)
File "/media/sda2/yueying/pykt-toolkit/pykt/models/akt.py", line 77, in base_emb
qa_embed_data = self.qa_embed(target)+q_embed_data
File "/media/sda1/yueying/miniconda3/pkgs/envs/pykt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/media/sda1/yueying/miniconda3/pkgs/envs/pykt/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 160, in forward
self.norm_type, self.scale_grad_by_freq, self.sparse)
File "/media/sda1/yueying/miniconda3/pkgs/envs/pykt/lib/python3.7/site-packages/torch/nn/functional.py", line 2199, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
希望作者能够提供一些错误产生的线索,感谢!
Thank you very much for open sourcing this framework, which I think greatly facilitates the development of knowledge tracing. I would like to ask if it is possible to publish the optimal hyperparameters for each model on each dataset.
Hi, your pykt is a very nice project. But I got the following traceback on my machine :
File "**/pykt-toolkit/examples/wandb_train.py", line 71, in main
(train_loader, valid_loader) = init_dataset4train(dataset_name, model_name, data_config, fold, batch_size)
ValueError: too many values to unpack (expected 2)
after my troubleshooted,I find we have a package named pykt
,and a module named pykt
too. The paths for both of them are as follows:
pykt
: **/pykt-toolkit/pyktpykt
:**/envs/pykt/lib/python3.7/site-packages/pyktAccording to the description of the Modules
section of the python website (https://docs.python.org/3/tutorial/modules.html),`from-import` will choise package pykt
frist.
So, have you encountered the same problem, or can you give me some suggestions to solve it?
thank you very much!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.