This is the first Neural Network I really work with. I trained the model a few days ago. But I dont quite get how to predict a Students probability of answer the next problem right. Like is there a way I can give a sequence of data from a student and predict what the probabilty of him/her answering the next problem for any specific module existing?

When calculating the influence probability from i to j, do you average all predicted conditional probability P(j|i)？Another question is that if the dataset contains N students, do you need to average the predicted conditional probability P(j|i)？

As the DKT paper didn't explain the details for calculating Eq.4, I want to know how to calculate this influence probability.

Thanks for your response.

Thanks for your project. Really appreciate the effor you've put here. Is it possible to get a breakdown of the logic of the padding sequences function in the DataGenerator class? Its not obvious to me what happens when you have 0 features apart from the correct sequences, skill ids, and user ids compared to when you do.

I keep getting the following error(s) for the following line of code: x[idx, -len(trunc):] = trunc

With features:

could not broadcast input array from shape (21) into shape (21,20)

Without features:

could not broadcast input array from shape (21) into shape(21,1)

hi, i tried to execute your code but i didn't get the same result AUC test 0.85
i get
`======== Data Summary ========
Data size: 4163
Training data size: 2665
Validation data size: 666
Testing data size: 832
Number of skills: 123

C:\Users\nesri\Anaconda3\envs\deeptens\lib\site-packages\h5py_init_.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Using TensorFlow backend.
==== Training Started ====
Epoch 1/10
1/1 [==============================] - 27s 27s/step - loss: 0.6924 - val_loss: 0.6720

Epoch 00001: val_loss improved from inf to 0.67200, saving model to saved_models/ASSISTments.best.model.weights.hdf5
Epoch 2/10
1/1 [==============================] - 17s 17s/step - loss: 0.6274 - val_loss: 0.7233 - val_auc: 0.5728 - val_acc: 0.6659 - val_pre: 0.8103

Epoch 00002: val_loss did not improve from 0.67200
Epoch 3/10
1/1 [==============================] - 12s 12s/step - loss: 0.6143 - val_loss: 0.7112 - val_auc: 0.5124 - val_acc: 0.5180 - val_pre: 0.7895

Epoch 00003: val_loss did not improve from 0.67200
Epoch 4/10
1/1 [==============================] - 12s 12s/step - loss: 0.8007 - val_loss: 0.7096 - val_auc: 0.4878 - val_acc: 0.5618 - val_pre: 0.7132

Epoch 00004: val_loss did not improve from 0.67200
Epoch 5/10
1/1 [==============================] - 12s 12s/step - loss: 0.6123 - val_loss: 0.7175 - val_auc: 0.5361 - val_acc: 0.5119 - val_pre: 0.7680

Epoch 00005: val_loss did not improve from 0.67200
Epoch 6/10
1/1 [==============================] - 15s 15s/step - loss: 0.5901 - val_loss: 0.7050 - val_auc: 0.5799 - val_acc: 0.5316 - val_pre: 0.7890

Epoch 00006: val_loss did not improve from 0.67200
Epoch 7/10
1/1 [==============================] - 21s 21s/step - loss: 0.5678 - val_loss: 0.6871 - val_auc: 0.5698 - val_acc: 0.6002 - val_pre: 0.8045

Epoch 00007: val_loss did not improve from 0.67200
Epoch 8/10
1/1 [==============================] - 18s 18s/step - loss: 0.5614 - val_loss: 0.6999 - val_auc: 0.5930 - val_acc: 0.5912 - val_pre: 0.8069

Epoch 00008: val_loss did not improve from 0.67200
Epoch 9/10
1/1 [==============================] - 15s 15s/step - loss: 0.5558 - val_loss: 0.6991 - val_auc: 0.5879 - val_acc: 0.5212 - val_pre: 0.7704

Epoch 00009: val_loss did not improve from 0.67200
Epoch 10/10
1/1 [==============================] - 15s 15s/step - loss: 0.5477 - val_loss: 0.6910 - val_auc: 0.5659 - val_acc: 0.5392 - val_pre: 0.7811

Epoch 00010: val_loss did not improve from 0.67200
==== Training Done ====
==== Evaluation Started ====
1/1 [==============================] - 6s 6s/step - auc: 0.6729 - acc: 0.7172 - pre: 0.9235
==== Evaluation Done ====
`

i used optimizer = "adagrad"
lstm_units = 250
batch_size = 20
epochs = 10
dropout_rate = 0.6
verbose = 1
validation_rate = 0.2 # Portion of training data to be used for validation
testing_rate = 0.2 # Portion of data to be used for testing
and also i tried all parameters that you put but i didn't get same result why ?

每个epoch都有80次迭代在代码哪里设置的呢？

Hi Lucas, Thank you for open sourcing your project. Your code is really clean and easy to understand. But I came across this error while trying to execute it.

File "examples/run_dkt.py", line 115, in <module>
    run(parse_args())
  File "examples/run_dkt.py", line 46, in run
    tf.keras.callbacks.TensorBoard(log_dir=args.log_dir)
  File "/content/Deep-Knowledge-Tracing/deepkt/deepkt.py", line 141, in fit
    validation_freq=validation_freq)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 108, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 1133, in fit
    return_dict=True)
TypeError: evaluate() got an unexpected keyword argument 'x'

I ran the code on Google colab and got this error after first epoch is over.

To me it seems that this error is because of version mismatch. What version(s) of libraries are required to run it ?

Hi. Thank you for the code.

I have a question about the input to the model.

I wanna know why you are using both the 24-bit representation and the 13-bit representation as inputs to the model.

Shouldn't one of them be enough?

Hey @lccasagrande,
You have done something like this while creating dataset.
df['skill_with_answer'] = df['skill'] * 2 + df['correct']

so when we want to predict the probability of getting a question right based on past answers, where we don't have "correct" column. How we would predict?

I'm a bit confused.

Hi there! Thanks for your implementation.

I would like to clarify a doubt, if possible. From the data_util.py file we can observe:

# Step 4 - Convert to a sequence per user id and shift features 1 timestep
seq = df.groupby('user_id').apply(
    lambda r: (
        r['skill_with_answer'].values[:-1],
        r['skill'].values[1:],
        r['correct'].values[1:],
    )
)

I could not understand the reason for making that shift. Can you clarify?
Thanks in advance.

Hi Luca, I am actually working on Knowledge Tracing for my thesis. I looked at your code to see if I could find a DKT model as baseline.
While trying it I found 2 errors:

1. distraction error in metrics.py in TruePositives function, which extends tensorflow TrueNegatives (instead of truePositives) resulting in wrong ACC and AUC.
2. using "get_target" function before computing loss function and metrics does not support the use of automatic masking.
   I know you changed all -1 s to zeros, making it work, but this way the padded values will be considered in your metrics and loss as correct anwered values, modifying real Accuracy and AUC for example. I suggest you to try incorporating "get_target" function in a final custom layer of the model instead, giving as input the target skill to the model too. This way you can use directly a custom layer (remember to define compute_mask function too) to do the reduce_sum over the skill_true array. I have done it and values of AUC, ACC and loss change a lot in the case you have very different length of the sequences as input (due to large padding).

Despite this errors thanks for the code available, I used to understand better how to implement my own DKT as baseline
Bye ;)

evaluate() got an unexpected keyword argument 'x' , l don't know how to solve it.

Thanks for your python implmentation !

I have the following question：
Should an separate instance of model be created for each student in actual use? Or, is the interaction sequence of all students going through a single model in order (according to timestamp)？

ValueError: Expected 2D array, got scalar array instead:
array=51.0.
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

I have the trained model, but I wonder how I can actually plug in a point to see its probability. I am not sure here how to do that. How could I plug in a problem number (one hot encoding) in order to get a probability of getting it right.

Your code is pretty good!
But some troubles I have when I run the code:

• I read the paper where used 12 features to analyze the dataset and why you only group by user_id and skill_id?
• I want to carry out a CART (classify tree) base on the data.How to achieve？
• how to use the model I saved where the file suffixed with .hdf5 to test data?

Hello there
Appreciate your codes first!
btw i wanna know its probability of skill and can you pls let me know how can i find it?
Many thanks