How to get the scores as Fig.4 in the paper did? about caster HOT 9 CLOSED

O=N+ is O=[N+]([O-]).

from caster.

Hi Yifan,

here is the notebook to generate the scores (https://github.com/kexinhuang12345/CASTER/blob/master/DDE/Interpretability-Check.ipynb). That is weird, because i remembered I switch 5 seeds to generate five models and calculate the correlation scores among these five seeds, they are very correlated and the nitrate groups are all very high (not necessarily the highest, but are all > 6). Here are the five runs results:

[array([ 0.63542426, 0.44687057, 6.33255529, 7.1794467 , 4.8289566 ,
7.77798128, 2.47209358, 7.77451897, 7.52848101, 3.0971632 ,
6.39500332, 5.20142174, 5.87682152, 4.52361298, 1.83678734,
6.79152107, 2.82770014, 7.7742219 , 1.89271498, 7.77455568,
0.4949483 , 6.2375164 , 0.934048 , 0.67924112, 7.77444696], dtype=float32),
array([ 0.01780428, 0.1331259 , 3.89042711, 7.06324005, 3.9596386 ,
6.61894989, 1.32479537, 6.9335289 , 6.64494944, 6.79534388,
4.23204994, 4.39468861, 6.81447411, 2.31668234, 4.62658739,
4.81248665, 5.48766088, 1.57640505, 6.49967289, 7.0633688 ,
0.71887422, 4.12155199, 0.1638066 , 0.17555533, 7.06337404], dtype=float32),
array([ 0.29192531, 0.48657787, 4.27260447, 6.35948324, 4.15854836,
6.19628334, 4.72457743, 6.28347683, 5.82954979, 6.35947514,
6.359622 , 4.14088583, 6.11369514, 3.15054655, 6.35948849,
5.28634882, 6.35953331, 3.35906601, 5.61386728, 6.18970537,
0.07247571, 2.49790788, 0.35807824, 0.99886924, 6.35951662], dtype=float32),
array([ 1.14123428, 2.53995657, 5.14567947, 7.20398521, 4.30770206,
7.6886797 , 4.89014482, 7.20399332, 5.61421061, 5.19274569,
5.81774426, 0.61560982, 6.16710186, 2.45936918, 0.8437506 ,
4.94349957, 4.3583622 , 7.20399761, 4.60464001, 6.39035606,
2.83984923, 2.44025087, 0.05058878, 0.03891326, 7.20398998], dtype=float32),
array([ 0.84122586, 0.92053282, 5.55640697, 5.47608137, 4.95133591,
4.66842747, 2.08443856, 5.47608232, 4.85661125, 5.47608709,
4.70946407, 4.87196779, 4.71470976, 0.4168151 , 0.18098168,
4.74224758, 3.11976147, 5.47608185, 4.18008804, 5.47608376,
0.61316276, 0.89249676, 0.78801113, 3.28678155, 5.4760828 ], dtype=float32)]

The nitrate group is the last one in every array.

Here are also the notebooks to generate these five models: https://github.com/kexinhuang12345/CASTER/blob/master/DDE/Run_Explainability_Models.ipynb

from caster.

Hi, I just used the Interpretability-Check.ipynb. I also loaded "model_train_checkpoint_SNAP_EarlyStopping_SemiSup_Full_Run1.pt" from the repo.
And I get
array([0.1980899 , 1.9535173 , 1.9696809 , 7.4186006 , 5.2999167 ,
4.2064114 , 7.661639 , 8.107381 , 7.21804 , 2.1968257 ,
7.060696 , 5.3268137 , 0.430464 , 1.2191231 , 6.175004 ,
4.7044473 , 4.5096974 , 8.107395 , 8.091772 , 7.7659383 ,
0.17249212, 2.4313147 , 0.18751104, 0.23494425, 1.8431011 ],
dtype=float32)
It is 1.8431011. I found that the little difference (I got 2.7843778 in the original post) was due to the difference in our smiles of Sildenafil.
I use CCCc1nn(C)c2c(=O)[nH]c(-c3cc(S(=O)(=O)N4CCN(C)CC4)ccc3OCC)nc12 while yours is CCCc1nn(C)c2c(=O)nc(-c3cc(S(=O)(=O)N4CCN(C)CC4)ccc3OCC)[nH]c12 . Anyway, it doesn't matter.

The problem still exists, while I continue training for more iteration, the scores changed a lot: (Step by 1 iteration here)
1.8431014 -> 0.5254684 -> 0.96302533 -> 0.85627586 ......
The scores can be reproduced if you use "model_train_checkpoint_SNAP_EarlyStopping_SemiSup_Full_Run1.pt"

So I wonder whether you can provide the model file. For example 'model_train_checkpoint_deepDDI_small_Run2_explainability_seed5.pt'. And continue the training for several iterations, to see how the score may change?

Thanks!

from caster.

yes, i can imagine that by further training, it would change the model weight and thus the final result, but i don't think this should be a lot of differences. Here is the link for the five models: https://hu-my.sharepoint.com/:u:/g/personal/kexinhuang_hsph_harvard_edu/EQLExRVRVCFOmxrB9r7lwpgBuy2cd7nkzVpHQQxJ4DCQMA?e=3tT71N

from caster.

let me know if it still does not work, thanks!

from caster.

Thanks! I just test the models. They work well and the scores didn't change greatly as the training went on.
7.7742643 -> 7.803271 -> 7.816995 -> 7.8237867
So I believed that there may be something wrong with the training procedure.
I generate my own substructures files. It contains 343 substructures only. And it doesn't contain the nitrate group O=[N+]([O-]). Instead, it has O=, [N+] and ([O-]) instead. I tried to use larger substructures files in fact. For example, your 1722 substructures. However, it runs tooooooo slow on GPU without tensor core (like P100, 1080ti) since the model needs to do matrix inversion. (I need to run many episodes since I am doing meta-learning work). Also, the scores still vibrate a lot after few iterations though the model has been trained well on the specific task.
So I have two problems and I look forward to your answer:

1. How can I assess the interpretability if I only use the 343 substructures files? By averaging the scores of O=, [N+] and ([O-])?
2. Do you have any experience or trick in the training of CASTER? Because the scores' vibration problems exist in "model_train_checkpoint_SNAP_EarlyStopping_SemiSup_Full_Run1.pt" model. And the problems disappear in 'model_train_checkpoint_deepDDI_small_Run2_explainability_seed5.pt' model.
   Thanks!

from caster.

hi, glad to hear it is working. yes, i remembered the training is slow due to the matrix inversion part. For problem 1, is there a specific reason why you only need to use 343 files? since given such a small vocabulary, i am worried the patterns learnt through the data driven module is not informative enough. For example, the nitrate group has not been frequent enough to be included in the vocabulary. I guess for CASTER, the average wouldn't work since the whole idea is to generate a unique score based on the linear combination of individual substructures. I am worried that given such a small vocabulary, these individual units such as O=, [N+] may occur many times even in a single drug, which makes the signal noisy, and hard to pinpoint if it is the score for the nitrate group or other groups. For the second one, honestly, I have no idea why the first model does not work, maybe it is because of the training dataset, since SNAP is a small set whereas DeepDDI is very large and the training is relatively stable, but i don't know on the top of my head, will keep you posted if i found anything.

from caster.

Thanks so much for your advice! I also have a concern about the limited numbers of substructures. The reason is that I need to perform meta-learning, which needs many episodes (iterations), for about 1,000,000, to obtain the meta-knowledge. So the matrix inversion for 1722*1722 size is too slow for me. In this situation, even my own RTX2060 runs much faster than the 1080ti and P100, since 2060 has tensor cores. Maybe I should try to find GPUs with tensor cores first.

from caster.

No problems! That makes sense, good luck!

from caster.