For reproducibility about pycox HOT 4 CLOSED

Hi,
It's been a while since that last time I looked into this, so this answer might be outdated.

Last time I checked, setting seeds in pytorch didn't really work as expected. In particular when using GPUs the don't really work.
They might have fixed it so that when running on CPU only you would be able to reproduce results, but I'm not completely sure.

Is this an issue both for running on GPU and CPU?

Could you maybe give me a code example of when reproducibility fails?

from pycox.

Thank you for responding!

Yes. On CPU, sometimes it works fine depending on the hyperparameters, but sometimes it doesn't. (not working on GPU)
I don't know why it works with certain hyperparameters.

I have one more question. Can the results obtained from the GPU be reproduced on the CPU using the same hyperparameters? If not, what do you think is the reason?

I'm using MLPVanilla too. There is probably no difference from the DeepSurv example for this problem :)

from pycox.

I don't think that there is anything special in pycox that would hurt reproducibility (only use random number generators from numpy and pytorch), so I don't think I can help you with this. As the pytorch docs 1.7.1 states:

results may not be reproducible between CPU and GPU executions, even when using identical seeds.

These docs also give some guidelines for how to improve reproducibility, so you might want to read through them.

I think it is strange that you get consistent results for a day, but the next day you get other results. I really don't know what might the reason for this. I'm sorry I'm not able to help.

Maybe if someone else have some insights on this topic, they might post them here in the future.

from pycox.

Thank you for your detailed answer!

from pycox.