Noise scaled by extra clip_norm about sprint_gan HOT 4 CLOSED

@SebFar Thank you very much for catching this, I’m reviewing now to confirm. I believe this may have been mistakenly changed due to differences in notation (variance vs. stddev). Rerunning to confirm, it looks like it still converges, but it will take some time to run all the way through and go back through the clinician analysis step.

We're confirming using Adam is not an issue so I’ll let @steven7woo confirm. On the initial check, it is ok to use Adam because the moments are calculated as a post-processing step based on the noisy gradient.

If you're interested we'd like to acknowledge you in any revised manuscript after re-analysing the results for your help identifying the error. Feel free to email me (brett_beaulieu-jones [at] hms.harvard.edu - if you'd rather discuss in private).

from sprint_gan.

@SebFar Thanks again for asking the question.
Similar to private SGD, private Adam incurs privacy loss only in the gradient calculation. All of the estimates for first-order and second-order moments can be viewed as post-processing of the privatized gradients. Therefore, it suffices to calculate the privacy loss incurs in releasing the noisy gradients, and for that purpose, the moments accountant from Abadi et al. suffices.

from sprint_gan.

Thanks Brett and Steven. I've emailed Brett with a more thorough explanation of why I think Adam might not work with the privacy accountant. I'd love to hear your thoughts about it.

from sprint_gan.

Thanks @SebFar, I am currently pushing the correction. To bring part of our offline conversation here, we discussed how the moments estimation steps of Adam are post-processing and therefore do not affect the privacy calculation.

We have rerun the results in the paper and are conducting the clinician review process now. I really appreciate your help!

from sprint_gan.