mnist_dpsgd_tutorial_keras.py getting accuracies of about 10% about privacy HOT 5 CLOSED

Can you describe your setup? OS, TF version, etc...

from privacy.

I've tried both tf 1.14.0 and 2.0.0-beta1, and on both MacOS High Sierra and colab notebook. Both gives same results. The remaining is as mnist_dpsgd_tutorial_keras.py does:

learning_rate = 0.15
noise_multiplier = 1.1 #Ratio of the standard deviation to the clipping norm
l2_norm_clip = 1.0 #Clipping norm
batch_size = 250
epochs = 60
microbatches = 250

train_data, train_labels, test_data, test_labels = load_mnist()

optimizer = DPGradientDescentGaussianOptimizer(
l2_norm_clip=l2_norm_clip,
noise_multiplier=noise_multiplier,
num_microbatches=microbatches,
learning_rate=learning_rate)
# Compute vector of per-example loss rather than its mean over a minibatch.
loss = tf.keras.losses.CategoricalCrossentropy(
from_logits=True, reduction=tf.losses.Reduction.NONE)

# Compile model with Keras
mnist_model.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])

# Train model with Keras
mnist_model.fit(train_data, train_labels,
epochs=epochs,
validation_data=(test_data, test_labels),
batch_size=batch_size)

eps = compute_epsilon(epochs * 60000 // batch_size)
print('For delta=1e-5, the current epsilon is: %.2f' % eps)

from privacy.

I just ran the tutorial on a TF14 fresh install and couldn't reproduce the bug. Did you make any modifications to the tutorial?

from privacy.

You're right. I retried and verified that it works well with TF14. I was experimenting with TF 2.0.0-beta1 and could run the code after changing the logging to abseil and switching out the tf.contrib dependencies. But this gives only about 10% accuracy.

So the issue is with using TF 2.0.0-beta1.

from privacy.

I'm closing for now since we are still building up support for TF2. If you have specific bugs you identify with TF2, feel free to open separate issues for them.

from privacy.