Thank you for your interesting work! :)

I wonder if the temperature very close to 0 (e.g., 1e-20) makes the backpropagation error in practice.

In addition, is there a proper temperature you recommend?

I don't really understand this code.

Are there any code examples for the application of this kind of VAE to semi-supervised classification, as described in section 4.3 of the Gumbel-Softmax paper?
I understand the technical details in the appendix, but I'm having trouble defining the right loss function for the model depicted in Figure 6. Any help would be much appreciated!

logits_py should be the log of the current logits_py or we can just give it as probs and not logits to the corresponding distribution.

Hi, Eric, I extent this code for evaluation. But I found the results in this paper can not be achieved if I discretized the output in the evaluation. Instead, if I keep the training and test consistent, the results are similar to those in the paper.

@ericjang Thanks for the good code! I wonder whether we could use gumbel-softmax for clustering under VAE framework (e.g. for Mnist, 392->..->10->..->392, and "10" represents the relaxed categorical distribution). I have tried, but got bad results. Do you have any idea?

Hi, I use TensorFlow 1.1.0 without gpu in mac osx 10.12.3 to test
gumbel_softmax_vae_v2.ipynb
I use ipython to run the code line by line, the error occurs in defining logits_y.
Hard to find in google or other areas. Any comments?

In [12]: logits_y = tf.reshape(slim.fully_connected(net,K*N,activation_fn=None),[-1,N,K])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-12-d72f3b04d264> in <module>()
----> 1 logits_y = tf.reshape(slim.fully_connected(net,K*N,activation_fn=None),[-1,N,K])

/Users/Tiger/anaconda/envs/pytorch3/lib/python3.5/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py in func_with_args(*args, **kwargs)
    179       current_args = current_scope[key_func].copy()
    180       current_args.update(kwargs)
--> 181     return func(*args, **current_args)
    182   _add_op(func)
    183   setattr(func_with_args, '_key_op', _key_op(func))

/Users/Tiger/anaconda/envs/pytorch3/lib/python3.5/site-packages/tensorflow/contrib/layers/python/layers/layers.py in fully_connected(inputs, num_outputs, activation_fn, normalizer_fn, normalizer_params, weights_initializer, weights_regularizer, biases_initializer, biases_regularizer, reuse, variables_collections, outputs_collections, trainable, scope)
   1408   """
   1409   if not isinstance(num_outputs, six.integer_types):
-> 1410     raise ValueError('num_outputs should be int or long, got %s.', num_outputs)
   1411
   1412   layer_variable_getter = _build_variable_getter({'bias': 'biases',

ValueError: ('num_outputs should be int or long, got %s.', 200.0)

Can you explain what recons stands for? Please not use acronyms or shorthand without annotation. Thank you!

def gumbel_softmax(logits, temperature, hard=False):
"""Sample from the Gumbel-Softmax distribution and optionally discretize.
Args:
logits: [batch_size, n_class] unnormalized log-probs
temperature: non-negative scalar
hard: if True, take argmax, but differentiate w.r.t. soft sample y
Returns:
[batch_size, n_class] sample from the Gumbel-Softmax distribution.
If hard=True, then the returned sample will be one-hot, otherwise it will
be a probabilitiy distribution that sums to 1 across classes
"""
y = gumbel_softmax_sample(logits, temperature)
if hard:
k = tf.shape(logits)[-1]
#y_hard = tf.cast(tf.one_hot(tf.argmax(y,1),k), y.dtype)
y_hard = tf.cast(tf.equal(y,tf.reduce_max(y,1,keep_dims=True)),y.dtype)
y = tf.stop_gradient(y_hard - y) + y
return y

i have a question here, why should there be a stop_gradient before y_hard-y.
y_hard comes from equal，as i think，the equal function could be backpropagated just as max function did。
am i wrong？

In the gumbel_softmax_vae_v2 notebook, for calculating the KL in case of Relaxed Prior, shouldn't the KL be y * (log q_y - log p_y). So the product with y is missing?

I was playing around with the notebook, trying to look at the intermediate representations of the training data. I was expecting that the output of the y layer would be (pretty) sparse and (nearly) binarized. But it seems like that's not the case:

...
Step 40001, ELBO: -101.598
Step 45001, ELBO: -99.799

>>> np_x, _ = data.next_batch(1)
>>> emb = sess.run(y, {x : np_x})
>>> emb.max(axis=-1) # Value of maximum of embedding -- would expect to be 1
array([ 0.13201179,  0.36978129,  0.41773844,  0.26891398,  0.24909849,
        0.21777716,  0.1552867 ,  0.47244716,  0.16195767,  0.39042374,
        0.17623694,  0.2765696 ,  0.19546057,  0.18048088,  0.12659149,
        0.64287513,  0.14742081,  0.2126791 ,  0.53717244,  0.23660626,
        0.14906606,  0.15466955,  0.1191797 ,  0.20597951,  0.25431085,
        0.1979771 ,  0.16981648,  0.2198326 ,  0.17538837,  0.27005175], dtype=float32)

>>> ((emb < 0.01) | (emb > 0.99)).mean()
0.12

So it looks like the intermediate representations are still dense and not very binary. Any thoughts? (I'm new to Tensorflow/VAEs, so I may be making some silly coding/conceptual mistake...)

Edit: Maybe this is a matter of the hard parameter in gumbel_softmax? I understand that forces the representation to be sparse/binary, but AFAIK it'd just be a sample from a categorical distribution that doesn't necessarily have most of it's mass on a single category.

I would like to test your code with the transformer architecture in fairseq.

Have you ever tried?

Could you please suggest me the better way to do that?