Hello,Think you for your work,I successfully define your targeted dropout'function according your help.In order to let this function work,But now I do not know where can I call this function in my model .
Because dropout in fully-connection layer,is targeted dropout only in fully-connection layer?except for you reply

I saw that the targeted dropout on the weights can either select train or test mode, instead the targeted dropout on the units is always on train mode even when the model does the inference. Is there any reason for that?

• Targeted Weight Dropout Function:

def targeted_weight_dropout(w, params, is_training):
    drop_rate = params.drop_rate
    targ_perc = params.targ_rate
    w_shape = w.shape
    w = tf.reshape(w, [-1, w_shape[-1]])
    norm = tf.abs(w)
    idx = tf.to_int32(targ_perc * tf.to_float(tf.shape(w)[0]))
    threshold = tf.contrib.framework.sort(norm, axis=0)[idx]
    mask = norm < threshold[None, :]
    if not is_training:
        w = (1. - tf.to_float(mask)) * w
        w = tf.reshape(w, w_shape)
        return w
    mask = tf.to_float(
        tf.logical_and(tf.random_uniform(tf.shape(w)) < drop_rate, mask))
    w = (1. - mask) * w
    w = tf.reshape(w, w_shape)
    return w

https://github.com/for-ai/TD/blob/87cf2a0682274badbfde962023c657779aa79975/models/utils/dropouts.py#L23

• Targeted Unit Dropout Function:

def targeted_unit_dropout(x, params, is_training):
        w = tf.reshape(x, [-1, x.shape[-1]])
        norm = tf.norm(w, axis=0)
        idx = int(params.targ_rate * int(w.shape[1]))
        sorted_norms = tf.contrib.framework.sort(norm)
        threshold = sorted_norms[idx]
        mask = (norm < threshold)[None, :]
        mask = tf.tile(mask, [w.shape[0], 1])
        mask = tf.where(
                tf.logical_and((1. - params.drop_rate) < tf.random_uniform(tf.shape(w)),
                mask), tf.ones_like(w, dtype=tf.float32),
                tf.zeros_like(w, dtype=tf.float32))
        x = tf.reshape((1 - mask) * w, x.shape)
        return x

https://github.com/for-ai/TD/blob/87cf2a0682274badbfde962023c657779aa79975/models/utils/dropouts.py#L217

Hello，think you for your code about the parper of targeted dropout，I see you said It is simultaneously easy to implement, consisting of a two-line change using neural network frameworks such as Tensorflow，I want to which line need to change？How embed your targeted dropout in other model？Expect for your reply。

Hi,
When i executed the command python3.5 -m TD.train --env=local --hparams=resnet_default
Iam using two 2080Ti GPUs. Kindly please help

i got the error below:

raise ValueError('Please wrap optimizers with TowerOptimizer'
ValueError: Please wrap optimizers with TowerOptimizer in order to use replicate_model_fn with multiple devices.

Thank you for your reply.I successfully put your code in my model,but when I call the function of targeted dropout,I meet a problem, the problem is:
TypeError: bad operand type for unary -: 'TopKV2'
can you help me solve this problem?my these code is that:

Hello, I' m confused by the difference between variational_dropout and variational_unit_dropout because I think they are the same through the program. I will be very grateful if you can explain it for me.

The hparams for ResNet-32 seem to define an input layer with 16 filters followed by stacks of layers with 32, 64, and then 128 filters:
https://github.com/for-ai/TD/blob/master/hparams/resnet.py#L12

This doesn't match the ResNet paper, which defines ResNet-32 with CIFAR-10 as having an input layer of 16 filters followed by stacks of layers with 16, 32, and 64 filters:

Also, the pruning code reports a total of 1 857 498 parameters, much more than the 0.46M reported in the paper. Changing hps.residual_filters to [16, 16, 32, 64] gives me a much closer value of 468 890 parameters.

Thin you for your work,What's the difference between unit dropout and weight dropout?How are they implemented?Looking froward for your reply.

Can your targeted dropout'function only be used after training on the network?Can I use this function for network training?

Hello,Think your work,I want to know why do you consider unit pruning and weight pruning in L1 norm and L2 norm?What do L1 norm and L2 norm do in your parper?

https://github.com/for-ai/TD/blob/877e3b9d1491e78e4e7f1d2e3622983540e415e7/models/utils/dropouts.py#L236

Is there a reason why here you do not scale the input (divide by the probability of keeping a neuron) in order to have the same energy on the layer during training and testing? The standard dropout rescales the input by keep_prob:

tf.nn.dropout(...) -> https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/nn/dropout

Thank you!