The details of "samples one sub-graph at one training iteration" about autodl-projects HOT 4 CLOSED

Sure, we use the hard mode and thus it is a one-shot vector. Something like this in PyTorch:

y_soft = Gumbel Softmax( ... )
y_hard = one_hot( y_soft )
y_hard = y_hard - y_soft.detach() + y_soft

During the forward, you could use:

cals = []
for i, w in enumerate(y_hard):
  if w.item() == 1:
    cals.append( op[i](x) * w )
  else:
    cals.append( x )
return sum(cals)

from autodl-projects.

Sure, we use the hard mode and thus it is a one-shot vector. Something like this in PyTorch:

y_soft = Gumbel Softmax( ... )
y_hard = one_hot( y_soft )
y_hard = y_hard - y_soft.detach() + y_soft

How do you implement the backward process?

from autodl-projects.

If you implement forward in the above style, it can automatically backward in PyTorch.

from autodl-projects.

Sure, we use the hard mode and thus it is a one-shot vector. Something like this in PyTorch:

y_soft = Gumbel Softmax( ... )
y_hard = one_hot( y_soft )
y_hard = y_hard - y_soft.detach() + y_soft

During the forward, you could use:

cals = []
for i, w in enumerate(y_hard):
  if w.item() == 1:
    cals.append( op[i](x) * w )
  else:
    cals.append( x )
return sum(cals)

Thanks for the code snippet!
Why would you append x for the paths with weight 0? Shouldn't there be no forward propagation?

from autodl-projects.