<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Policy out softmax with illegal moves about reversi-alpha-zero HOT 4 OPEN

apollo-time commented on August 26, 2024

Policy out softmax with illegal moves

from reversi-alpha-zero.

Comments (4)

mokemokechicken commented on August 26, 2024 2

@apollo-time

oh, I was careless.
How about like this?

  legal_mask = Input((8 * 8))  # (0: illegal, 1: legal)
  legal_mask_2 = Lambda(lambda x: (x-1)*1000000)(legal_mask)  # illegal -> -1000000, legal -> 0
  ...
  # no output for 'pass'
  x = Dense(8*8, kernel_regularizer=l2(mc.l2_reg))(x)
  x = Add()([x, legal_mask_2])
  policy_out = Activation("softmax", name="policy_out")(x)
  
  ...
  
  self.model = Model([in_x, legal_mask], [policy_out, value_out], name="reversi_model")

> softmax([-0.5, 0.5])
[ 0.26894142,  0.73105858]

> softmax([-0.5, 0.5, -1000000])
[ 0.26894142,  0.73105858,  0.        ]

from reversi-alpha-zero.

mokemokechicken commented on August 26, 2024 1

I see calculate policy softmax on the all moves contains illegal.
How can calculate softmax on the only legal moves, if set placeholder for legal moves?

Does this answer understand the intent of the question?

For example,

  # no output for 'pass'
  policy_out = Dense(8*8, kernel_regularizer=l2(mc.l2_reg), activation="softmax", name="policy_out")(x)

↓

  legal_mask = Input((8 * 8))  # (0: illegal, 1: legal)
  ...
  # no output for 'pass'
  x = Dense(8*8, kernel_regularizer=l2(mc.l2_reg))(x)
  x = Multiply()([x, legal_mask])
  policy_out = Activation("softmax", name="policy_out")(x)
  
  ...
  
  self.model = Model([in_x, legal_mask], [policy_out, value_out], name="reversi_model")