Question about the difference between Transformer implementation vs original architecture in the paper. about eend_pytorch HOT 2 CLOSED

Hi, actually my model implementation strictly follows the one in paper.

If you look into PyTorch's TransformerEncoderLayer implementation, you will find it is in the order: self_attn->residual->norm->pointwise_ff->residual->norm. However in End-to-End Neural Speaker Diarization with Self-attention/Fig. 2, the encoder block is defined as: norm->self_attn->residual->norm->pointwise_ff->residual, and with a layer_norm at the end (before linear+sigmoid).

Thus, applying layer_norm at the beginning in PyTorch code is equal to what they have done in their paper.

from eend_pytorch.

Yeah, then everything makes sense. What a neat adaptation, thanks. Closing this now.

from eend_pytorch.