lucidrains / conformer Goto Github PK

As the paper described, LayerNorm layer are added to FeedForward module and Attention module. But I can't find it in this project.

Is it something wrong?

I am new in speech processing and cannot understand how to train this model. if you could add it will be great for starting point

Hi @lucidrains, here's Conformer encoder model architecture, it has a few other Modules:

I have not started working on it, should I do a PR to include Multi-Headed Self-Attention Module and Feed forward module?

env:

• PyTorch 1.6.0

log:

conformer/demo.py", line 14
    x = layer(x) + x
RuntimeError: The size of tensor a (1026) must match the size of tensor b (1024) at non-singleton dimension 1

Hey lucidrains, firstly thanks for providing this implementation!

I've noticed in the ConformerConvModule the inner dimension is increased by an expansion_factor which defaults to 2
inner_dim = dim * expansion_factor
In the implementations by espnet (https://espnet.github.io/espnet/_modules/espnet/nets/pytorch_backend/conformer/convolution.html) and nvidia (https://github.com/NVIDIA/NeMo/blob/94a464fc4eb2927140940cc835a0ab69ee0347b5/nemo/collections/asr/parts/submodules/conformer_modules.py) the dimensions of the convolutional module are the same as the encoder dimensions i.e no expansion.

Obviously this just defaults to two and is configurable but - I'm wondering whether this is unintentionally the default, or if you think/have found this to be a good use of parameters?

the conformer paper is a bit sparse on details but I think the expansion factor they refer to in there corresponds to the out_channels for the pointwise convolution, which is also applied here i.e
nn.Conv1d(dim, inner_dim * 2, 1),

Thanks ! (:

Hi, thanks for releasing this code. I'm enjoying playing with this.

Could you please explain what is 'context' in your Attention module, if possible? link.

I guess it has a similar role to the previous hidden state for reuse in Transformer-XL paper link (which is explained in Section 3.2), but the way you implemented is little different.

Can you support streaming conformer like how this paper proposes to do it ?

why in conformer, you do: q*k * sqrt(embedding_dim)

@lucidrains I enjoy your super-readable implementations, and I think we can make them ever simpler.
Here are a couple of suggestions:

Einops provides layers (einops.layers.torch) which could make it more verbose:

# was 
Transpose((1, 2))
<convolutional part>
Transpose((1, 2))

# now
Rearrange('b token c -> b c token'),
<convolutional part>
Rearrange('b c token -> b token c'),

This part of code looks a bit over-engineered:

q, k, v = (self.to_q(x), *self.to_kv(context).chunk(2, dim = -1))
q, k, v = map(lambda t: rearrange(t, 'b n (h d) -> b h n d', h = h), (q, k, v))

How about using a new layer from einops:

self.to_qkv = WeightedEinsum('b n c -> qkv b h n d', qkv=3, h=heads, d=inner_dim//heads)
# then in forward
q, k, v = self.to_qkv(x)

This change will also remove need to store/pass heads info and will remove 2-4 more lines.

Sounds like a great place to use einsum? Still need add one None-axis afterwards, but I'd give it a go to make it obvious

mask = mask[:, None, :, None] * context_mask[:, None, None, :]

PS. Actually enjoyed reading the code. Like read a paper in several minutes :)

Hello Lucidrains, thank you for your wonderful work!

When I followed the usage of README by:

import torch
from conformer import ConformerBlock

block = ConformerBlock(
    dim = 512,
    dim_head = 64,
    heads = 8,
    ff_mult = 4,
    conv_expansion_factor = 2,
    conv_kernel_size = 31,
    attn_dropout = 0.,
    ff_dropout = 0.,
    conv_dropout = 0.
)

x = torch.randn(1, 1024, 512)
block(x) # (1, 1024, 512)

Some Attribute Error happened:

Traceback (most recent call last):
  File "/home/xxx/ASR/lucidrains-conformer/try.py", line 17, in <module>
    block(x) # (1, 1024, 512)
  File "/home/xxx/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xxx/ASR/lucidrains-conformer/conformer/conformer.py", line 197, in forward
    x = self.attn(x, mask = mask) + x
  File "/home/xxx/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xxx/ASR/lucidrains-conformer/conformer/conformer.py", line 64, in forward
    return self.fn(x, **kwargs)
  File "/home/xxx/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xxx/ASR/lucidrains-conformer/conformer/conformer.py", line 100, in forward
    dist = dist.clip(-max_pos_emb, max_pos_emb) + max_pos_emb
AttributeError: 'Tensor' object has no attribute 'clip'

May I ask what the problem is?

Thank you very much for your help.

Hi
I think there are two smal mistakes ni your implementation.

1- The feed forward module starts with a normalization layer which is omitted in your code. Furthermore, the module is more the module as your code it plus the identity. That's done in the conformer block, but maybe is can be more consistent to do it directly in the FFN ?
2- In the conformer block, the article mentionned that the MHSA is sandwitched between two half-step feed-forward modules, not full modules