Can you provide some pre training weights？Thank you

Hello! Nice work with some slight drawbacks.

1. In 3.5 "The head dim in E-MHSA and MHCA is set as 32".
   But in MHCA, actually you set groups =32 but not groups = dim // 32.
   And in E-MHSA, you set inner dim to be 64(maybe you could check the overall operation and dim)
2. In 3.5 "spatial reduction ratios in E-MHSA is [8, 4, 2, 1] in different stages"
   But you set a uniform sr_ratio, or "stride" in your code
3. In 3.5 "The expansion ratios of MLP layer are set as 3 for NCB and 2 for NTB, respectively."
   You simply ignore this. And this will cause a dramatic drop in parameters and FLOPs, as well as the model performance.
4. With respect to Block, I think here is a bug, because the author didn't mentioned it detailed so we can't determined the input and output dim between "L" blocks. I think it proper to implement this as you did, i.e, in_dim for "L-1" and turn to out_dim at last repeat.