as is said in tittle, could u plz share the baiduyun code to extracte the pretrained ckpt files? I found that it is really too slow for me to download ckpts using github urls. T.T

is it still 7 in 384*384 size of imagenet or in ade20k?
That may raise errors?so how about the setting policy?
Or I just misuderstanding the paper.

output = model(image)
#output.shape=torch.Size([1, 1000])
output.topk(5,dim=1)#add the code behind

I use the weight file is swin_tiny_patch4_window7_224.pth

I use a bird image(above) as the model input .
The result of output.topk is
torch.return_types.topk(
values=tensor([[1.7864, 1.6988, 1.6899, 1.6638, 1.5892]]),
indices=tensor([[627, 468, 450, 41, 160]]))

I want to know the Imagenet output tensor index to label mapping.

Hi,
Could you share the checkpoint of Swin-Large trained on ImageNet-1K only?
ImageNet-22k is too difficult to get.

Thanks a lot.

Hi, I noted that for the 4 object detection frameworks in your paper, you use the same lr setting: AdamW with lr=0.0001. But the base lr settings for them are different.
Cascade mask r-cnn: SGD with lr=0.02
ATSS: SGD with lr=0.01
RepPoints v2: SGD with lr=0.01
Sparse RCNN: AdamW with lr=0.000025
Leave the optimizer type alone, how do you decide the lr when using swin-tranformer as the backbone for these 4 frameworks? Seems that your lr has nothing to do with their original ones. This puzzled me. From my point, lr should be adjusted according to the network structure and the loss formation. But you just use the same setting. How to explain this? Any advice, thanks.

When I run the command

python -m torch.distributed.launch --nproc_per_node 1 --master_port 12345 main.py --eval --cfg configs/swin_base_patch4_window7_224.yaml --resume swin_base_patch4_window7_224.pth --data-path ../imagenet --amp-opt-level O0

I get the following error

Could anyone tell me what's wrong with that? How to solve this problem? I newly created a conda virtual environment swin and installed everything as the get_started.md said

Hi, I wonder that what's the naive/kernel sliding window. The paper didn't provide much description of them.

Why the speeding-up is so obvious?

Looking forward to your reply!
Thanks

Hello, can you provide us with the method of using the pre-trained model you gave in image classification?

when to support mscoco? thx

ValueError: Error initializing torch.distributed using env:// rendezvous: environment variable RANK expected, but not set

When I use the O1 train the swin-net, but
Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 131072.0 [2021-04-25 05:35:19 swin_base_patch4_window7_224](main_prune.py 310): INFO Train: [0/300][4050/5004] eta 0:17:06 lr 0.000500 time 1.1737 (1.0765) loss 3.2572 (3.3279) grad_norm 1.0323 (nan) mem 4814MB

Is this normal?

hi,thanks for your project! i have some questions.
Does the structure of swin transformer limit its ability to fine-tune with a larger resolution? For example, similar to DeiT, it can use a lower training resolution and fine-tune the network at the larger resolution by interpolating the positional encoding. Moreover,Is there a larger resolution pretrain model on imagenet22k(e.g 448)

Is your code written in the python framework?

If I pre-train Swin-T for 224 input image size. How can I finetune it and get Swin-T for 320 input image size?
In your paper, you claimed 384^2 input models are obtained by fine-tuning:

For other resolutions such as 384^2, we fine-tune the models trained at 224^2 resolution, instead of training from scratch, to reduce GPU consumption.

However, in this implementation, the fine-tuning is hard to do because the existence of parameters: relative_position_bias_table and attn_mask. If I change the input size, these two parameters are also changed.

So how can I modify the code to support fine-tuning on larger input image size?
Thanks!!

Hello, thank you for sharing. When testing the model inference speed, I used your swin_tiny_patch4_window7_224.pth, but the inference time of my 224x224 tensor input on V100 is about 15ms, which is a bit different from the speed of 755.2/s in the paper.



May I ask where I deal with the problem?

hi！Which and how many gpus did you use when you train each(Swin-T,S,B,L) model?

Hi, i download the models from get_start.md and want to eval on ImageNet1k.

• The eval acc is 0, which is as follows.

[2021-04-13 18:15:43 swin_tiny_patch4_window7_224](main.py 266): INFO Test: [0/98]	Time 5.697 (5.697)	Loss 9.3819 (9.3819)	Acc@1 0.000 (0.000)	Acc@5 0.586 (0.586)	Mem 2502MB
[2021-04-13 18:15:47 swin_tiny_patch4_window7_224](main.py 266): INFO Test: [10/98]	Time 0.285 (0.893)	Loss 9.3991 (9.4262)	Acc@1 0.000 (0.018)	Acc@5 0.391 (0.178)	Mem 2503MB
[2021-04-13 18:15:50 swin_tiny_patch4_window7_224](main.py 266): INFO Test: [20/98]	Time 0.554 (0.638)	Loss 9.4262 (9.4286)	Acc@1 0.195 (0.028)	Acc@5 0.391 (0.270)	Mem 2503MB
[2021-04-13 18:15:53 swin_tiny_patch4_window7_224](main.py 266): INFO Test: [30/98]	Time 0.472 (0.535)	Loss 9.3771 (9.4292)	Acc@1 0.195 (0.063)	Acc@5 0.391 (0.290)	Mem 2503MB
[2021-04-13 18:15:57 swin_tiny_patch4_window7_224](main.py 266): INFO Test: [40/98]	Time 0.507 (0.490)	Loss 9.4310 (9.4236)	Acc@1 0.195 (0.067)	Acc@5 0.586 (0.286)	Mem 2503MB
[2021-04-13 18:16:00 swin_tiny_patch4_window7_224](main.py 266): INFO Test: [50/98]	Time 0.299 (0.458)	Loss 9.4321 (9.4172)	Acc@1 0.000 (0.092)	Acc@5 0.195 (0.341)	Mem 2503MB
[2021-04-13 18:16:03 swin_tiny_patch4_window7_224](main.py 266): INFO Test: [60/98]	Time 0.197 (0.436)	Loss 9.4335 (9.4172)	Acc@1 0.195 (0.090)	Acc@5 0.195 (0.336)	Mem 2503MB
[2021-04-13 18:16:07 swin_tiny_patch4_window7_224](main.py 266): INFO Test: [70/98]	Time 0.235 (0.420)	Loss 9.4177 (9.4207)	Acc@1 0.000 (0.091)	Acc@5 0.391 (0.322)	Mem 2503MB
[2021-04-13 18:16:10 swin_tiny_patch4_window7_224](main.py 266): INFO Test: [80/98]	Time 0.240 (0.408)	Loss 9.3358 (9.4199)	Acc@1 0.586 (0.096)	Acc@5 0.586 (0.323)	Mem 2503MB
[2021-04-13 18:16:13 swin_tiny_patch4_window7_224](main.py 266): INFO Test: [90/98]	Time 0.232 (0.396)	Loss 9.3683 (9.4161)	Acc@1 0.195 (0.097)	Acc@5 0.391 (0.324)	Mem 2503MB
[2021-04-13 18:16:15 swin_tiny_patch4_window7_224](main.py 272): INFO  * Acc@1 0.096 Acc@5 0.318
[2021-04-13 18:16:15 swin_tiny_patch4_window7_224](main.py 121): INFO Accuracy of the network on the 50000 test images: 0.1%

• The shell is as follows.

python3.7 -m torch.distributed.launch \
    --nproc_per_node 4 \
    --master_port 12345 \
    main.py \
        --eval \
        --cfg="configs/swin_tiny_patch4_window7_224.yaml"  \
        --resume="./swin_tiny_patch4_window7_224.pth" \
        --data-path="/data/ILSVRC2012"

• The only diff is

• Could you please help me see why it does not work? thanks!

Performance on small datasets, such as Cifar..?

How can I get the following dataset？

data
└── ImageNet-Zip
├── train_map.txt
├── train.zip
├── val_map.txt
└── val.zip

We are now using the standard folder dataset, the speech of which is slow, about 2 hours/ 1 epoch.

Thanks for your sharing.

Thanks for releasing the code!

However, we use your codebase for training and testing some models and evaluate their performance on ImageNet1K(w/o pretrained on ImageNet22K). However, according to the training log, the max accuracy is 78.58. But when we evaluate the model with evaluation mode by resuming the best-performed checkpoint, the accuracy becomes 78.7. This discrepancy between training log reported accuracy and evaluation accuracy happens on many models.

Here I provide our evaluation script. Would you please let us know why?

CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch \
                                    --nproc_per_node=1 \
                                    --master_port 12345 \
                                    main.py \
                                    --eval \
                                    --cfg configs/our_model.yaml \
                                    --batch-size 2 \
                                    --resume 'output/our_model/best_ckpt.pth' \
                                    --data-path "path/to/dataset" \
                                    --zip 

Many thanks!

https://github.com/kamalkraj/Swin-Transformer-Serve

Nice work!And i reading your code recently. But i cannot understand well about the implementation of the mask in shifted window attention.

I simply draw a picture like below. The red mean the mask, and i choose windowsize as 2, shiftsize as 1.

I think the mask should be like this

but i use your code to generate mask like this:

import torch
import torch.nn as nn


def window_partition(x, window_size):
    """
    Args:
        x: (B, H, W, C)
        window_size (int): window size

    Returns:
        windows: (num_windows*B, window_size, window_size, C)
    """
    B, H, W, C = x.shape
    x = x.view(B, H // window_size, window_size, W // window_size, window_size, C)
    windows = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(-1, window_size, window_size, C)
    return windows


window_size = 2
shift_size = 1
H, W = 4, 4
img_mask = torch.zeros((1, H, W, 1))  # 1 H W 1
h_slices = (slice(0, -window_size),
            slice(-window_size, -shift_size),
            slice(-shift_size, None))
w_slices = (slice(0, -window_size),
            slice(-window_size, -shift_size),
            slice(-shift_size, None))

cnt = 0
for h in h_slices:
for w in w_slices:
img_mask[:, h, w, :] = cnt
cnt += 1

mask_windows = window_partition(img_mask, window_size)  # nW, window_size, window_size, 1
mask_windows = mask_windows.view(-1, window_size * window_size)
attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2)
attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))
attn_mask = attn_mask.unsqueeze(1).unsqueeze(0)

"""
tensor([[[[[   0.,    0.,    0.,    0.],
           [   0.,    0.,    0.,    0.],
           [   0.,    0.,    0.,    0.],
           [   0.,    0.,    0.,    0.]]],


         [[[   0., -100.,    0., -100.],
           [-100.,    0., -100.,    0.],
           [   0., -100.,    0., -100.],
           [-100.,    0., -100.,    0.]]],


         [[[   0.,    0., -100., -100.],
           [   0.,    0., -100., -100.],
           [-100., -100.,    0.,    0.],
           [-100., -100.,    0.,    0.]]],


         [[[   0., -100., -100., -100.],
           [-100.,    0., -100., -100.],
           [-100., -100.,    0., -100.],
           [-100., -100., -100.,    0.]]]]])
"""

I cannot understand it, can you give me a favor?

[2021-04-25 21:57:25 swin_tiny_patch4_window7_224](main.py 221): INFO Train: [3/300][370/1251] eta 0:08:57 lr 0.000166 time 0.6068 (0.6098) loss 6.3568 (6.1324) grad_norm 2.0382 (inf) mem 11818MB
[2021-04-25 21:57:31 swin_tiny_patch4_window7_224](main.py 221): INFO Train: [3/300][380/1251] eta 0:08:51 lr 0.000166 time 0.6025 (0.6097) loss 5.5428 (6.1310) grad_norm 2.3143 (inf) mem 11818MB
[2021-04-25 21:57:37 swin_tiny_patch4_window7_224](main.py 221): INFO Train: [3/300][390/1251] eta 0:08:44 lr 0.000166 time 0.6019 (0.6095) loss 5.6160 (6.1325) grad_norm 2.1987 (inf) mem 11818MB
[2021-04-25 21:57:43 swin_tiny_patch4_window7_224](main.py 221): INFO Train: [3/300][400/1251] eta 0:08:38 lr 0.000167 time 0.6021 (0.6094) loss 5.8623 (6.1306) grad_norm 2.3630 (inf) mem 11818MB
[2021-04-25 21:57:49 swin_tiny_patch4_window7_224](main.py 221): INFO Train: [3/300][410/1251] eta 0:08:32 lr 0.000167 time 0.6053 (0.6092) loss 6.3662 (6.1315) grad_norm 1.8832 (inf) mem 11818MB
[2021-04-25 21:57:55 swin_tiny_patch4_window7_224](main.py 221): INFO Train: [3/300][420/1251] eta 0:08:26 lr 0.000168 time 0.6034 (0.6091) loss 6.3618 (6.1325) grad_norm 2.0002 (inf) mem 11818MB
[2021-04-25 21:58:01 swin_tiny_patch4_window7_224](main.py 221): INFO Train: [3/300][430/1251] eta 0:08:19 lr 0.000168 time 0.6029 (0.6090) loss 6.3323 (6.1320) grad_norm 2.1190 (inf) mem 11818MB
[2021-04-25 21:58:07 swin_tiny_patch4_window7_224](main.py 221): INFO Train: [3/300][440/1251] eta 0:08:13 lr 0.000168 time 0.6033 (0.6089) loss 5.6586 (6.1266) grad_norm 1.6829 (inf) mem 11818MB
[2021-04-25 21:58:13 swin_tiny_patch4_window7_224](main.py 221): INFO Train: [3/300][450/1251] eta 0:08:07 lr 0.000169 time 0.5933 (0.6088) loss 5.9927 (6.1264) grad_norm 2.2850 (inf) mem 11818MB
[2021-04-25 21:58:19 swin_tiny_patch4_window7_224](main.py 221): INFO Train: [3/300][460/1251] eta 0:08:01 lr 0.000169 time 0.6342 (0.6087) loss 6.3210 (6.1273) grad_norm 1.8936 (inf) mem 11818MB
[2021-04-25 21:58:25 swin_tiny_patch4_window7_224](main.py 221): INFO Train: [3/300][470/1251] eta 0:07:55 lr 0.000170 time 0.6032 (0.6086) loss 5.5658 (6.1270) grad_norm 1.9723 (inf) mem 11818MB
[2021-04-25 21:58:31 swin_tiny_patch4_window7_224](main.py 221): INFO Train: [3/300][480/1251] eta 0:07:49 lr 0.000170 time 0.6028 (0.6085) loss 6.1317 (6.1250) grad_norm 1.9728 (inf) mem 11818MB
[2021-04-25 21:58:38 swin_tiny_patch4_window7_224](main.py 221): INFO Train: [3/300][490/1251] eta 0:07:42 lr 0.000170 time 0.6038 (0.6084) loss 6.2407 (6.1258) grad_norm 1.9203 (inf) mem 11818MB

Here is a snapshot of logs. Why grad_norm goes to infinity while training still goes on?

Best Wishes

Dear author:
when I run the train.py:
errors happened:
FileNotFoundError:CityscapesDataset:No such file or directory:'data/cityscapes/leftImg8bit/train'

I was using swin-tiny as the backbone of my model, but I encountered this error:

RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument find_unused_parameters=True to torch.nn.parallel.DistributedDataParallel; (2) making sure all forward function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's forward function. Please include the loss function and the structure of the return value of forward of your module when reporting this issue (e.g. list, dict, iterable).

I did not modify any parameter after the model was wrapped by DDP. Do you know why?

https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22kto1k.pth\

What does the models under the 1k model column in the ImageNet-22k pre-trained models mean?

Does it mean that it is pre-trained under the Imagent-22k and then trained on the 1k model?

Hi, I would like to ask if swin-t or swin-s pretrained on IamgeNet-21K is available.
If possible, please add the link for the model!

Best,
Hwanjun

        mask_windows = window_partition(img_mask, self.window_size)  # nW, window_size, window_size, 1
        mask_windows = mask_windows.view(-1, self.window_size * self.window_size)
        attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2)                                              # i think its error here !!!!!!!!!!!
        attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))

attn_mask = mask_windows.unsqueeze(2) - mask_windows.unsqueeze(1)

Can you explain how the cyclic shift changes the feature map, and what position of the tokens is masked during the calculation of the attention? As in your paper's figure , it's too abstract for me. In your code, you use torch.roll() to implemented cyclic shift, and then From Line 209 To Line 227
you calculate the mask, How the mask help to compute the attention?

Why the relative position bias in different blocks are not shared, and even the position bias for different heads in same block are different?

Hello! I have tried the default config with swin_tiny_224 to train without any pre-training model. And in the 65-th epoches, the top1 get to 71.3. I wonder whether I can get the performance like 81.5 as the paper say ? After all, it really takes too long time to train.

Hi.Could it support non-square window_size?
I think it is useful when W is much larger than H. For example, the input size is [224, 224x8]

Thanks for providing the excellent work. I am trying to reproduce the results of Swin-T model. Could you please provide a accuracy curve w.r.t epochs, which can serve as a reference for me to validate the correctness of codes and experiments.

Thanks a lot!

Hello. I find the implementation of relative PE is different from the description in the paper "Self-Attention with Relative Position Representations". So, could you introduce a little more about how you do to implement it please? Thanks a lot.

It is a interesting and may be the revolution of CNN

I used the "pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./" to install apex.

when I "from apex import amp", the error is "ModuleNotFoundError: No module named 'amp'".

My solution is "pip install -v --no-cache-dir ./", then everything is OK !!!!

The link 404,

Here is where the link sends you.

https://github.com/impiga/SwinTransformer-object-detection/blob/main/docs/get_started.md.

Hi,
thanks for releasing this awesome repo.

I wanted to know if it is possible to train on input with a certain resolution and then run inference on inputs with variable resolutions (smaller\larger than the resolution used for training).
just to clarify - I mean without resizing\croping the input to the training resolution during inference.

thanks

I am wondering if you could provide the inference code for any given images or image folder? I can run model on any given images, but it is not possible to know the class labels without downloading imagenet

The MAP of object detection. @0.5:0.95 or @0.5 ?