leaderj1001 / bottlenecktransformers Goto Github PK

Change Code In main.py line 76
model = ResNet50()
INTO
model = ResNet50(resolution=(32, 32))
can solve that problem.
Because the
self.rel_h = nn.Parameter(torch.randn([1, heads, n_dims // heads, 1, height]), requires_grad=True)
self.rel_w = nn.Parameter(torch.randn([1, heads, n_dims // heads, width, 1]), requires_grad=True)
in model.py line 25, 26
width and height number is calculated by resolution
init width and height = 14 was calculated in resolution=(224, 224)

Hello.
In your paper, you said that you use multi head self attention, and this head num is four.
But in this code, I only see 1 head in mhsa module.
And another question is why use this add to get postion encoding.
Looking forward for your reply!
Thank you!

could you please provide a pre-trained model?

When I set batch_size 1, a bug shows as "ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 512, 1, 1])". I wonder how to solve this problem.

Is this repository the official implementation?

Current Learning Rate: [0.030934962553363768]
[Epoch: 251], Loss: 0.085, Acc: 97.030, Correct 12544.0 / Total 12928.0
[Epoch: 251], Loss: 0.084, Acc: 97.104, Correct 24983.0 / Total 25728.0
[Epoch: 251], Loss: 0.081, Acc: 97.194, Correct 37447.0 / Total 38528.0
[Epoch: 251], Acc: 87.820
Current Learning Rate: [0.030032595786498105]
[Epoch: 252], Loss: 0.075, Acc: 97.401, Correct 12592.0 / Total 12928.0
[Epoch: 252], Loss: 0.077, Acc: 97.365, Correct 25050.0 / Total 25728.0
[Epoch: 252], Loss: 0.077, Acc: 97.366, Correct 37513.0 / Total 38528.0
[Epoch: 252], Acc: 86.980
Current Learning Rate: [0.029137946110005482]
[Epoch: 253], Loss: 0.064, Acc: 97.857, Correct 12651.0 / Total 12928.0
[Epoch: 253], Loss: 0.066, Acc: 97.777, Correct 25156.0 / Total 25728.0
[Epoch: 253], Loss: 0.070, Acc: 97.610, Correct 37607.0 / Total 38528.0
[Epoch: 253], Acc: 87.350
Current Learning Rate: [0.02825135842836657]
[Epoch: 254], Loss: 0.073, Acc: 97.563, Correct 12613.0 / Total 12928.0
[Epoch: 254], Loss: 0.074, Acc: 97.477, Correct 25079.0 / Total 25728.0
[Epoch: 254], Loss: 0.070, Acc: 97.560, Correct 37588.0 / Total 38528.0
[Epoch: 254], Acc: 86.640
Current Learning Rate: [0.02737317453800964]
[Epoch: 255], Loss: 0.063, Acc: 97.826, Correct 12647.0 / Total 12928.0
[Epoch: 255], Loss: 0.061, Acc: 97.889, Correct 25185.0 / Total 25728.0
[Epoch: 255], Loss: 0.065, Acc: 97.742, Correct 37658.0 / Total 38528.0
[Epoch: 255], Acc: 87.650
Current Learning Rate: [0.026503732995541415]
[Epoch: 256], Loss: 0.064, Acc: 97.803, Correct 12644.0 / Total 12928.0
[Epoch: 256], Loss: 0.064, Acc: 97.831, Correct 25170.0 / Total 25728.0
[Epoch: 256], Loss: 0.063, Acc: 97.861, Correct 37704.0 / Total 38528.0
[Epoch: 256], Acc: 87.970
Current Learning Rate: [0.025643368987227095]
[Epoch: 257], Loss: 0.066, Acc: 97.788, Correct 12642.0 / Total 12928.0
[Epoch: 257], Loss: 0.066, Acc: 97.804, Correct 25163.0 / Total 25728.0
[Epoch: 257], Loss: 0.065, Acc: 97.835, Correct 37694.0 / Total 38528.0
[Epoch: 257], Acc: 87.380
Current Learning Rate: [0.02479241419976968]
[Epoch: 258], Loss: 0.055, Acc: 98.198, Correct 12695.0 / Total 12928.0
[Epoch: 258], Loss: 0.053, Acc: 98.204, Correct 25266.0 / Total 25728.0
[Epoch: 258], Loss: 0.055, Acc: 98.183, Correct 37828.0 / Total 38528.0
[Epoch: 258], Acc: 87.930
Current Learning Rate: [0.023951196692438358]
[Epoch: 259], Loss: 0.049, Acc: 98.213, Correct 12697.0 / Total 12928.0
[Epoch: 259], Loss: 0.051, Acc: 98.231, Correct 25273.0 / Total 25728.0
[Epoch: 259], Loss: 0.054, Acc: 98.108, Correct 37799.0 / Total 38528.0
[Epoch: 259], Acc: 87.880
Current Learning Rate: [0.023120040770595558]
[Epoch: 260], Loss: 0.046, Acc: 98.430, Correct 12725.0 / Total 12928.0
[Epoch: 260], Loss: 0.050, Acc: 98.340, Correct 25301.0 / Total 25728.0
[Epoch: 260], Loss: 0.051, Acc: 98.271, Correct 37862.0 / Total 38528.0
[Epoch: 260], Acc: 87.700
Current Learning Rate: [0.022299266860670866]
[Epoch: 261], Loss: 0.052, Acc: 98.229, Correct 12699.0 / Total 12928.0
[Epoch: 261], Loss: 0.051, Acc: 98.278, Correct 25285.0 / Total 25728.0
[Epoch: 261], Loss: 0.050, Acc: 98.279, Correct 37865.0 / Total 38528.0
[Epoch: 261], Acc: 88.190
Current Learning Rate: [0.021489191386630774]
[Epoch: 262], Loss: 0.046, Acc: 98.391, Correct 12720.0 / Total 12928.0
[Epoch: 262], Loss: 0.046, Acc: 98.395, Correct 25315.0 / Total 25728.0
[Epoch: 262], Loss: 0.047, Acc: 98.378, Correct 37903.0 / Total 38528.0
[Epoch: 262], Acc: 87.440
Current Learning Rate: [0.020690126647990973]
[Epoch: 263], Loss: 0.041, Acc: 98.577, Correct 12744.0 / Total 12928.0
[Epoch: 263], Loss: 0.043, Acc: 98.496, Correct 25341.0 / Total 25728.0
[Epoch: 263], Loss: 0.045, Acc: 98.435, Correct 37925.0 / Total 38528.0
[Epoch: 263], Acc: 87.740
Current Learning Rate: [0.019902380699419107]
[Epoch: 264], Loss: 0.041, Acc: 98.700, Correct 12760.0 / Total 12928.0
[Epoch: 264], Loss: 0.040, Acc: 98.706, Correct 25395.0 / Total 25728.0
[Epoch: 264], Loss: 0.040, Acc: 98.679, Correct 38019.0 / Total 38528.0
[Epoch: 264], Acc: 87.720
Current Learning Rate: [0.019126257231973805]
[Epoch: 265], Loss: 0.037, Acc: 98.824, Correct 12776.0 / Total 12928.0
[Epoch: 265], Loss: 0.038, Acc: 98.764, Correct 25410.0 / Total 25728.0
[Epoch: 265], Loss: 0.039, Acc: 98.713, Correct 38032.0 / Total 38528.0
[Epoch: 265], Acc: 88.020
Current Learning Rate: [0.018362055456025896]
[Epoch: 266], Loss: 0.034, Acc: 98.971, Correct 12795.0 / Total 12928.0
[Epoch: 266], Loss: 0.034, Acc: 98.865, Correct 25436.0 / Total 25728.0
[Epoch: 266], Loss: 0.038, Acc: 98.731, Correct 38039.0 / Total 38528.0
[Epoch: 266], Acc: 88.280
Current Learning Rate: [0.01761006998590733]
[Epoch: 267], Loss: 0.028, Acc: 99.033, Correct 12803.0 / Total 12928.0
[Epoch: 267], Loss: 0.030, Acc: 98.978, Correct 25465.0 / Total 25728.0
[Epoch: 267], Loss: 0.031, Acc: 98.936, Correct 38118.0 / Total 38528.0
[Epoch: 267], Acc: 88.010
Current Learning Rate: [0.016870590726331475]
[Epoch: 268], Loss: 0.030, Acc: 99.033, Correct 12803.0 / Total 12928.0
[Epoch: 268], Loss: 0.031, Acc: 98.989, Correct 25468.0 / Total 25728.0
[Epoch: 268], Loss: 0.032, Acc: 98.928, Correct 38115.0 / Total 38528.0
[Epoch: 268], Acc: 88.690
Current Learning Rate: [0.016143902760629568]
[Epoch: 269], Loss: 0.028, Acc: 99.087, Correct 12810.0 / Total 12928.0
[Epoch: 269], Loss: 0.027, Acc: 99.122, Correct 25502.0 / Total 25728.0
[Epoch: 269], Loss: 0.028, Acc: 99.110, Correct 38185.0 / Total 38528.0
[Epoch: 269], Acc: 88.200
Current Learning Rate: [0.015430286240845494]
[Epoch: 270], Loss: 0.028, Acc: 99.010, Correct 12800.0 / Total 12928.0
[Epoch: 270], Loss: 0.025, Acc: 99.090, Correct 25494.0 / Total 25728.0
[Epoch: 270], Loss: 0.026, Acc: 99.079, Correct 38173.0 / Total 38528.0
[Epoch: 270], Acc: 88.660
Current Learning Rate: [0.014730016279731955]
[Epoch: 271], Loss: 0.028, Acc: 99.103, Correct 12812.0 / Total 12928.0
[Epoch: 271], Loss: 0.025, Acc: 99.172, Correct 25515.0 / Total 25728.0
[Epoch: 271], Loss: 0.026, Acc: 99.164, Correct 38206.0 / Total 38528.0
[Epoch: 271], Acc: 88.780
Best Model Saving...
Current Learning Rate: [0.014043362844689204]
[Epoch: 272], Loss: 0.023, Acc: 99.273, Correct 12834.0 / Total 12928.0
[Epoch: 272], Loss: 0.026, Acc: 99.122, Correct 25502.0 / Total 25728.0
[Epoch: 272], Loss: 0.025, Acc: 99.167, Correct 38207.0 / Total 38528.0
[Epoch: 272], Acc: 88.590
Current Learning Rate: [0.0133705906536875]
[Epoch: 273], Loss: 0.024, Acc: 99.188, Correct 12823.0 / Total 12928.0
[Epoch: 273], Loss: 0.020, Acc: 99.386, Correct 25570.0 / Total 25728.0
[Epoch: 273], Loss: 0.019, Acc: 99.403, Correct 38298.0 / Total 38528.0
[Epoch: 273], Acc: 88.260
Current Learning Rate: [0.0127119590732133]
[Epoch: 274], Loss: 0.023, Acc: 99.157, Correct 12819.0 / Total 12928.0
[Epoch: 274], Loss: 0.021, Acc: 99.265, Correct 25539.0 / Total 25728.0
[Epoch: 274], Loss: 0.020, Acc: 99.299, Correct 38258.0 / Total 38528.0
[Epoch: 274], Acc: 88.300
Current Learning Rate: [0.012067722018278455]
[Epoch: 275], Loss: 0.018, Acc: 99.373, Correct 12847.0 / Total 12928.0
[Epoch: 275], Loss: 0.018, Acc: 99.378, Correct 25568.0 / Total 25728.0
[Epoch: 275], Loss: 0.018, Acc: 99.377, Correct 38288.0 / Total 38528.0
[Epoch: 275], Acc: 88.700
Current Learning Rate: [0.011438127854531303]
[Epoch: 276], Loss: 0.015, Acc: 99.590, Correct 12875.0 / Total 12928.0
[Epoch: 276], Loss: 0.015, Acc: 99.569, Correct 25617.0 / Total 25728.0
[Epoch: 276], Loss: 0.015, Acc: 99.564, Correct 38360.0 / Total 38528.0
[Epoch: 276], Acc: 88.500
Current Learning Rate: [0.010823419302506784]
[Epoch: 277], Loss: 0.017, Acc: 99.404, Correct 12851.0 / Total 12928.0
[Epoch: 277], Loss: 0.016, Acc: 99.456, Correct 25588.0 / Total 25728.0
[Epoch: 277], Loss: 0.016, Acc: 99.452, Correct 38317.0 / Total 38528.0
[Epoch: 277], Acc: 88.690
Current Learning Rate: [0.010223833344053286]
[Epoch: 278], Loss: 0.014, Acc: 99.520, Correct 12866.0 / Total 12928.0
[Epoch: 278], Loss: 0.015, Acc: 99.518, Correct 25604.0 / Total 25728.0
[Epoch: 278], Loss: 0.014, Acc: 99.525, Correct 38345.0 / Total 38528.0
[Epoch: 278], Acc: 89.290
Best Model Saving...
Current Learning Rate: [0.00963960113097138]
[Epoch: 279], Loss: 0.012, Acc: 99.629, Correct 12880.0 / Total 12928.0
[Epoch: 279], Loss: 0.013, Acc: 99.600, Correct 25625.0 / Total 25728.0
[Epoch: 279], Loss: 0.013, Acc: 99.624, Correct 38383.0 / Total 38528.0
[Epoch: 279], Acc: 89.070
Current Learning Rate: [0.009070947895900596]
[Epoch: 280], Loss: 0.011, Acc: 99.675, Correct 12886.0 / Total 12928.0
[Epoch: 280], Loss: 0.011, Acc: 99.666, Correct 25642.0 / Total 25728.0
[Epoch: 280], Loss: 0.011, Acc: 99.678, Correct 38404.0 / Total 38528.0
[Epoch: 280], Acc: 89.250
Current Learning Rate: [0.008518092865487875]
[Epoch: 281], Loss: 0.011, Acc: 99.667, Correct 12885.0 / Total 12928.0
[Epoch: 281], Loss: 0.011, Acc: 99.708, Correct 25653.0 / Total 25728.0
[Epoch: 281], Loss: 0.011, Acc: 99.655, Correct 38395.0 / Total 38528.0
[Epoch: 281], Acc: 89.110
Current Learning Rate: [0.007981249175871482]
[Epoch: 282], Loss: 0.011, Acc: 99.652, Correct 12883.0 / Total 12928.0
[Epoch: 282], Loss: 0.011, Acc: 99.670, Correct 25643.0 / Total 25728.0
[Epoch: 282], Loss: 0.010, Acc: 99.689, Correct 38408.0 / Total 38528.0
[Epoch: 282], Acc: 89.260
Current Learning Rate: [0.007460623790513096]
[Epoch: 283], Loss: 0.008, Acc: 99.737, Correct 12894.0 / Total 12928.0
[Epoch: 283], Loss: 0.009, Acc: 99.740, Correct 25661.0 / Total 25728.0
[Epoch: 283], Loss: 0.009, Acc: 99.725, Correct 38422.0 / Total 38528.0
[Epoch: 283], Acc: 89.420
Best Model Saving...
Current Learning Rate: [0.006956417420409298]
[Epoch: 284], Loss: 0.010, Acc: 99.683, Correct 12887.0 / Total 12928.0
[Epoch: 284], Loss: 0.010, Acc: 99.697, Correct 25650.0 / Total 25728.0
[Epoch: 284], Loss: 0.010, Acc: 99.689, Correct 38408.0 / Total 38528.0
[Epoch: 284], Acc: 89.370
Current Learning Rate: [0.0064688244467137924]
[Epoch: 285], Loss: 0.006, Acc: 99.838, Correct 12907.0 / Total 12928.0
[Epoch: 285], Loss: 0.007, Acc: 99.802, Correct 25677.0 / Total 25728.0
[Epoch: 285], Loss: 0.007, Acc: 99.785, Correct 38445.0 / Total 38528.0
[Epoch: 285], Acc: 89.180
Current Learning Rate: [0.005998032845799671]
[Epoch: 286], Loss: 0.006, Acc: 99.845, Correct 12908.0 / Total 12928.0
[Epoch: 286], Loss: 0.006, Acc: 99.817, Correct 25681.0 / Total 25728.0
[Epoch: 286], Loss: 0.006, Acc: 99.795, Correct 38449.0 / Total 38528.0
[Epoch: 286], Acc: 89.140
Current Learning Rate: [0.0055442241167910295]
[Epoch: 287], Loss: 0.008, Acc: 99.737, Correct 12894.0 / Total 12928.0
[Epoch: 287], Loss: 0.007, Acc: 99.759, Correct 25666.0 / Total 25728.0
[Epoch: 287], Loss: 0.007, Acc: 99.779, Correct 38443.0 / Total 38528.0
[Epoch: 287], Acc: 89.310
Current Learning Rate: [0.005107573211591536]
[Epoch: 288], Loss: 0.007, Acc: 99.729, Correct 12893.0 / Total 12928.0
[Epoch: 288], Loss: 0.007, Acc: 99.759, Correct 25666.0 / Total 25728.0
[Epoch: 288], Loss: 0.007, Acc: 99.756, Correct 38434.0 / Total 38528.0
[Epoch: 288], Acc: 89.330
Current Learning Rate: [0.004688248467437186]
[Epoch: 289], Loss: 0.007, Acc: 99.783, Correct 12900.0 / Total 12928.0
[Epoch: 289], Loss: 0.007, Acc: 99.817, Correct 25681.0 / Total 25728.0
[Epoch: 289], Loss: 0.006, Acc: 99.834, Correct 38464.0 / Total 38528.0
[Epoch: 289], Acc: 89.370
Current Learning Rate: [0.004286411541999064]
[Epoch: 290], Loss: 0.006, Acc: 99.830, Correct 12906.0 / Total 12928.0
[Epoch: 290], Loss: 0.005, Acc: 99.841, Correct 25687.0 / Total 25728.0
[Epoch: 290], Loss: 0.005, Acc: 99.839, Correct 38466.0 / Total 38528.0
[Epoch: 290], Acc: 89.200
Current Learning Rate: [0.0039022173510612273]
[Epoch: 291], Loss: 0.006, Acc: 99.845, Correct 12908.0 / Total 12928.0
[Epoch: 291], Loss: 0.006, Acc: 99.829, Correct 25684.0 / Total 25728.0
[Epoch: 291], Loss: 0.005, Acc: 99.834, Correct 38464.0 / Total 38528.0
[Epoch: 291], Acc: 89.350
Current Learning Rate: [0.003535814008797773]
[Epoch: 292], Loss: 0.004, Acc: 99.876, Correct 12912.0 / Total 12928.0
[Epoch: 292], Loss: 0.005, Acc: 99.837, Correct 25686.0 / Total 25728.0
[Epoch: 292], Loss: 0.005, Acc: 99.826, Correct 38461.0 / Total 38528.0
[Epoch: 292], Acc: 89.340
Current Learning Rate: [0.003187342770671916]
[Epoch: 293], Loss: 0.004, Acc: 99.930, Correct 12919.0 / Total 12928.0
[Epoch: 293], Loss: 0.004, Acc: 99.880, Correct 25697.0 / Total 25728.0
[Epoch: 293], Loss: 0.004, Acc: 99.875, Correct 38480.0 / Total 38528.0
[Epoch: 293], Acc: 89.560
Best Model Saving...
Current Learning Rate: [0.002856937978979447]
[Epoch: 294], Loss: 0.004, Acc: 99.884, Correct 12913.0 / Total 12928.0
[Epoch: 294], Loss: 0.004, Acc: 99.883, Correct 25698.0 / Total 25728.0
[Epoch: 294], Loss: 0.004, Acc: 99.873, Correct 38479.0 / Total 38528.0
[Epoch: 294], Acc: 89.490
Current Learning Rate: [0.002544727011057081]
[Epoch: 295], Loss: 0.004, Acc: 99.892, Correct 12914.0 / Total 12928.0
[Epoch: 295], Loss: 0.004, Acc: 99.883, Correct 25698.0 / Total 25728.0
[Epoch: 295], Loss: 0.004, Acc: 99.881, Correct 38482.0 / Total 38528.0
[Epoch: 295], Acc: 89.650
Best Model Saving...
Current Learning Rate: [0.002250830230176169]
[Epoch: 296], Loss: 0.004, Acc: 99.853, Correct 12909.0 / Total 12928.0
[Epoch: 296], Loss: 0.004, Acc: 99.887, Correct 25699.0 / Total 25728.0
[Epoch: 296], Loss: 0.004, Acc: 99.881, Correct 38482.0 / Total 38528.0
[Epoch: 296], Acc: 89.450
Current Learning Rate: [0.001975360939140324]
[Epoch: 297], Loss: 0.003, Acc: 99.899, Correct 12915.0 / Total 12928.0
[Epoch: 297], Loss: 0.003, Acc: 99.903, Correct 25703.0 / Total 25728.0
[Epoch: 297], Loss: 0.003, Acc: 99.901, Correct 38490.0 / Total 38528.0
[Epoch: 297], Acc: 89.440
Current Learning Rate: [0.0017184253366050195]
[Epoch: 298], Loss: 0.004, Acc: 99.884, Correct 12913.0 / Total 12928.0
[Epoch: 298], Loss: 0.004, Acc: 99.868, Correct 25694.0 / Total 25728.0
[Epoch: 298], Loss: 0.004, Acc: 99.868, Correct 38477.0 / Total 38528.0
[Epoch: 298], Acc: 89.690
Best Model Saving...
Current Learning Rate: [0.001480122476136056]
[Epoch: 299], Loss: 0.003, Acc: 99.915, Correct 12917.0 / Total 12928.0
[Epoch: 299], Loss: 0.003, Acc: 99.911, Correct 25705.0 / Total 25728.0
[Epoch: 299], Loss: 0.004, Acc: 99.899, Correct 38489.0 / Total 38528.0
[Epoch: 299], Acc: 89.680
Current Learning Rate: [0.0012605442280224245]
[Epoch: 300], Loss: 0.003, Acc: 99.930, Correct 12919.0 / Total 12928.0
[Epoch: 300], Loss: 0.003, Acc: 99.934, Correct 25711.0 / Total 25728.0
[Epoch: 300], Loss: 0.003, Acc: 99.922, Correct 38498.0 / Total 38528.0
[Epoch: 300], Acc: 89.660
Current Learning Rate: [0.00105977524385864]
[Epoch: 301], Loss: 0.002, Acc: 99.954, Correct 12922.0 / Total 12928.0
[Epoch: 301], Loss: 0.002, Acc: 99.934, Correct 25711.0 / Total 25728.0
[Epoch: 301], Loss: 0.003, Acc: 99.907, Correct 38492.0 / Total 38528.0
[Epoch: 301], Acc: 89.690
Current Learning Rate: [0.0008778929239099148]
[Epoch: 302], Loss: 0.003, Acc: 99.907, Correct 12916.0 / Total 12928.0
[Epoch: 302], Loss: 0.003, Acc: 99.918, Correct 25707.0 / Total 25728.0
[Epoch: 302], Loss: 0.003, Acc: 99.920, Correct 38497.0 / Total 38528.0
[Epoch: 302], Acc: 89.660
Current Learning Rate: [0.000714967387272874]
[Epoch: 303], Loss: 0.003, Acc: 99.884, Correct 12913.0 / Total 12928.0
[Epoch: 303], Loss: 0.003, Acc: 99.899, Correct 25702.0 / Total 25728.0
[Epoch: 303], Loss: 0.003, Acc: 99.904, Correct 38491.0 / Total 38528.0
[Epoch: 303], Acc: 89.600
Current Learning Rate: [0.0005710614448433164]
[Epoch: 304], Loss: 0.004, Acc: 99.899, Correct 12915.0 / Total 12928.0
[Epoch: 304], Loss: 0.003, Acc: 99.911, Correct 25705.0 / Total 25728.0
[Epoch: 304], Loss: 0.003, Acc: 99.914, Correct 38495.0 / Total 38528.0
[Epoch: 304], Acc: 89.660
Current Learning Rate: [0.0004462305751014317]
[Epoch: 305], Loss: 0.003, Acc: 99.892, Correct 12914.0 / Total 12928.0
[Epoch: 305], Loss: 0.003, Acc: 99.914, Correct 25706.0 / Total 25728.0
[Epoch: 305], Loss: 0.003, Acc: 99.927, Correct 38500.0 / Total 38528.0
[Epoch: 305], Acc: 89.630
Current Learning Rate: [0.00034052290272376895]
[Epoch: 306], Loss: 0.002, Acc: 99.954, Correct 12922.0 / Total 12928.0
[Epoch: 306], Loss: 0.002, Acc: 99.934, Correct 25711.0 / Total 25728.0
[Epoch: 306], Loss: 0.003, Acc: 99.912, Correct 38494.0 / Total 38528.0
[Epoch: 306], Acc: 89.620
Current Learning Rate: [0.0002539791800302582]
[Epoch: 307], Loss: 0.002, Acc: 99.946, Correct 12921.0 / Total 12928.0
[Epoch: 307], Loss: 0.002, Acc: 99.949, Correct 25715.0 / Total 25728.0
[Epoch: 307], Loss: 0.002, Acc: 99.943, Correct 38506.0 / Total 38528.0
[Epoch: 307], Acc: 89.630
Current Learning Rate: [0.00018663277127344463]
[Epoch: 308], Loss: 0.003, Acc: 99.884, Correct 12913.0 / Total 12928.0
[Epoch: 308], Loss: 0.003, Acc: 99.883, Correct 25698.0 / Total 25728.0
[Epoch: 308], Loss: 0.003, Acc: 99.891, Correct 38486.0 / Total 38528.0
[Epoch: 308], Acc: 89.550
Current Learning Rate: [0.0001385096397758911]
[Epoch: 309], Loss: 0.003, Acc: 99.938, Correct 12920.0 / Total 12928.0
[Epoch: 309], Loss: 0.003, Acc: 99.930, Correct 25710.0 / Total 25728.0
[Epoch: 309], Loss: 0.003, Acc: 99.920, Correct 38497.0 / Total 38528.0
[Epoch: 309], Acc: 89.700
Best Model Saving...
Current Learning Rate: [0.00010962833792086233]
[Epoch: 310], Loss: 0.002, Acc: 99.961, Correct 12923.0 / Total 12928.0
[Epoch: 310], Loss: 0.002, Acc: 99.953, Correct 25716.0 / Total 25728.0
[Epoch: 310], Loss: 0.003, Acc: 99.945, Correct 38507.0 / Total 38528.0
[Epoch: 310], Acc: 89.670
Current Learning Rate: [0.1]

Is there a pre-trained model of botnet on imagenet?

I trained for about 10k steps, and the values of v and attention matrix became Nan, and the loss also became Nan. I would like to ask how to solve this problem？ Thank you!

An error occurs when I load the untrained model:RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

RuntimeError: The size of tensor a (4) must match the size of tensor b (196) at non-singleton dimension 2

I'm using your code. And I didn't make any changes

Hi, I just change your code with ResNet50(num_classes=10, resolution=(224, 224)), which end with a lower accuaracy of 90.15%. do you have other changes to get the 9511%?

out = torch.matmul(v, attention.permute(0, 1, 3, 2))
out = torch.matmul(v, attention) is ok?

when I first step to run the model.py, An error occurred:

The size of tensor a (196) must match the size of tensor b (256) at non-singleton dimension 1

It seems that MHSA only has one head in the released code. But in the paper, 4 heads are used in MHSA. Is it a simplification for CIFAR dataset?

Hello, I would like to ask whether H and W of input vectors must be fixed when using your Resnet50 and MHSA modules?Is it possible to automatically adjust the image size according to each batsize?？

File "/home/sc/carl_test/BottleneckTransformers-main/model.py", line 43, in forward
energy = content_content + content_position
RuntimeError: The size of tensor a (4) must match the size of tensor b (196) at non-singleton dimension 1

Hi ! when run the model.py i meet this error

Hello, the height of the picture is 256 and the width is 128. It will report an error.What should I change in the code?

when i run main function where is model.py i get an error,for example
"
energy = content_content + content_position
RuntimeError: The size of tensor a (196) must match the size of tensor b (256) at non-singleton dimension 1
"