Comments (16)
lucidrains
commented on May 5, 2024
5
thanks! I'll dig into it later this week. GANs are always full of issues like these, and this is precisely why no one likes to use them these days with DDPMs as an alternative
from gigagan-pytorch.
sansavision
commented on May 5, 2024
1
@lucidrains Didn't notice the 0.2.12 update sorry. This is on 0.2.11, has this issue been fixed on 0.2.12?
from gigagan-pytorch.
sansavision
commented on May 5, 2024
1
Hi again, @lucidrains ! With the new version, 0.2.12, it failed after 10 hours of training. It seems the be the same error as before. Here's the complete log:
A100 GPU detected, using flash attention if input tensor is on cuda
Generator: 43.71M
Discriminator: 30.74M
Total images: 200000
0%| | 1/1000000 [00:00<?, ?it/s]G: -5.47 | MSG: 0.44 | VG: 0.00 | D: 6.56 | MSD: 3.88 | VD: 0.00 | GP: 0.00 | SSL: 3.60 | CL: 0.00 | MAL: 0.00
0%| | 20/1000000 [08:30<7288:42:34, 26.24s/it]G: 96.02 | MSG: -14.98 | VG: 0.00 | D: 0.00 | MSD: 16.26 | VD: 0.00 | GP: 27032.36 | SSL: 0.26 | CL: 0.00 | MAL: 0.00
0%| | 40/1000000 [17:25<7203:12:38, 25.93s/it]G: 1249.11 | MSG: 8.87 | VG: 0.00 | D: 0.00 | MSD: 15.67 | VD: 0.00 | GP: 17236.77 | SSL: 0.64 | CL: 0.00 | MAL: 0.00
0%| | 60/1000000 [26:20<7285:42:15, 26.23s/it]G: 1.41 | MSG: -0.41 | VG: 0.00 | D: 3.07 | MSD: 2.52 | VD: 0.00 | GP: 3316.55 | SSL: 0.09 | CL: 0.00 | MAL: 0.00
0%| | 80/1000000 [35:15<7301:19:51, 26.29s/it]G: 363.97 | MSG: 3.43 | VG: 0.00 | D: 0.02 | MSD: 1.39 | VD: 0.00 | GP: 1186.02 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%| | 100/1000000 [44:10<7349:03:38, 26.46s/it]G: 21.10 | MSG: 0.58 | VG: 0.00 | D: 10.84 | MSD: 1.60 | VD: 0.00 | GP: 891.27 | SSL: 0.17 | CL: 0.00 | MAL: 0.00
0%| | 120/1000000 [53:07<7222:42:49, 26.00s/it]G: 24.29 | MSG: -0.45 | VG: 0.00 | D: 0.49 | MSD: 1.69 | VD: 0.00 | GP: 973.06 | SSL: 0.15 | CL: 0.00 | MAL: 0.00
0%| | 140/1000000 [1:02:00<7234:42:26, 26.05s/it]G: 75.15 | MSG: -0.35 | VG: 0.00 | D: 0.00 | MSD: 2.37 | VD: 0.00 | GP: 519.11 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%| | 160/1000000 [1:10:54<7270:03:33, 26.18s/it]G: 6.35 | MSG: -0.35 | VG: 0.00 | D: 2.99 | MSD: 2.14 | VD: 0.00 | GP: 552.92 | SSL: 0.09 | CL: 0.00 | MAL: 0.00
0%| | 180/1000000 [1:19:50<7259:04:35, 26.14s/it]G: 1.06 | MSG: -0.12 | VG: 0.00 | D: 1.62 | MSD: 2.10 | VD: 0.00 | GP: 365.89 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%| | 200/1000000 [1:28:48<7315:27:46, 26.34s/it]G: 1.85 | MSG: -0.08 | VG: 0.00 | D: 2.85 | MSD: 2.04 | VD: 0.00 | GP: 264.83 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%| | 220/1000000 [1:37:49<7265:18:19, 26.16s/it]G: 150.54 | MSG: -0.32 | VG: 0.00 | D: 0.00 | MSD: 1.84 | VD: 0.00 | GP: 211.16 | SSL: 0.17 | CL: 0.00 | MAL: 0.00
0%| | 240/1000000 [1:46:45<7298:23:35, 26.28s/it]G: 16.83 | MSG: -0.19 | VG: 0.00 | D: 3.22 | MSD: 2.06 | VD: 0.00 | GP: 99.23 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%| | 260/1000000 [1:55:38<7255:20:52, 26.13s/it]G: -0.55 | MSG: -0.13 | VG: 0.00 | D: 1.21 | MSD: 1.99 | VD: 0.00 | GP: 78.56 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%| | 280/1000000 [2:04:32<7234:03:11, 26.05s/it]G: 1.54 | MSG: -0.17 | VG: 0.00 | D: 0.27 | MSD: 1.96 | VD: 0.00 | GP: 67.88 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%| 0%| 0%| 0%| 0%| 0%| 0%| 0%| 0%| 0%| 0%| 0%| 0%| 0%| | 300/1000000 [2:13:30<7302:07:06, 26.30s/it]G: 5.64 | MSG: -0.22 | VG: 0.00 | D: 0.03 | MSD: 1.96 | VD: 0.00 | GP: 41.95 | SSL: 0.24 | CL: 0.00 | MAL: 0.00
0%| 0%| 0%| 0%| 0%| 0%| 0%| 0%| 0%| 0%| 0%| 0%| | 312/100000 0%| | 320/1000000 [2:22:34<7311:15:38, 26.33s/it]G: 0.91 | MSG: -0.20 | VG: 0.00 | D: 2.99 | MSD: 1.95 | VD: 0.00 | GP: 43.50 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%| | 340/1000000 [2:31:31<7315:23:32, 26.34s/it]G: 2.16 | MSG: -0.19 | VG: 0.00 | D: 4.30 | MSD: 2.00 | VD: 0.00 | GP: 20.14 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%| | 360/1000000 [2:40:28<7296:09:26, 26.28s/it]G: 16.71 | MSG: -0.20 | VG: 0.00 | D: 0.82 | MSD: 1.92 | VD: 0.00 | GP: 18.90 | SSL: 0.12 | CL: 0.00 | MAL: 0.00
0%| | 380/1000000 [2:49:24<7223:40:18, 26.02s/it]G: 20.96 | MSG: -0.13 | VG: 0.00 | D: 0.53 | MSD: 1.81 | VD: 0.00 | GP: 12.01 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%| | 400/1000000 [2:58:20<7301:46:40, 26.30s/it]G: 32.78 | MSG: -0.14 | VG: 0.00 | D: 1.07 | MSD: 1.92 | VD: 0.00 | GP: 6.71 | SSL: 0.10 | CL: 0.00 | MAL: 0.00
0%| | 420/1000000 [3:07:22<7216:05:46, 25.99s/it]G: 30.05 | MSG: -0.20 | VG: 0.00 | D: 0.48 | MSD: 1.98 | VD: 0.00 | GP: 6.36 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%| | 440/1000000 [3:16:20<7255:33:38, 26.13s/it]G: 0.72 | MSG: -0.21 | VG: 0.00 | D: 1.31 | MSD: 2.00 | VD: 0.00 | GP: 10.62 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 460/1000000 [3:25:14<7261:23:57, 26.15s/it]G: 2.50 | MSG: -0.21 | VG: 0.00 | D: 0.17 | MSD: 2.00 | VD: 0.00 | GP: 3.05 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 480/1000000 [3:34:09<7221:32:49, 26.01s/it]G: 21.48 | MSG: -0.22 | VG: 0.00 | D: 2.93 | MSD: 2.06 | VD: 0.00 | GP: 2.66 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%|▏ | 500/1000000 [3:43:07<7314:22:33, 26.34s/it]G: 10.03 | MSG: -0.20 | VG: 0.00 | D: 0.77 | MSD: 2.04 | VD: 0.00 | GP: 2.47 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 520/1000000 [3:52:12<7315:53:07, 26.35s/it]G: 10.70 | MSG: -0.20 | VG: 0.00 | D: 0.03 | MSD: 2.04 | VD: 0.00 | GP: 0.63 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%|▏ | 540/1000000 [4:01:09<7366:30:13, 26.53s/it]G: 33.12 | MSG: -0.20 | VG: 0.00 | D: 0.00 | MSD: 1.91 | VD: 0.00 | GP: 1.13 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 560/1000000 [4:09:50<7076:34:51, 25.49s/it]G: 0.71 | MSG: -0.22 | VG: 0.00 | D: 1.49 | MSD: 2.00 | VD: 0.00 | GP: 0.54 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 580/1000000 [4:18:29<7049:34:54, 25.39s/it]G: 21.83 | MSG: -0.26 | VG: 0.00 | D: 0.14 | MSD: 2.06 | VD: 0.00 | GP: 0.37 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 600/1000000 [4:27:11<7076:39:25, 25.49s/it]G: 6.63 | MSG: -0.27 | VG: 0.00 | D: 0.11 | MSD: 2.01 | VD: 0.00 | GP: 1.10 | SSL: 0.09 | CL: 0.00 | MAL: 0.00
0%|▏ | 620/1000000 [4:35:58<7102:45:51, 25.59s/it]G: 0.40 | MSG: -0.24 | VG: 0.00 | D: 2.00 | MSD: 2.00 | VD: 0.00 | GP: 0.32 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%|▏ | 640/1000000 [4:44:38<7025:04:29, 25.31s/it]G: 1.29 | MSG: -0.24 | VG: 0.00 | D: 1.23 | MSD: 2.00 | VD: 0.00 | GP: 0.24 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%|▏ | 660/1000000 [4:53:18<7003:13:13, 25.23s/it]G: 3.68 | MSG: -0.19 | VG: 0.00 | D: 0.00 | MSD: 2.11 | VD: 0.00 | GP: 0.91 | SSL: 0.21 | CL: 0.00 | MAL: 0.00
0%|▏ | 680/1000000 [5:01:56<7056:04:48, 25.42s/it]G: 17.65 | MSG: 0.06 | VG: 0.00 | D: 0.00 | MSD: 1.91 | VD: 0.00 | GP: 0.17 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 700/1000000 [5:10:38<7102:50:10, 25.59s/it]G: 1.21 | MSG: -0.25 | VG: 0.00 | D: 0.81 | MSD: 2.00 | VD: 0.00 | GP: 0.18 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%|▏ | 720/1000000 [5:19:23<7032:46:09, 25.34s/it]G: 0.78 | MSG: -0.27 | VG: 0.00 | D: 1.30 | MSD: 1.99 | VD: 0.00 | GP: 1.13 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 740/1000000 [5:28:06<7125:11:09, 25.67s/it]G: 3.68 | MSG: -0.26 | VG: 0.00 | D: 0.96 | MSD: 2.01 | VD: 0.00 | GP: 1.28 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 760/1000000 [5:36:45<6999:11:30, 25.22s/it]G: 3.94 | MSG: -0.27 | VG: 0.00 | D: 0.02 | MSD: 1.99 | VD: 0.00 | GP: 1.87 | SSL: 0.11 | CL: 0.00 | MAL: 0.00
0%|▏ | 780/1000000 [5:45:25<7096:51:23, 25.57s/it]G: 0.91 | MSG: -0.27 | VG: 0.00 | D: 1.40 | MSD: 1.99 | VD: 0.00 | GP: 0.09 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 800/1000000 [5:54:07<7064:01:01, 25.45s/it]G: 1.74 | MSG: -0.31 | VG: 0.00 | D: 1.38 | MSD: 2.02 | VD: 0.00 | GP: 0.20 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%|▏ | 820/1000000 [6:02:56<7117:04:00, 25.64s/it]G: 1.12 | MSG: -0.31 | VG: 0.00 | D: 1.63 | MSD: 1.99 | VD: 0.00 | GP: 0.35 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 840/1000000 [6:11:37<7038:37:51, 25.36s/it]G: 1.18 | MSG: -0.29 | VG: 0.00 | D: 0.66 | MSD: 1.99 | VD: 0.00 | GP: 0.07 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 860/1000000 [6:20:13<7006:16:02, 25.24s/it]G: 2.80 | MSG: -0.27 | VG: 0.00 | D: 0.43 | MSD: 2.00 | VD: 0.00 | GP: 0.28 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 880/1000000 [6:28:53<7069:19:09, 25.47s/it]G: 0.60 | MSG: -0.27 | VG: 0.00 | D: 2.00 | MSD: 2.00 | VD: 0.00 | GP: 0.32 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▎ | 900/1000000 [6:37:35<7029:19:08, 25.33s/it]G: 1.68 | MSG: -0.32 | VG: 0.00 | D: 0.20 | MSD: 2.01 | VD: 0.00 | GP: 0.40 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 920/1000000 [6:46:18<7051:55:57, 25.41s/it]G: 0.91 | MSG: -0.28 | VG: 0.00 | D: 1.49 | MSD: 2.00 | VD: 0.00 | GP: 0.56 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▎ | 940/1000000 [6:54:55<7026:58:37, 25.32s/it]G: 10.57 | MSG: -0.25 | VG: 0.00 | D: 0.69 | MSD: 1.82 | VD: 0.00 | GP: 29.22 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%|▎ | 960/1000000 [7:03:35<7081:14:13, 25.52s/it]G: 5.75 | MSG: -0.33 | VG: 0.00 | D: 0.44 | MSD: 2.07 | VD: 0.00 | GP: 0.19 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 980/1000000 [7:12:13<7005:16:15, 25.24s/it]G: 5.05 | MSG: -0.33 | VG: 0.00 | D: 0.51 | MSD: 2.02 | VD: 0.00 | GP: 0.29 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 1000/1000000 [7:20:56<7095:10:31, 25.57s/it]G: 3.80 | MSG: -0.30 | VG: 0.00 | D: 0.71 | MSD: 2.05 | VD: 0.00 | GP: 0.14 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 1020/1000000 [7:29:46<7142:00:29, 25.74s/it]G: 4.50 | MSG: -0.30 | VG: 0.00 | D: 0.05 | MSD: 1.97 | VD: 0.00 | GP: 0.23 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▎ | 1040/1000000 [7:38:27<7014:49:50, 25.28s/it]G: 1.50 | MSG: -0.38 | VG: 0.00 | D: 0.87 | MSD: 2.01 | VD: 0.00 | GP: 0.19 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1060/1000000 [7:47:06<7066:06:38, 25.46s/it]G: 1.45 | MSG: -0.20 | VG: 0.00 | D: 0.01 | MSD: 1.97 | VD: 0.00 | GP: 0.15 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▎ | 1080/1000000 [7:55:45<7078:25:25, 25.51s/it]G: 1.17 | MSG: -0.31 | VG: 0.00 | D: 0.01 | MSD: 2.02 | VD: 0.00 | GP: 0.59 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%|▎ | 1100/1000000 [8:04:25<7093:11:14, 25.56s/it]G: 1.33 | MSG: -0.32 | VG: 0.00 | D: 0.41 | MSD: 2.00 | VD: 0.00 | GP: 0.50 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1120/1000000 [8:13:14<7142:12:30, 25.74s/it]G: 1.82 | MSG: -0.32 | VG: 0.00 | D: 0.25 | MSD: 2.03 | VD: 0.00 | GP: 0.69 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▎ | 1140/1000000 [8:21:55<7063:58:34, 25.46s/it]G: 1.18 | MSG: -0.29 | VG: 0.00 | D: 0.13 | MSD: 2.00 | VD: 0.00 | GP: 0.29 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1160/1000000 [8:30:36<7027:43:12, 25.33s/it]G: 2.07 | MSG: -0.30 | VG: 0.00 | D: 0.98 | MSD: 2.02 | VD: 0.00 | GP: 0.86 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1180/1000000 [8:39:15<7052:29:16, 25.42s/it]G: 0.71 | MSG: -0.31 | VG: 0.00 | D: 1.99 | MSD: 2.00 | VD: 0.00 | GP: 0.22 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 1200/1000000 [8:47:53<7067:27:58, 25.47s/it]G: 2.26 | MSG: -0.27 | VG: 0.00 | D: 0.49 | MSD: 2.00 | VD: 0.00 | GP: 0.14 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▎ | 1220/1000000 [8:56:41<7066:18:37, 25.47s/it]G: 1.69 | MSG: -0.32 | VG: 0.00 | D: 0.37 | MSD: 1.99 | VD: 0.00 | GP: 3.05 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 1240/1000000 [9:05:24<7059:30:15, 25.45s/it]G: 1.38 | MSG: -0.30 | VG: 0.00 | D: 0.69 | MSD: 1.99 | VD: 0.00 | GP: 0.96 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 1260/1000000 [9:14:05<7024:29:40, 25.32s/it]G: 2.82 | MSG: -0.33 | VG: 0.00 | D: 0.26 | MSD: 2.00 | VD: 0.00 | GP: 0.14 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 1280/1000000 [9:22:44<7022:21:35, 25.31s/it]G: 1.91 | MSG: -0.29 | VG: 0.00 | D: 0.60 | MSD: 2.00 | VD: 0.00 | GP: 0.34 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1300/1000000 [9:31:23<7018:32:32, 25.30s/it]G: 1.60 | MSG: -0.28 | VG: 0.00 | D: 0.65 | MSD: 2.00 | VD: 0.00 | GP: 0.18 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 1320/1000000 [9:40:12<7102:16:37, 25.60s/it]G: 2.82 | MSG: -0.30 | VG: 0.00 | D: 0.49 | MSD: 2.00 | VD: 0.00 | GP: 0.18 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▍ | 1340/1000000 [9:48:52<7018:29:13, 25.30s/it]G: 1.12 | MSG: -0.33 | VG: 0.00 | D: 0.95 | MSD: 2.01 | VD: 0.00 | GP: 0.25 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▍ | 1360/1000000 [9:57:32<7020:13:02, 25.31s/it]G: 1.03 | MSG: -0.32 | VG: 0.00 | D: 0.80 | MSD: 1.99 | VD: 0.00 | GP: 0.34 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▍ | 1380/1000000 [10:06:12<7043:34:34, 25.39s/it]G: 2.30 | MSG: -0.31 | VG: 0.00 | D: 0.60 | MSD: 2.00 | VD: 0.00 | GP: 0.22 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▍ | 1400/1000000 [10:14:51<7052:21:02, 25.42s/it]G: 3.13 | MSG: -0.32 | VG: 0.00 | D: 1.31 | MSD: 2.00 | VD: 0.00 | GP: 0.18 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▍ | 1420/1000000 [10:23:37<7004:11:48, 25.25s/it]G: 1.38 | MSG: -0.31 | VG: 0.00 | D: 1.21 | MSD: 2.00 | VD: 0.00 | GP: 1.39 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▍ | 1425/1000000 [10:25:51<7402:18:05, 26.69s/it]/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/autograd/__init__.py:200: UserWarning: Error detected in AddmmBackward0. Traceback of forward call that caused the error:
File "/home/ubuntu/giga/train.py", line 49, in <module>
gan(
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/gigagan_pytorch/gigagan_pytorch.py", line 2559, in forward
) = self.train_discriminator_step(
File "<@beartype(gigagan_pytorch.gigagan_pytorch.GigaGAN.train_discriminator_step) at 0x7f5b31cdb880>", line 31, in train_discriminator_step
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/gigagan_pytorch/gigagan_pytorch.py", line 2210, in train_discriminator_step
real_logits, real_multiscale_logits, aux_recon_losses = self.D(
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/accelerate/utils/operations.py", line 632, in forward
return model_forward(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/accelerate/utils/operations.py", line 620, in __call__
return convert_to_fp32(self.model_forward(*args, **kwargs))
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
return func(*args, **kwargs)
File "<@beartype(gigagan_pytorch.gigagan_pytorch.Discriminator.forward) at 0x7f5b31cda0c0>", line 94, in forward
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/gigagan_pytorch/gigagan_pytorch.py", line 1721, in forward
logits = self.to_logits(x)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
(Triggered internally at /opt/conda/conda-bld/pytorch_1682343995622/work/torch/csrc/autograd/python_anomaly_mode.cpp:114.)
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
0%|▍ | 1425/1000000 [10:25:52<7314:52:08, 26.37s/it]
Traceback (most recent call last):
File "/home/ubuntu/giga/train.py", line 49, in <module>
gan(
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/gigagan_pytorch/gigagan_pytorch.py", line 2559, in forward
) = self.train_discriminator_step(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<@beartype(gigagan_pytorch.gigagan_pytorch.GigaGAN.train_discriminator_step) at 0x7f5b31cdb880>", line 31, in train_discriminator_step
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/gigagan_pytorch/gigagan_pytorch.py", line 2297, in train_discriminator_step
self.accelerator.backward(total_loss / grad_accum_every)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/accelerate/accelerator.py", line 1921, in backward
self.scaler.scale(loss).backward(**kwargs)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/_tensor.py", line 487, in backward
torch.autograd.backward(
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/autograd/__init__.py", line 200, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: Function 'AddmmBackward0' returned nan values in its 2th output.
Have you successfully trained it? What am I doing wrong?
from gigagan-pytorch.
lucidrains
commented on May 5, 2024
1
@sansavision thanks for rerunning it! that tells me it isn't something wrong with the gradient penalty. looking over the code, also realize i wasn't updating the vision aided discriminator (fixed in 0.2.14)
i haven't trained it yet, and i don't think you are doing anything wrong; the problem is on my end. that error seems to suggest something is wrong with the discriminator
i'll have to setup the upscaler training once i have access to my deep learning rig next week and debug it
if you would like to continue poking around, what would be helpful is turning off the vision aided auxiliary loss (setting this to 0.)
from gigagan-pytorch.
sansavision
commented on May 5, 2024
1
@lucidrains Hi, I updated to 0.2.14, set vision_aided_divergence_loss_weight to 0. and re-trained it and it crashed after 14h. Here's the log, I hope it helps you track down the problem faster.
A100 GPU detected, using flash attention if input tensor is on cuda
Generator: 43.71M
Discriminator: 30.74M
Total images: 200000
0%| | 1/1000000 [00:00<?, ?it/s]G: 5.73 | MSG: 2.26 | VG: 0.00 | D: 2.02 | MSD: 1.45 | VD: 0.00 | GP: 0.00 | SSL: 0.88 | CL: 0.00 | MAL: 0.00
0%| | 20/1000000 [08:20<7116:06:35, 25.62s/it]G: 390.72 | MSG: 42.26 | VG: 0.00 | D: 0.00 | MSD: 0.85 | VD: 0.00 | GP: 39929.54 | SSL: 1.13 | CL: 0.00 | MAL: 0.00
0%| | 40/1000000 [17:00<7089:54:28, 25.52s/it]G: 133.95 | MSG: 1.58 | VG: 0.00 | D: 0.00 | MSD: 5.90 | VD: 0.00 | GP: 2645.54 | SSL: 0.16 | CL: 0.00 | MAL: 0.00
0%| | 60/1000000 [25:41<7043:57:22, 25.36s/it]G: 146.18 | MSG: 3.87 | VG: 0.00 | D: 0.00 | MSD: 3.37 | VD: 0.00 | GP: 2154.81 | SSL: 0.13 | CL: 0.00 | MAL: 0.00
0%| | 80/1000000 [34:22<7081:24:25, 25.50s/it]G: 7.82 | MSG: 1.46 | VG: 0.00 | D: 0.00 | MSD: 0.93 | VD: 0.00 | GP: 3411.81 | SSL: 0.28 | CL: 0.00 | MAL: 0.00
0%| | 100/1000000 [43:00<7024:39:11, 25.29s/it]G: 15.13 | MSG: 0.51 | VG: 0.00 | D: 0.04 | MSD: 1.10 | VD: 0.00 | GP: 1039.07 | SSL: 0.09 | CL: 0.00 | MAL: 0.00
0%| | 120/1000000 [51:46<7066:03:43, 25.44s/it]G: 65.04 | MSG: -0.32 | VG: 0.00 | D: 5.36 | MSD: 1.84 | VD: 0.00 | GP: 687.24 | SSL: 0.22 | CL: 0.00 | MAL: 0.00
0%| | 140/1000000 [1:00:26<7049:01:52, 25.38s/it]G: 2.85 | MSG: 0.08 | VG: 0.00 | D: 0.01 | MSD: 2.51 | VD: 0.00 | GP: 570.95 | SSL: 0.26 | CL: 0.00 | MAL: 0.00
0%| | 160/1000000 [1:09:07<7091:36:39, 25.53s/it]G: 16.74 | MSG: -0.18 | VG: 0.00 | D: 4.64 | MSD: 1.81 | VD: 0.00 | GP: 504.00 | SSL: 0.20 | CL: 0.00 | MAL: 0.00
0%| | 180/1000000 [1:17:46<7002:11:21, 25.21s/it]G: 136.43 | MSG: -0.80 | VG: 0.00 | D: 0.00 | MSD: 1.13 | VD: 0.00 | GP: 319.52 | SSL: 0.21 | CL: 0.00 | MAL: 0.00
0%| | 200/1000000 [1:26:25<7042:02:01, 25.36s/it]G: 46.28 | MSG: -0.12 | VG: 0.00 | D: 2.36 | MSD: 2.10 | VD: 0.00 | GP: 206.94 | SSL: 0.15 | CL: 0.00 | MAL: 0.00
0%| | 220/1000000 [1:35:12<7061:12:05, 25.43s/it]G: 33.96 | MSG: -0.45 | VG: 0.00 | D: 0.68 | MSD: 2.01 | VD: 0.00 | GP: 169.43 | SSL: 0.13 | CL: 0.00 | MAL: 0.00
0%| | 240/1000000 [1:43:50<7009:35:46, 25.24s/it]G: 3.09 | MSG: -0.24 | VG: 0.00 | D: 5.45 | MSD: 2.04 | VD: 0.00 | GP: 106.36 | SSL: 0.15 | CL: 0.00 | MAL: 0.00
0%| | 260/1000000 [1:52:30<7008:05:09, 25.24s/it]G: 111.48 | MSG: -0.19 | VG: 0.00 | D: 0.00 | MSD: 2.47 | VD: 0.00 | GP: 125.12 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%| | 280/1000000 [2:01:11<7057:03:17, 25.41s/it]G: 26.55 | MSG: -0.30 | VG: 0.00 | D: 0.24 | MSD: 2.00 | VD: 0.00 | GP: 52.12 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%| | 300/1000000 [2:09:52<7110:37:33, 25.61s/it]G: 84.80 | MSG: -0.23 | VG: 0.00 | D: 0.98 | MSD: 2.23 | VD: 0.00 | GP: 40.69 | SSL: 0.12 | CL: 0.00 | MAL: 0.00
0%| | 320/1000000 [2:18:39<7034:31:00, 25.33s/it]G: 2.76 | MSG: -0.29 | VG: 0.00 | D: 1.29 | MSD: 2.02 | VD: 0.00 | GP: 38.55 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%|▏ | 340/1000000 [2:27:23<7081:52:33, 25.50s/it]G: 7.64 | MSG: -0.26 | VG: 0.00 | D: 0.06 | MSD: 2.03 | VD: 0.00 | GP: 26.93 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%|▏ | 360/1000000 [2:36:02<7027:01:03, 25.31s/it]G: 6.64 | MSG: -0.39 | VG: 0.00 | D: 1.62 | MSD: 2.02 | VD: 0.00 | GP: 46.32 | SSL: 0.14 | CL: 0.00 | MAL: 0.00
0%|▏ | 380/1000000 [2:44:43<7071:22:11, 25.47s/it]G: 7.58 | MSG: -0.29 | VG: 0.00 | D: 0.00 | MSD: 2.03 | VD: 0.00 | GP: 17.08 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 400/1000000 [2:53:23<7101:15:25, 25.57s/it]G: 4.15 | MSG: -0.29 | VG: 0.00 | D: 0.05 | MSD: 2.10 | VD: 0.00 | GP: 17.22 | SSL: 0.14 | CL: 0.00 | MAL: 0.00
0%|▏ | 420/1000000 [3:02:10<7054:59:04, 25.41s/it]G: 3.91 | MSG: -0.29 | VG: 0.00 | D: 0.01 | MSD: 1.97 | VD: 0.00 | GP: 14.41 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%|▏ | 440/1000000 [3:10:51<7070:09:26, 25.46s/it]G: 18.80 | MSG: -0.23 | VG: 0.00 | D: 0.46 | MSD: 2.04 | VD: 0.00 | GP: 10.64 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%|▏ | 460/1000000 [3:19:32<7050:38:31, 25.39s/it]G: 4.52 | MSG: -0.26 | VG: 0.00 | D: 0.00 | MSD: 1.99 | VD: 0.00 | GP: 8.09 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 480/1000000 [3:28:14<7058:40:29, 25.42s/it]G: 7.27 | MSG: -0.08 | VG: 0.00 | D: 1.91 | MSD: 2.23 | VD: 0.00 | GP: 5.05 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%|▏ | 500/1000000 [3:36:54<7020:02:14, 25.28s/it]G: 6.16 | MSG: -0.26 | VG: 0.00 | D: 0.01 | MSD: 1.98 | VD: 0.00 | GP: 3.85 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 520/1000000 [3:45:40<7075:50:22, 25.49s/it]G: 3.82 | MSG: -0.26 | VG: 0.00 | D: 0.28 | MSD: 1.99 | VD: 0.00 | GP: 2.64 | SSL: 0.15 | CL: 0.00 | MAL: 0.00
0%|▏ | 540/1000000 [3:54:19<7096:15:12, 25.56s/it]G: 6.02 | MSG: -0.23 | VG: 0.00 | D: 0.00 | MSD: 1.79 | VD: 0.00 | GP: 5.41 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 560/1000000 [4:02:58<7055:18:51, 25.41s/it]G: 0.23 | MSG: -0.23 | VG: 0.00 | D: 1.52 | MSD: 2.02 | VD: 0.00 | GP: 3.43 | SSL: 0.15 | CL: 0.00 | MAL: 0.00
0%|▏ | 580/1000000 [4:11:38<7023:35:11, 25.30s/it]G: 6.62 | MSG: -0.32 | VG: 0.00 | D: 0.01 | MSD: 2.08 | VD: 0.00 | GP: 6.19 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 600/1000000 [4:20:16<7036:23:42, 25.35s/it]G: 17.90 | MSG: -0.12 | VG: 0.00 | D: 0.03 | MSD: 2.19 | VD: 0.00 | GP: 1.85 | SSL: 0.10 | CL: 0.00 | MAL: 0.00
0%|▏ | 620/1000000 [4:29:04<7083:22:09, 25.52s/it]G: 1.24 | MSG: -0.35 | VG: 0.00 | D: 0.04 | MSD: 2.02 | VD: 0.00 | GP: 0.21 | SSL: 0.09 | CL: 0.00 | MAL: 0.00
0%|▏ | 640/1000000 [4:37:43<7079:07:23, 25.50s/it]G: 5.60 | MSG: -0.25 | VG: 0.00 | D: 0.18 | MSD: 2.07 | VD: 0.00 | GP: 0.38 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%|▎ | 660/1000000 [4:46:24<7155:22:55, 25.78s/it]G: 4.12 | MSG: -0.36 | VG: 0.00 | D: 0.01 | MSD: 2.03 | VD: 0.00 | GP: 0.19 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 680/1000000 [4:55:06<7121:26:33, 25.65s/it]G: 3.43 | MSG: -0.33 | VG: 0.00 | D: 0.01 | MSD: 2.07 | VD: 0.00 | GP: 0.26 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%|▎ | 700/1000000 [5:03:45<7060:22:06, 25.44s/it]G: 5.73 | MSG: -0.33 | VG: 0.00 | D: 0.01 | MSD: 2.05 | VD: 0.00 | GP: 0.36 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▎ | 720/1000000 [5:12:32<7094:42:48, 25.56s/it]G: 12.38 | MSG: -0.11 | VG: 0.00 | D: 0.01 | MSD: 2.23 | VD: 0.00 | GP: 24.71 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▎ | 740/1000000 [5:21:13<7067:08:19, 25.46s/it]G: 11.72 | MSG: -0.36 | VG: 0.00 | D: 0.92 | MSD: 2.01 | VD: 0.00 | GP: 0.61 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%|▎ | 760/1000000 [5:29:55<7126:06:06, 25.67s/it]G: 3.84 | MSG: 0.06 | VG: 0.00 | D: 0.20 | MSD: 2.23 | VD: 0.00 | GP: 1.38 | SSL: 0.14 | CL: 0.00 | MAL: 0.00
0%|▎ | 780/1000000 [5:38:36<7085:18:29, 25.53s/it]G: 1.31 | MSG: -0.36 | VG: 0.00 | D: 2.41 | MSD: 2.00 | VD: 0.00 | GP: 0.37 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%|▎ | 800/1000000 [5:47:18<7090:35:32, 25.55s/it]G: 1.72 | MSG: -0.35 | VG: 0.00 | D: 1.75 | MSD: 2.00 | VD: 0.00 | GP: 0.20 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%|▎ | 820/1000000 [5:56:01<7014:53:33, 25.27s/it]G: 4.73 | MSG: -0.47 | VG: 0.00 | D: 0.01 | MSD: 1.93 | VD: 0.00 | GP: 1.68 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%|▎ | 840/1000000 [6:04:42<7071:33:16, 25.48s/it]G: 1.06 | MSG: -0.37 | VG: 0.00 | D: 0.67 | MSD: 1.94 | VD: 0.00 | GP: 0.87 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%|▎ | 860/1000000 [6:13:19<7021:21:46, 25.30s/it]G: 2.06 | MSG: -0.33 | VG: 0.00 | D: 7.41 | MSD: 2.01 | VD: 0.00 | GP: 0.16 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▎ | 880/1000000 [6:22:00<7072:41:55, 25.48s/it]G: 0.20 | MSG: -0.32 | VG: 0.00 | D: 1.60 | MSD: 2.15 | VD: 0.00 | GP: 4.02 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▎ | 900/1000000 [6:30:37<7044:30:02, 25.38s/it]G: 1.66 | MSG: -0.38 | VG: 0.00 | D: 4.08 | MSD: 2.00 | VD: 0.00 | GP: 0.21 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▎ | 920/1000000 [6:39:23<7082:30:11, 25.52s/it]G: 1.17 | MSG: -0.18 | VG: 0.00 | D: 0.00 | MSD: 1.74 | VD: 0.00 | GP: 1.46 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▎ | 940/1000000 [6:48:04<7097:25:13, 25.57s/it]G: 10.39 | MSG: -0.28 | VG: 0.00 | D: 0.55 | MSD: 1.97 | VD: 0.00 | GP: 0.28 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%|▎ | 960/1000000 [6:56:44<7055:15:04, 25.42s/it]G: 1.83 | MSG: -0.39 | VG: 0.00 | D: 0.21 | MSD: 2.00 | VD: 0.00 | GP: 0.20 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%|▍ | 980/1000000 [7:05:24<7039:53:00, 25.37s/it]G: 0.41 | MSG: -0.47 | VG: 0.00 | D: 0.86 | MSD: 2.00 | VD: 0.00 | GP: 0.47 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▍ | 1000/1000000 [7:14:04<7049:50:22, 25.40s/it]G: 3.37 | MSG: -0.35 | VG: 0.00 | D: 1.96 | MSD: 1.99 | VD: 0.00 | GP: 0.61 | SSL: 0.12 | CL: 0.00 | MAL: 0.00
0%|▍ | 1020/1000000 [7:22:48<7011:43:17, 25.27s/it]G: 4.37 | MSG: -0.39 | VG: 0.00 | D: 2.19 | MSD: 2.00 | VD: 0.00 | GP: 0.37 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▍ | 1040/1000000 [7:31:25<7015:42:22, 25.28s/it]G: 3.14 | MSG: -0.42 | VG: 0.00 | D: 0.40 | MSD: 2.00 | VD: 0.00 | GP: 0.31 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▍ | 1060/1000000 [7:40:02<7065:55:52, 25.46s/it]G: 2.98 | MSG: -0.45 | VG: 0.00 | D: 0.09 | MSD: 2.13 | VD: 0.00 | GP: 0.15 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%|▍ | 1080/1000000 [7:48:42<7060:32:58, 25.45s/it]G: 0.64 | MSG: -0.36 | VG: 0.00 | D: 1.31 | MSD: 2.00 | VD: 0.00 | GP: 0.18 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▍ | 1100/1000000 [7:57:21<7045:23:38, 25.39s/it]G: 3.92 | MSG: -0.31 | VG: 0.00 | D: 0.01 | MSD: 1.98 | VD: 0.00 | GP: 0.19 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▍ | 1120/1000000 [8:06:10<7058:08:45, 25.44s/it]G: 0.21 | MSG: -0.31 | VG: 0.00 | D: 1.74 | MSD: 2.00 | VD: 0.00 | GP: 0.38 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▍ | 1140/1000000 [8:14:49<7044:12:12, 25.39s/it]G: 4.35 | MSG: -0.37 | VG: 0.00 | D: 1.45 | MSD: 2.00 | VD: 0.00 | GP: 0.39 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▍ | 1160/1000000 [8:23:30<7085:46:00, 25.54s/it]G: 4.22 | MSG: -0.39 | VG: 0.00 | D: 0.00 | MSD: 2.02 | VD: 0.00 | GP: 0.35 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▍ | 1180/1000000 [8:32:12<7080:14:34, 25.52s/it]G: 5.46 | MSG: -0.39 | VG: 0.00 | D: 0.37 | MSD: 1.84 | VD: 0.00 | GP: 35.70 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▍ | 1200/1000000 [8:40:53<7062:41:52, 25.46s/it]G: 2.19 | MSG: -0.40 | VG: 0.00 | D: 0.81 | MSD: 2.00 | VD: 0.00 | GP: 0.10 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▍ | 1220/1000000 [8:49:37<7050:07:19, 25.41s/it]G: 4.33 | MSG: -0.39 | VG: 0.00 | D: 0.82 | MSD: 2.01 | VD: 0.00 | GP: 0.11 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▍ | 1240/1000000 [8:58:18<7098:16:22, 25.59s/it]G: 3.86 | MSG: -0.42 | VG: 0.00 | D: 1.91 | MSD: 2.00 | VD: 0.00 | GP: 0.45 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▍ | 1260/1000000 [9:06:58<7036:38:14, 25.36s/it]G: 3.23 | MSG: -0.47 | VG: 0.00 | D: 0.00 | MSD: 1.93 | VD: 0.00 | GP: 0.41 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▍ | 1280/1000000 [9:15:38<7031:15:38, 25.34s/it]G: 0.90 | MSG: -0.31 | VG: 0.00 | D: 1.88 | MSD: 2.00 | VD: 0.00 | GP: 0.17 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▌ | 1300/1000000 [9:24:18<7062:15:41, 25.46s/it]G: 4.47 | MSG: -0.34 | VG: 0.00 | D: 0.15 | MSD: 1.98 | VD: 0.00 | GP: 0.21 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%|▌ | 1320/1000000 [9:33:06<7137:18:00, 25.73s/it]G: 1.70 | MSG: -0.37 | VG: 0.00 | D: 0.49 | MSD: 1.99 | VD: 0.00 | GP: 0.56 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▌ | 1340/1000000 [9:41:46<7043:37:28, 25.39s/it]G: 1.59 | MSG: -0.33 | VG: 0.00 | D: 0.73 | MSD: 2.00 | VD: 0.00 | GP: 0.12 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▌ | 1360/1000000 [9:50:26<7047:50:58, 25.41s/it]G: 0.75 | MSG: -0.41 | VG: 0.00 | D: 1.82 | MSD: 2.00 | VD: 0.00 | GP: 0.37 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▌ | 1380/1000000 [9:59:05<7002:39:50, 25.24s/it]G: 1.60 | MSG: -0.38 | VG: 0.00 | D: 0.75 | MSD: 1.99 | VD: 0.00 | GP: 0.23 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▌ | 1400/1000000 [10:07:46<7085:19:02, 25.54s/it]G: 1.74 | MSG: -0.33 | VG: 0.00 | D: 0.98 | MSD: 1.97 | VD: 0.00 | GP: 7.27 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▌ | 1420/1000000 [10:16:32<7107:04:05, 25.62s/it]G: 3.46 | MSG: -0.28 | VG: 0.00 | D: 0.27 | MSD: 1.95 | VD: 0.00 | GP: 0.13 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▌ | 1440/1000000 [10:25:11<7037:34:33, 25.37s/it]G: 0.30 | MSG: -0.37 | VG: 0.00 | D: 2.15 | MSD: 2.02 | VD: 0.00 | GP: 0.09 | SSL: 0.11 | CL: 0.00 | MAL: 0.00
0%|▌ | 1460/1000000 [10:33:52<7081:15:12, 25.53s/it]G: 1.06 | MSG: -0.31 | VG: 0.00 | D: 0.62 | MSD: 1.96 | VD: 0.00 | GP: 0.49 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%|▌ | 1480/1000000 [10:42:29<7037:35:37, 25.37s/it]G: 1.09 | MSG: -0.30 | VG: 0.00 | D: 0.61 | MSD: 2.00 | VD: 0.00 | GP: 0.29 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▌ | 1500/1000000 [10:51:10<7149:14:00, 25.78s/it]G: 1.19 | MSG: -0.28 | VG: 0.00 | D: 0.83 | MSD: 1.99 | VD: 0.00 | GP: 0.20 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▌ | 1520/1000000 [10:59:56<7077:59:31, 25.52s/it]G: 2.03 | MSG: -0.34 | VG: 0.00 | D: 1.57 | MSD: 1.98 | VD: 0.00 | GP: 1.25 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▌ | 1540/1000000 [11:08:38<7069:34:56, 25.49s/it]G: 0.86 | MSG: -0.34 | VG: 0.00 | D: 0.51 | MSD: 1.99 | VD: 0.00 | GP: 0.55 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▌ | 1560/1000000 [11:17:19<7060:44:18, 25.46s/it]G: 1.54 | MSG: -0.32 | VG: 0.00 | D: 1.16 | MSD: 2.00 | VD: 0.00 | GP: 0.40 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▌ | 1580/1000000 [11:25:59<6993:50:02, 25.22s/it]G: 1.26 | MSG: -0.34 | VG: 0.00 | D: 0.51 | MSD: 2.00 | VD: 0.00 | GP: 0.44 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▌ | 1600/1000000 [11:34:39<7044:04:31, 25.40s/it]G: 1.91 | MSG: -0.37 | VG: 0.00 | D: 0.42 | MSD: 2.06 | VD: 0.00 | GP: 2.40 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▋ | 1620/1000000 [11:43:25<7029:28:22, 25.35s/it]G: 3.55 | MSG: -0.29 | VG: 0.00 | D: 0.18 | MSD: 1.94 | VD: 0.00 | GP: 0.28 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▋ | 1640/1000000 [11:52:02<7043:24:47, 25.40s/it]G: 1.94 | MSG: -0.32 | VG: 0.00 | D: 0.56 | MSD: 2.04 | VD: 0.00 | GP: 1.09 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1660/1000000 [12:00:41<7043:12:42, 25.40s/it]G: 0.90 | MSG: -0.35 | VG: 0.00 | D: 1.42 | MSD: 2.00 | VD: 0.00 | GP: 0.34 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▋ | 1680/1000000 [12:09:22<7029:52:46, 25.35s/it]G: 1.04 | MSG: -0.40 | VG: 0.00 | D: 0.71 | MSD: 2.05 | VD: 0.00 | GP: 0.32 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▋ | 1700/1000000 [12:18:05<7110:46:20, 25.64s/it]G: 1.56 | MSG: -0.33 | VG: 0.00 | D: 1.26 | MSD: 2.02 | VD: 0.00 | GP: 1.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1720/1000000 [12:26:51<7065:34:51, 25.48s/it]G: 3.42 | MSG: -0.37 | VG: 0.00 | D: 0.49 | MSD: 2.02 | VD: 0.00 | GP: 0.75 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1740/1000000 [12:35:31<6995:04:09, 25.23s/it]G: 2.00 | MSG: -0.34 | VG: 0.00 | D: 0.79 | MSD: 2.16 | VD: 0.00 | GP: 10.92 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▋ | 1760/1000000 [12:44:12<7013:38:47, 25.29s/it]G: 1.58 | MSG: -0.30 | VG: 0.00 | D: 1.54 | MSD: 1.99 | VD: 0.00 | GP: 0.20 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▋ | 1780/1000000 [12:52:53<7047:06:41, 25.41s/it]G: 1.59 | MSG: -0.31 | VG: 0.00 | D: 2.06 | MSD: 2.00 | VD: 0.00 | GP: 0.15 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▋ | 1800/1000000 [13:01:33<7115:02:39, 25.66s/it]G: 0.93 | MSG: -0.34 | VG: 0.00 | D: 0.44 | MSD: 2.06 | VD: 0.00 | GP: 1.85 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1820/1000000 [13:10:22<7127:30:35, 25.71s/it]G: 1.31 | MSG: -0.32 | VG: 0.00 | D: 0.24 | MSD: 1.99 | VD: 0.00 | GP: 0.09 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▋ | 1840/1000000 [13:19:01<7015:35:35, 25.30s/it]G: 1.69 | MSG: -0.32 | VG: 0.00 | D: 0.31 | MSD: 1.99 | VD: 0.00 | GP: 0.14 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1860/1000000 [13:27:40<7008:35:16, 25.28s/it]G: 0.85 | MSG: -0.34 | VG: 0.00 | D: 0.30 | MSD: 1.98 | VD: 0.00 | GP: 1.14 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▋ | 1880/1000000 [13:36:21<7048:38:45, 25.42s/it]G: 1.61 | MSG: -0.34 | VG: 0.00 | D: 0.26 | MSD: 2.00 | VD: 0.00 | GP: 0.07 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▋ | 1900/1000000 [13:45:02<7035:09:18, 25.37s/it]G: 0.97 | MSG: -0.39 | VG: 0.00 | D: 1.22 | MSD: 2.00 | VD: 0.00 | GP: 0.15 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1920/1000000 [13:53:49<7044:50:59, 25.41s/it]G: 0.95 | MSG: -0.34 | VG: 0.00 | D: 0.90 | MSD: 1.98 | VD: 0.00 | GP: 0.37 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▊ | 1940/1000000 [14:02:30<7061:10:54, 25.47s/it]/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/autograd/__init__.py:303: UserWarning: Error detected in MulBackward0. Traceback of forward call that caused the error:
File "/home/ubuntu/giga/train.py", line 50, in <module>
gan(
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/gigagan_pytorch/gigagan_pytorch.py", line 2566, in forward
) = self.train_discriminator_step(
File "<@beartype(gigagan_pytorch.gigagan_pytorch.GigaGAN.train_discriminator_step) at 0x7f7a5b9c7880>", line 31, in train_discriminator_step
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/gigagan_pytorch/gigagan_pytorch.py", line 2214, in train_discriminator_step
real_logits, real_multiscale_logits, aux_recon_losses = self.D(
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/accelerate/utils/operations.py", line 632, in forward
return model_forward(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/accelerate/utils/operations.py", line 620, in __call__
return convert_to_fp32(self.model_forward(*args, **kwargs))
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
return func(*args, **kwargs)
File "<@beartype(gigagan_pytorch.gigagan_pytorch.Discriminator.forward) at 0x7f7a5b9c60c0>", line 94, in forward
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/gigagan_pytorch/gigagan_pytorch.py", line 1653, in forward
x = x * excite
(Triggered internally at /opt/conda/conda-bld/pytorch_1682343995622/work/torch/csrc/autograd/python_anomaly_mode.cpp:114.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
0%|▊ | 1940/1000000 [14:02:31<7227:52:39, 26.07s/it]
Traceback (most recent call last):
File "/home/ubuntu/giga/train.py", line 50, in <module>
gan(
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/gigagan_pytorch/gigagan_pytorch.py", line 2566, in forward
) = self.train_discriminator_step(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<@beartype(gigagan_pytorch.gigagan_pytorch.GigaGAN.train_discriminator_step) at 0x7f7a5b9c7880>", line 31, in train_discriminator_step
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/gigagan_pytorch/gigagan_pytorch.py", line 2242, in train_discriminator_step
gp_loss = gradient_penalty(
^^^^^^^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/gigagan_pytorch/gigagan_pytorch.py", line 133, in gradient_penalty
maybe_scaled_gradients, *_ = torch_grad(
^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/autograd/__init__.py", line 303, in grad
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Function 'MulBackward0' returned nan values in its 1th output.
I'll continue with this again later next week when I have more time.
from gigagan-pytorch.
sansavision
commented on May 5, 2024
1
@lucidrains sure, should I keep "vision_aided_divergence_loss_weight" set to 0. this time as well?
from gigagan-pytorch.
sansavision
commented on May 5, 2024
1
@lucidrains It didn't change, at the end it just started producing all black images with nan values. I wonder what the issue is.
A100 GPU detected, using flash attention if input tensor is on cuda
Generator: 43.71M
Discriminator: 30.74M
Total images: 200000
0%| | 1/1000000 [00:00<?, ?it/s]G: 1.61 | MSG: -2.99 | VG: 0.00 | D: 3.59 | MSD: 4.03 | VD: 0.00 | GP: 0.00 | SSL: 1.42 | CL: 0.00 | MAL: 0.00
0%| | 20/1000000 [06:08<5216:57:39, 18.78s/it]G: 711.23 | MSG: 59.48 | VG: 0.00 | D: 6.95 | MSD: 3.85 | VD: 0.00 | GP: 24906.48 | SSL: 0.34 | CL: 0.00 | MAL: 0.00
0%| | 40/1000000 [12:32<5186:15:26, 18.67s/it]G: 57.88 | MSG: -1.01 | VG: 0.00 | D: 12.78 | MSD: 4.41 | VD: 0.00 | GP: 13244.12 | SSL: 0.27 | CL: 0.00 | MAL: 0.00
0%| | 60/1000000 [18:54<5212:08:45, 18.76s/it]G: 4.31 | MSG: 0.02 | VG: 0.00 | D: 1.39 | MSD: 2.37 | VD: 0.00 | GP: 3309.06 | SSL: 0.11 | CL: 0.00 | MAL: 0.00
0%| | 80/1000000 [25:16<5163:41:22, 18.59s/it]G: 144.49 | MSG: 2.62 | VG: 0.00 | D: 0.01 | MSD: 1.39 | VD: 0.00 | GP: 2278.78 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%| | 100/1000000 [31:38<5144:01:14, 18.52s/it]G: 8.01 | MSG: -0.57 | VG: 0.00 | D: 0.00 | MSD: 1.89 | VD: 0.00 | GP: 1835.44 | SSL: 1.43 | CL: 0.00 | MAL: 0.00
0%| | 120/1000000 [38:06<5208:37:38, 18.75s/it]G: 9.09 | MSG: 0.08 | VG: 0.00 | D: 4.24 | MSD: 2.14 | VD: 0.00 | GP: 1039.87 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%| | 140/1000000 [44:27<5143:28:11, 18.52s/it]G: 4.71 | MSG: 0.06 | VG: 0.00 | D: 0.00 | MSD: 1.41 | VD: 0.00 | GP: 637.45 | SSL: 0.18 | CL: 0.00 | MAL: 0.00
0%| | 160/1000000 [50:48<5173:15:37, 18.63s/it]G: -0.44 | MSG: -0.36 | VG: 0.00 | D: 3.27 | MSD: 2.02 | VD: 0.00 | GP: 511.06 | SSL: 0.10 | CL: 0.00 | MAL: 0.00
0%| | 180/1000000 [57:10<5170:20:19, 18.62s/it]G: 114.36 | MSG: 0.08 | VG: 0.00 | D: 7.45 | MSD: 1.99 | VD: 0.00 | GP: 427.41 | SSL: 0.36 | CL: 0.00 | MAL: 0.00
0%| | 200/1000000 [1:03:30<5176:03:10, 18.64s/it]G: 59.06 | MSG: -0.02 | VG: 0.00 | D: 0.00 | MSD: 2.10 | VD: 0.00 | GP: 360.38 | SSL: 0.21 | CL: 0.00 | MAL: 0.00
0%| | 220/1000000 [1:09:58<5198:04:58, 18.72s/it]G: 162.94 | MSG: -0.04 | VG: 0.00 | D: 0.00 | MSD: 2.33 | VD: 0.00 | GP: 229.12 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%| | 240/1000000 [1:16:17<5137:57:45, 18.50s/it]G: 11.01 | MSG: -0.05 | VG: 0.00 | D: 0.00 | MSD: 2.19 | VD: 0.00 | GP: 170.95 | SSL: 0.22 | CL: 0.00 | MAL: 0.00
0%| | 260/1000000 [1:22:40<5188:03:23, 18.68s/it]G: -0.99 | MSG: -0.11 | VG: 0.00 | D: 2.66 | MSD: 1.98 | VD: 0.00 | GP: 89.25 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%| | 280/1000000 [1:29:02<5184:58:31, 18.67s/it]G: 1.75 | MSG: -0.16 | VG: 0.00 | D: 2.11 | MSD: 1.99 | VD: 0.00 | GP: 71.47 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%| | 300/1000000 [1:35:21<5146:14:38, 18.53s/it]G: 21.75 | MSG: -0.00 | VG: 0.00 | D: 3.01 | MSD: 1.94 | VD: 0.00 | GP: 67.14 | SSL: 0.26 | CL: 0.00 | MAL: 0.00
0%| | 320/1000000 [1:41:50<5209:24:42, 18.76s/it]G: 1.79 | MSG: -0.04 | VG: 0.00 | D: 1.69 | MSD: 2.00 | VD: 0.00 | GP: 52.02 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%|▏ | 340/1000000 [1:48:12<5193:15:35, 18.70s/it]G: 2.47 | MSG: -0.05 | VG: 0.00 | D: 3.18 | MSD: 1.97 | VD: 0.00 | GP: 32.49 | SSL: 0.12 | CL: 0.00 | MAL: 0.00
0%|▏ | 360/1000000 [1:54:36<5144:37:19, 18.53s/it]G: -1.00 | MSG: -0.04 | VG: 0.00 | D: 2.67 | MSD: 2.03 | VD: 0.00 | GP: 36.76 | SSL: 0.09 | CL: 0.00 | MAL: 0.00
0%|▏ | 380/1000000 [2:00:57<5127:56:33, 18.47s/it]G: 23.69 | MSG: -0.04 | VG: 0.00 | D: 0.04 | MSD: 1.98 | VD: 0.00 | GP: 14.17 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%|▏ | 400/1000000 [2:07:18<5135:01:34, 18.49s/it]G: 2.43 | MSG: -0.09 | VG: 0.00 | D: 0.57 | MSD: 2.01 | VD: 0.00 | GP: 10.65 | SSL: 0.18 | CL: 0.00 | MAL: 0.00
0%|▏ | 420/1000000 [2:13:43<5139:00:15, 18.51s/it]G: 5.77 | MSG: -0.07 | VG: 0.00 | D: 2.06 | MSD: 2.00 | VD: 0.00 | GP: 12.92 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%|▏ | 440/1000000 [2:20:02<5145:30:34, 18.53s/it]G: 5.50 | MSG: -0.09 | VG: 0.00 | D: 0.00 | MSD: 1.99 | VD: 0.00 | GP: 8.64 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%|▏ | 460/1000000 [2:26:24<5211:08:33, 18.77s/it]G: 3.64 | MSG: -0.08 | VG: 0.00 | D: 1.17 | MSD: 2.01 | VD: 0.00 | GP: 5.13 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%|▏ | 480/1000000 [2:32:46<5185:07:37, 18.68s/it]G: 2.67 | MSG: -0.09 | VG: 0.00 | D: 1.34 | MSD: 1.99 | VD: 0.00 | GP: 3.57 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%|▏ | 500/1000000 [2:39:06<5146:03:06, 18.54s/it]G: 13.76 | MSG: -0.05 | VG: 0.00 | D: 2.83 | MSD: 1.89 | VD: 0.00 | GP: 4.17 | SSL: 0.11 | CL: 0.00 | MAL: 0.00
0%|▏ | 520/1000000 [2:45:33<5162:11:46, 18.59s/it]G: 1.33 | MSG: -0.10 | VG: 0.00 | D: 0.27 | MSD: 1.99 | VD: 0.00 | GP: 2.78 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%|▏ | 540/1000000 [2:51:56<5117:31:17, 18.43s/it]G: 13.61 | MSG: -0.09 | VG: 0.00 | D: 0.04 | MSD: 2.05 | VD: 0.00 | GP: 0.55 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 560/1000000 [2:58:18<5154:49:54, 18.57s/it]G: 6.38 | MSG: -0.08 | VG: 0.00 | D: 1.28 | MSD: 1.98 | VD: 0.00 | GP: 0.65 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 580/1000000 [3:04:40<5205:40:04, 18.75s/it]G: 2.67 | MSG: -0.10 | VG: 0.00 | D: 0.10 | MSD: 1.99 | VD: 0.00 | GP: 4.64 | SSL: 0.15 | CL: 0.00 | MAL: 0.00
0%|▏ | 600/1000000 [3:11:02<5133:24:53, 18.49s/it]G: 2.39 | MSG: -0.08 | VG: 0.00 | D: 0.99 | MSD: 1.91 | VD: 0.00 | GP: 0.53 | SSL: 0.11 | CL: 0.00 | MAL: 0.00
0%|▏ | 620/1000000 [3:17:28<5150:57:20, 18.55s/it]G: 1.77 | MSG: -0.11 | VG: 0.00 | D: 0.45 | MSD: 2.01 | VD: 0.00 | GP: 3.07 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 640/1000000 [3:23:48<5164:21:46, 18.60s/it]G: 3.01 | MSG: -0.09 | VG: 0.00 | D: 0.84 | MSD: 2.00 | VD: 0.00 | GP: 0.23 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 660/1000000 [3:30:09<5117:07:11, 18.43s/it]G: 2.23 | MSG: -0.04 | VG: 0.00 | D: 0.06 | MSD: 1.98 | VD: 0.00 | GP: 0.18 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%|▎ | 680/1000000 [3:36:32<5170:57:14, 18.63s/it]G: 2.22 | MSG: -0.12 | VG: 0.00 | D: 0.87 | MSD: 1.98 | VD: 0.00 | GP: 0.92 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▎ | 700/1000000 [3:42:57<5254:05:19, 18.93s/it]G: 4.45 | MSG: -0.09 | VG: 0.00 | D: 0.29 | MSD: 2.03 | VD: 0.00 | GP: 0.29 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▎ | 720/1000000 [3:49:26<5182:21:02, 18.67s/it]G: 1.55 | MSG: -0.09 | VG: 0.00 | D: 1.68 | MSD: 2.00 | VD: 0.00 | GP: 0.17 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 740/1000000 [3:55:47<5199:02:09, 18.73s/it]G: 4.80 | MSG: -0.08 | VG: 0.00 | D: 0.53 | MSD: 2.24 | VD: 0.00 | GP: 3.99 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 760/1000000 [4:02:07<5135:18:37, 18.50s/it]G: 1.98 | MSG: -0.08 | VG: 0.00 | D: 2.22 | MSD: 2.01 | VD: 0.00 | GP: 0.28 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▎ | 780/1000000 [4:08:28<5140:46:34, 18.52s/it]G: 1.96 | MSG: -0.08 | VG: 0.00 | D: 2.02 | MSD: 2.00 | VD: 0.00 | GP: 0.19 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 800/1000000 [4:14:49<5209:15:05, 18.77s/it]G: 1.54 | MSG: -0.07 | VG: 0.00 | D: 1.45 | MSD: 2.00 | VD: 0.00 | GP: 0.25 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%|▎ | 820/1000000 [4:21:14<5116:20:12, 18.43s/it]G: 3.15 | MSG: -0.08 | VG: 0.00 | D: 0.15 | MSD: 2.01 | VD: 0.00 | GP: 0.16 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 840/1000000 [4:27:36<5147:31:57, 18.55s/it]G: 3.44 | MSG: -0.02 | VG: 0.00 | D: 0.16 | MSD: 1.98 | VD: 0.00 | GP: 0.19 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%|▎ | 860/1000000 [4:34:00<5233:43:26, 18.86s/it]G: 3.03 | MSG: -0.07 | VG: 0.00 | D: 0.54 | MSD: 1.97 | VD: 0.00 | GP: 0.74 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▎ | 880/1000000 [4:40:22<5172:47:42, 18.64s/it]G: 1.21 | MSG: -0.09 | VG: 0.00 | D: 3.38 | MSD: 2.00 | VD: 0.00 | GP: 0.43 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 900/1000000 [4:46:41<5187:55:05, 18.69s/it]G: 1.58 | MSG: -0.08 | VG: 0.00 | D: 0.70 | MSD: 1.99 | VD: 0.00 | GP: 0.16 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 920/1000000 [4:53:07<5094:33:49, 18.36s/it]G: 1.13 | MSG: -0.04 | VG: 0.00 | D: 1.23 | MSD: 1.99 | VD: 0.00 | GP: 0.15 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▎ | 940/1000000 [4:59:30<5207:22:10, 18.76s/it]G: 0.81 | MSG: -0.07 | VG: 0.00 | D: 0.40 | MSD: 2.00 | VD: 0.00 | GP: 0.15 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 960/1000000 [5:05:54<5208:49:41, 18.77s/it]G: 6.50 | MSG: -0.02 | VG: 0.00 | D: 0.61 | MSD: 1.98 | VD: 0.00 | GP: 0.15 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▍ | 980/1000000 [5:12:15<5159:44:31, 18.59s/it]G: 1.53 | MSG: -0.09 | VG: 0.00 | D: 0.40 | MSD: 2.01 | VD: 0.00 | GP: 0.30 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▍ | 1000/1000000 [5:18:36<5171:05:15, 18.63s/it]G: 7.18 | MSG: -0.13 | VG: 0.00 | D: 6.30 | MSD: 1.73 | VD: 0.00 | GP: 4.40 | SSL: 0.21 | CL: 0.00 | MAL: 0.00
0%|▍ | 1020/1000000 [5:25:03<5240:38:25, 18.89s/it]G: 2.15 | MSG: -0.10 | VG: 0.00 | D: 1.96 | MSD: 2.02 | VD: 0.00 | GP: 0.63 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▍ | 1040/1000000 [5:31:23<5143:04:52, 18.53s/it]G: 5.23 | MSG: -0.11 | VG: 0.00 | D: 0.50 | MSD: 2.02 | VD: 0.00 | GP: 0.15 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▍ | 1060/1000000 [5:37:42<5146:27:16, 18.55s/it]G: 2.93 | MSG: -0.13 | VG: 0.00 | D: 0.20 | MSD: 2.02 | VD: 0.00 | GP: 0.41 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▍ | 1080/1000000 [5:44:04<5151:26:14, 18.57s/it]G: 1.39 | MSG: -0.12 | VG: 0.00 | D: 1.23 | MSD: 2.02 | VD: 0.00 | GP: 0.93 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▍ | 1100/1000000 [5:50:25<5129:17:53, 18.49s/it]G: 2.21 | MSG: -0.12 | VG: 0.00 | D: 1.79 | MSD: 2.01 | VD: 0.00 | GP: 0.29 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▍ | 1120/1000000 [5:56:52<5167:37:50, 18.62s/it]G: 2.00 | MSG: -0.10 | VG: 0.00 | D: 1.12 | MSD: 2.00 | VD: 0.00 | GP: 0.16 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▍ | 1140/1000000 [6:03:13<5175:56:15, 18.65s/it]G: 1.11 | MSG: -0.06 | VG: 0.00 | D: 1.40 | MSD: 1.99 | VD: 0.00 | GP: 0.33 | SSL: 0.10 | CL: 0.00 | MAL: 0.00
0%|▍ | 1160/1000000 [6:09:34<5145:54:35, 18.55s/it]G: 1.68 | MSG: -0.13 | VG: 0.00 | D: 0.56 | MSD: 2.00 | VD: 0.00 | GP: 0.21 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▍ | 1180/1000000 [6:15:56<5192:49:53, 18.72s/it]G: 1.44 | MSG: -0.10 | VG: 0.00 | D: 0.47 | MSD: 2.01 | VD: 0.00 | GP: 0.51 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▍ | 1200/1000000 [6:22:19<5184:03:34, 18.69s/it]G: 4.70 | MSG: -0.17 | VG: 0.00 | D: 0.75 | MSD: 1.99 | VD: 0.00 | GP: 0.41 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▍ | 1220/1000000 [6:28:47<5166:22:25, 18.62s/it]G: 1.85 | MSG: -0.12 | VG: 0.00 | D: 0.75 | MSD: 1.99 | VD: 0.00 | GP: 0.39 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▍ | 1240/1000000 [6:35:08<5160:00:16, 18.60s/it]G: 1.20 | MSG: -0.15 | VG: 0.00 | D: 0.55 | MSD: 1.97 | VD: 0.00 | GP: 0.28 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▍ | 1260/1000000 [6:41:31<5110:53:03, 18.42s/it]G: 1.82 | MSG: -0.11 | VG: 0.00 | D: 0.56 | MSD: 2.00 | VD: 0.00 | GP: 0.14 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▍ | 1280/1000000 [6:47:52<5186:26:20, 18.70s/it]G: 2.64 | MSG: -0.08 | VG: 0.00 | D: 0.99 | MSD: 2.01 | VD: 0.00 | GP: 0.26 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▌ | 1300/1000000 [6:54:13<5128:58:44, 18.49s/it]G: 2.10 | MSG: -0.12 | VG: 0.00 | D: 0.81 | MSD: 1.99 | VD: 0.00 | GP: 0.18 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▌ | 1320/1000000 [7:00:40<5130:20:27, 18.49s/it]G: 1.94 | MSG: -0.12 | VG: 0.00 | D: 0.21 | MSD: 2.01 | VD: 0.00 | GP: 0.16 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▌ | 1340/1000000 [7:07:03<5173:57:42, 18.65s/it]G: 1.50 | MSG: -0.08 | VG: 0.00 | D: 0.74 | MSD: 2.00 | VD: 0.00 | GP: 0.15 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▌ | 1360/1000000 [7:13:25<5224:20:06, 18.83s/it]G: 2.46 | MSG: -0.13 | VG: 0.00 | D: 1.27 | MSD: 2.01 | VD: 0.00 | GP: 0.15 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▌ | 1380/1000000 [7:19:46<5169:37:13, 18.64s/it]G: 1.44 | MSG: -0.12 | VG: 0.00 | D: 0.77 | MSD: 2.00 | VD: 0.00 | GP: 0.23 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▌ | 1400/1000000 [7:26:03<5148:06:08, 18.56s/it]G: 2.29 | MSG: -0.11 | VG: 0.00 | D: 0.77 | MSD: 2.00 | VD: 0.00 | GP: 0.17 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▌ | 1420/1000000 [7:32:30<5110:59:38, 18.43s/it]G: 1.84 | MSG: -0.11 | VG: 0.00 | D: 1.08 | MSD: 2.02 | VD: 0.00 | GP: 0.60 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▌ | 1440/1000000 [7:38:50<5140:52:18, 18.53s/it]G: 1.58 | MSG: -0.12 | VG: 0.00 | D: 0.72 | MSD: 2.02 | VD: 0.00 | GP: 0.10 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▌ | 1460/1000000 [7:45:12<5228:16:55, 18.85s/it]G: 1.50 | MSG: -0.12 | VG: 0.00 | D: 2.00 | MSD: 2.00 | VD: 0.00 | GP: 0.20 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▌ | 1480/1000000 [7:51:32<5152:46:58, 18.58s/it]G: 2.79 | MSG: -0.08 | VG: 0.00 | D: 0.89 | MSD: 2.00 | VD: 0.00 | GP: 0.15 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▌ | 1500/1000000 [7:57:54<5159:04:34, 18.60s/it]G: 1.16 | MSG: -0.09 | VG: 0.00 | D: 0.78 | MSD: 1.99 | VD: 0.00 | GP: 0.54 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▌ | 1520/1000000 [8:04:24<5171:58:29, 18.65s/it]G: 1.75 | MSG: -0.16 | VG: 0.00 | D: 0.56 | MSD: 2.01 | VD: 0.00 | GP: 0.18 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▌ | 1540/1000000 [8:10:46<5165:00:54, 18.62s/it]G: 0.92 | MSG: -0.11 | VG: 0.00 | D: 0.85 | MSD: 2.00 | VD: 0.00 | GP: 0.17 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▌ | 1560/1000000 [8:17:10<5171:27:33, 18.65s/it]G: 2.19 | MSG: -0.11 | VG: 0.00 | D: 0.58 | MSD: 2.01 | VD: 0.00 | GP: 0.16 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▌ | 1580/1000000 [8:23:32<5185:30:11, 18.70s/it]G: 1.00 | MSG: -0.11 | VG: 0.00 | D: 1.38 | MSD: 1.99 | VD: 0.00 | GP: 0.30 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▌ | 1600/1000000 [8:29:54<5191:50:29, 18.72s/it]G: 2.36 | MSG: -0.09 | VG: 0.00 | D: 0.23 | MSD: 2.00 | VD: 0.00 | GP: 0.12 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▋ | 1620/1000000 [8:36:22<5171:53:51, 18.65s/it]G: 1.91 | MSG: -0.09 | VG: 0.00 | D: 1.35 | MSD: 2.00 | VD: 0.00 | GP: 0.09 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▋ | 1640/1000000 [8:42:43<5224:10:58, 18.84s/it]G: 2.38 | MSG: -0.13 | VG: 0.00 | D: 0.17 | MSD: 2.11 | VD: 0.00 | GP: 1.05 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1660/1000000 [8:49:07<5177:48:50, 18.67s/it]G: 2.00 | MSG: -0.14 | VG: 0.00 | D: 0.48 | MSD: 2.00 | VD: 0.00 | GP: 0.24 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▋ | 1680/1000000 [8:55:28<5175:57:21, 18.66s/it]G: 1.63 | MSG: -0.13 | VG: 0.00 | D: 0.22 | MSD: 1.97 | VD: 0.00 | GP: 0.34 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1700/1000000 [9:01:50<5187:14:49, 18.71s/it]G: nan | MSG: -0.11 | VG: 0.00 | D: nan | MSD: 2.01 | VD: 0.00 | GP: 0.16 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1720/1000000 [9:08:17<5097:13:27, 18.38s/it]G: nan | MSG: -0.11 | VG: 0.00 | D: nan | MSD: 2.01 | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1740/1000000 [9:14:39<5145:40:04, 18.56s/it]G: nan | MSG: -0.11 | VG: 0.00 | D: nan | MSD: 2.01 | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1760/1000000 [9:21:00<5183:13:37, 18.69s/it]G: nan | MSG: -0.11 | VG: 0.00 | D: nan | MSD: 2.01 | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1780/1000000 [9:27:20<5075:50:08, 18.31s/it]G: nan | MSG: -0.11 | VG: 0.00 | D: nan | MSD: 2.01 | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1800/1000000 [9:33:39<5155:04:00, 18.59s/it]G: nan | MSG: -0.11 | VG: 0.00 | D: nan | MSD: 2.01 | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1820/1000000 [9:40:04<5117:20:46, 18.46s/it]G: nan | MSG: -0.11 | VG: 0.00 | D: nan | MSD: 2.01 | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1840/1000000 [9:46:27<5167:02:25, 18.64s/it]G: nan | MSG: -0.11 | VG: 0.00 | D: nan | MSD: 2.01 | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1860/1000000 [9:52:49<5188:27:53, 18.71s/it]G: nan | MSG: -0.11 | VG: 0.00 | D: nan | MSD: 2.01 | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1880/1000000 [9:59:13<5253:48:29, 18.95s/it]G: nan | MSG: -0.11 | VG: 0.00 | D: nan | MSD: 2.01 | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1900/1000000 [10:05:36<5163:30:53, 18.62s/it]G: nan | MSG: -0.11 | VG: 0.00 | D: nan | MSD: 2.01 | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▋ | 1920/1000000 [10:12:04<5145:48:29, 18.56s/it]G: nan | MSG: -0.11 | VG: 0.00 | D: nan | MSD: 2.01 | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▊ | 1940/1000000 [10:18:25<5129:05:49, 18.50s/it]G: nan | MSG: -0.11 | VG: 0.00 | D: nan | MSD: 2.01 | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▊ | 1960/1000000 [10:24:48<5171:21:15, 18.65s/it]G: nan | MSG: -0.11 | VG: 0.00 | D: nan | MSD: 2.01 | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▊ | 1980/1000000 [10:31:08<5154:10:10, 18.59s/it]G: nan | MSG: -0.11 | VG: 0.00 | D: nan | MSD: 2.01 | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▊ | 2000/1000000 [10:37:33<5162:38:33, 18.62s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▊ | 2020/1000000 [10:44:01<5129:59:09, 18.51s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▊ | 2040/1000000 [10:50:24<5198:44:06, 18.75s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▊ | 2060/1000000 [10:56:46<5206:30:07, 18.78s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▊ | 2080/1000000 [11:03:08<5140:44:42, 18.55s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▊ | 2100/1000000 [11:09:31<5223:37:30, 18.84s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▊ | 2120/1000000 [11:15:57<5136:37:05, 18.53s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▊ | 2140/1000000 [11:22:19<5140:42:48, 18.55s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▊ | 2160/1000000 [11:28:39<5129:03:58, 18.50s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▊ | 2180/1000000 [11:35:01<5226:47:40, 18.86s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▊ | 2200/1000000 [11:41:21<5175:08:32, 18.67s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▊ | 2220/1000000 [11:47:47<5137:21:10, 18.54s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▊ | 2240/1000000 [11:54:06<5118:10:00, 18.47s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▉ | 2260/1000000 [12:00:29<5200:00:00, 18.76s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▉ | 2280/1000000 [12:06:50<5154:45:33, 18.60s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▉ | 2300/1000000 [12:13:14<5198:35:34, 18.76s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▉ | 2320/1000000 [12:19:39<5191:31:16, 18.73s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▉ | 2340/1000000 [12:26:00<5184:41:16, 18.71s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▉ | 2360/1000000 [12:32:20<5172:55:21, 18.67s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▉ | 2380/1000000 [12:38:41<5140:45:05, 18.55s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▉ | 2400/1000000 [12:45:01<5106:48:53, 18.43s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▉ | 2420/1000000 [12:51:28<5195:56:31, 18.75s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▉ | 2440/1000000 [12:57:50<5140:45:16, 18.55s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▉ | 2460/1000000 [13:04:13<5171:31:45, 18.66s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▉ | 2480/1000000 [13:10:40<5215:39:52, 18.82s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▉ | 2500/1000000 [13:17:03<5165:15:31, 18.64s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▉ | 2520/1000000 [13:23:30<5159:41:00, 18.62s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▉ | 2540/1000000 [13:29:55<5260:06:48, 18.98s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▉ | 2560/1000000 [13:36:18<5161:08:53, 18.63s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█ | 2580/1000000 [13:42:41<5183:42:26, 18.71s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█ | 2600/1000000 [13:49:03<5163:54:20, 18.64s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█ | 2620/1000000 [13:55:26<5174:34:58, 18.68s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█ | 2640/1000000 [14:01:48<5173:09:48, 18.67s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█ | 2660/1000000 [14:08:09<5139:19:50, 18.55s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█ | 2680/1000000 [14:14:30<5090:33:21, 18.38s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█ | 2700/1000000 [14:20:54<5202:06:53, 18.78s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█ | 2720/1000000 [14:27:17<5162:23:25, 18.64s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█ | 2740/1000000 [14:33:37<5099:42:08, 18.41s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█ | 2760/1000000 [14:39:59<5153:36:42, 18.60s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█ | 2780/1000000 [14:46:19<5132:04:30, 18.53s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█ | 2800/1000000 [14:52:39<5089:52:25, 18.37s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█ | 2820/1000000 [14:59:03<5139:38:18, 18.56s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█ | 2840/1000000 [15:05:25<5208:50:45, 18.81s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█ | 2860/1000000 [15:11:46<5150:03:57, 18.59s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█ | 2880/1000000 [15:18:07<5130:07:50, 18.52s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█▏ | 2900/1000000 [15:24:30<5211:33:26, 18.82s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█▏ | 2920/1000000 [15:30:52<5159:12:29, 18.63s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█▏ | 2940/1000000 [15:37:15<5181:02:20, 18.71s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█▏ | 2960/1000000 [15:43:36<5179:14:01, 18.70s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█▏ | 2980/1000000 [15:49:57<5137:19:16, 18.55s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█▏ | 3000/1000000 [15:56:20<5190:25:27, 18.74s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█▏ | 3020/1000000 [16:02:44<5116:23:12, 18.47s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█▏ | 3040/1000000 [16:09:07<5160:35:19, 18.63s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█▏ | 3060/1000000 [16:15:30<5183:08:05, 18.72s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█▏ | 3080/1000000 [16:21:50<5084:32:44, 18.36s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█▏ | 3100/1000000 [16:28:11<5158:50:43, 18.63s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█▏ | 3120/1000000 [16:34:34<5181:26:22, 18.71s/it]G: nan | MSG: nan | VG: 0.00 | D: nan | MSD: nan | VD: 0.00 | GP: 10.00 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|█▏ | 3125/1000000 [16:36:13<5463:29:38, 19.73s/it] 0%|█▏
from gigagan-pytorch.
lucidrains
commented on May 5, 2024
@sansavision after importing torch, could you run torch.autograd.set_detect_anomaly(True) and then when it NaN, paste the entire trace?
from gigagan-pytorch.
sansavision
commented on May 5, 2024
Hi @lucidrains ! Done, this is after 15H training using that set_detect_anomaly(True):
A100 GPU detected, using flash attention if input tensor is on cuda
Generator: 43.71M
Discriminator: 30.74M
Total images: 200000
0%| | 1/1000000 [00:00<?, ?it/s]G: 5.45 | MSG: -3.50 | VG: 0.00 | D: 3.13 | MSD: 5.62 | VD: 0.00 | GP: 0.00 | SSL: 8.31 | CL: 0.00 | MAL: 0.00
0%| | 20/1000000 [08:34<7286:56:06, 26.23s/it]G: 52.83 | MSG: 19.52 | VG: 0.00 | D: 0.00 | MSD: 0.59 | VD: 0.00 | GP: 19943.84 | SSL: 0.16 | CL: 0.00 | MAL: 0.00
0%| | 40/1000000 [17:29<7235:16:13, 26.05s/it]G: 223.25 | MSG: 18.89 | VG: 0.00 | D: 7.98 | MSD: 1.48 | VD: 0.00 | GP: 3601.65 | SSL: 1.00 | CL: 0.00 | MAL: 0.00
0%| | 60/1000000 [26:25<7308:12:38, 26.31s/it]G: 2.26 | MSG: -0.12 | VG: 0.00 | D: 8.49 | MSD: 2.15 | VD: 0.00 | GP: 1998.72 | SSL: 0.27 | CL: 0.00 | MAL: 0.00
0%| | 80/1000000 [35:21<7279:56:54, 26.21s/it]G: 61.50 | MSG: -0.52 | VG: 0.00 | D: 0.01 | MSD: 1.83 | VD: 0.00 | GP: 950.38 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%| | 100/1000000 [44:17<7256:30:13, 26.13s/it]G: 10.69 | MSG: -0.77 | VG: 0.00 | D: 0.03 | MSD: 2.36 | VD: 0.00 | GP: 757.33 | SSL: 0.21 | CL: 0.00 | MAL: 0.00
0%| | 120/1000000 [53:22<7394:54:31, 26.62s/it]G: 19.12 | MSG: 0.31 | VG: 0.00 | D: 0.00 | MSD: 1.87 | VD: 0.00 | GP: 1525.14 | SSL: 0.37 | CL: 0.00 | MAL: 0.00
0%| | 140/1000000 [1:02:20<7319:37:04, 26.35s/it]G: 5.06 | MSG: -0.39 | VG: 0.00 | D: 1.91 | MSD: 1.98 | VD: 0.00 | GP: 567.16 | SSL: 0.15 | CL: 0.00 | MAL: 0.00
0%| | 160/1000000 [1:11:15<7254:19:38, 26.12s/it]G: 27.77 | MSG: -0.33 | VG: 0.00 | D: 0.06 | MSD: 2.00 | VD: 0.00 | GP: 614.37 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%| | 180/1000000 [1:20:14<7356:00:34, 26.49s/it]G: 3.05 | MSG: -0.34 | VG: 0.00 | D: 1.59 | MSD: 2.03 | VD: 0.00 | GP: 463.66 | SSL: 0.12 | CL: 0.00 | MAL: 0.00
0%| | 200/1000000 [1:29:14<7321:18:25, 26.36s/it]G: 0.98 | MSG: -0.31 | VG: 0.00 | D: 1.30 | MSD: 1.98 | VD: 0.00 | GP: 241.51 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%| | 220/1000000 [1:38:19<7393:20:01, 26.62s/it]G: 6.35 | MSG: -0.39 | VG: 0.00 | D: 0.04 | MSD: 1.97 | VD: 0.00 | GP: 186.17 | SSL: 0.11 | CL: 0.00 | MAL: 0.00
0%| | 240/1000000 [1:47:16<7386:49:42, 26.60s/it]G: 13.48 | MSG: -0.31 | VG: 0.00 | D: 0.26 | MSD: 2.14 | VD: 0.00 | GP: 163.55 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%| | 260/1000000 [1:56:14<7216:54:11, 25.99s/it]G: 0.87 | MSG: -0.23 | VG: 0.00 | D: 1.03 | MSD: 1.99 | VD: 0.00 | GP: 99.92 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%| | 280/1000000 [2:05:14<7333:18:34, 26.41s/it]G: 7.12 | MSG: -0.23 | VG: 0.00 | D: 0.00 | MSD: 2.01 | VD: 0.00 | GP: 81.75 | SSL: 0.17 | CL: 0.00 | MAL: 0.00
0%| | 300/1000000 [2:14:12<7344:15:55, 26.45s/it]G: 14.10 | MSG: -0.25 | VG: 0.00 | D: 0.21 | MSD: 1.96 | VD: 0.00 | GP: 62.11 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%| | 320/1000000 [2:23:18<7388:52:16, 26.61s/it]G: 65.02 | MSG: -0.00 | VG: 0.00 | D: 0.20 | MSD: 2.21 | VD: 0.00 | GP: 61.11 | SSL: 0.96 | CL: 0.00 | MAL: 0.00
0%| | 340/1000000 [2:32:16<7354:02:31, 26.48s/it]G: -0.65 | MSG: -0.25 | VG: 0.00 | D: 2.93 | MSD: 2.00 | VD: 0.00 | GP: 28.09 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%| | 360/1000000 [2:41:13<7274:17:59, 26.20s/it]G: 9.41 | MSG: -0.23 | VG: 0.00 | D: 1.26 | MSD: 1.99 | VD: 0.00 | GP: 23.36 | SSL: 0.09 | CL: 0.00 | MAL: 0.00
0%| | 380/1000000 [2:50:10<7355:52:35, 26.49s/it]G: 50.07 | MSG: -0.25 | VG: 0.00 | D: 0.11 | MSD: 2.03 | VD: 0.00 | GP: 18.48 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%| | 400/1000000 [2:59:09<7358:36:06, 26.50s/it]G: 7.24 | MSG: -0.22 | VG: 0.00 | D: 0.29 | MSD: 2.01 | VD: 0.00 | GP: 14.09 | SSL: 0.10 | CL: 0.00 | MAL: 0.00
0%| | 420/1000000 [3:08:11<7275:43:35, 26.20s/it]G: 1.03 | MSG: -0.25 | VG: 0.00 | D: 1.51 | MSD: 2.00 | VD: 0.00 | GP: 12.15 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%| | 440/1000000 [3:17:14<7348:27:10, 26.47s/it]G: 10.70 | MSG: -0.25 | VG: 0.00 | D: 0.12 | MSD: 2.00 | VD: 0.00 | GP: 3.91 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%| | 460/1000000 [3:26:10<7339:52:12, 26.44s/it]G: 5.13 | MSG: -0.23 | VG: 0.00 | D: 0.26 | MSD: 2.04 | VD: 0.00 | GP: 5.08 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%| | 480/1000000 [3:35:08<7368:58:59, 26.54s/it]G: 3.85 | MSG: -0.26 | VG: 0.00 | D: 4.70 | MSD: 2.02 | VD: 0.00 | GP: 4.06 | SSL: 0.09 | CL: 0.00 | MAL: 0.00
0%| | 500/1000000 [3:44:03<7221:23:11, 26.01s/it]G: 0.94 | MSG: -0.25 | VG: 0.00 | D: 1.45 | MSD: 2.00 | VD: 0.00 | GP: 2.08 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%| | 520/1000000 [3:53:06<7286:34:11, 26.25s/it]G: 10.84 | MSG: -0.27 | VG: 0.00 | D: 1.74 | MSD: 2.03 | VD: 0.00 | GP: 1.63 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%| | 540/1000000 [4:02:03<7332:26:59, 26.41s/it]G: 3.04 | MSG: -0.25 | VG: 0.00 | D: 2.05 | MSD: 2.01 | VD: 0.00 | GP: 0.94 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%| | 560/1000000 [4:11:00<7280:41:46, 26.23s/it]G: 2.60 | MSG: -0.31 | VG: 0.00 | D: 0.03 | MSD: 2.03 | VD: 0.00 | GP: 1.21 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%| | 580/1000000 [4:19:54<7318:23:40, 26.36s/it]G: 1.70 | MSG: -0.27 | VG: 0.00 | D: 0.33 | MSD: 1.99 | VD: 0.00 | GP: 0.43 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%| | 600/1000000 [4:28:52<7311:32:44, 26.34s/it]G: 1.75 | MSG: -0.27 | VG: 0.00 | D: 0.61 | MSD: 2.01 | VD: 0.00 | GP: 0.44 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%| | 620/1000000 [4:37:57<7352:22:45, 26.48s/it]G: 0.20 | MSG: -0.28 | VG: 0.00 | D: 2.98 | MSD: 2.01 | VD: 0.00 | GP: 1.03 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%| | 640/1000000 [4:46:55<7314:49:58, 26.35s/it]G: 7.32 | MSG: -0.26 | VG: 0.00 | D: 1.04 | MSD: 2.00 | VD: 0.00 | GP: 0.54 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%| | 660/1000000 [4:55:51<7239:14:35, 26.08s/it]G: 1.67 | MSG: -0.27 | VG: 0.00 | D: 0.80 | MSD: 1.99 | VD: 0.00 | GP: 0.18 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 680/1000000 [5:04:46<7237:51:07, 26.07s/it]G: 4.09 | MSG: -0.29 | VG: 0.00 | D: 0.51 | MSD: 2.01 | VD: 0.00 | GP: 0.19 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 700/1000000 [5:13:45<7296:21:16, 26.29s/it]G: 1.57 | MSG: -0.26 | VG: 0.00 | D: 1.70 | MSD: 2.00 | VD: 0.00 | GP: 1.96 | SSL: 0.08 | CL: 0.00 | MAL: 0.00
0%|▏ | 720/1000000 [5:22:48<7276:16:40, 26.21s/it]G: 3.20 | MSG: -0.24 | VG: 0.00 | D: 0.08 | MSD: 1.99 | VD: 0.00 | GP: 0.16 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 740/1000000 [5:31:46<7255:13:07, 26.14s/it]G: 0.67 | MSG: -0.27 | VG: 0.00 | D: 1.61 | MSD: 2.00 | VD: 0.00 | GP: 0.17 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%|▏ | 760/1000000 [5:40:43<7255:42:58, 26.14s/it]G: 0.77 | MSG: -0.27 | VG: 0.00 | D: 0.77 | MSD: 2.01 | VD: 0.00 | GP: 1.01 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%|▏ | 780/1000000 [5:49:39<7240:15:04, 26.09s/it]G: 3.79 | MSG: -0.20 | VG: 0.00 | D: 0.19 | MSD: 1.99 | VD: 0.00 | GP: 0.14 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%|▏ | 800/1000000 [5:58:36<7238:12:38, 26.08s/it]G: 1.39 | MSG: -0.25 | VG: 0.00 | D: 2.55 | MSD: 2.01 | VD: 0.00 | GP: 0.09 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%|▏ | 820/1000000 [6:07:40<7253:06:47, 26.13s/it]G: 4.45 | MSG: -0.29 | VG: 0.00 | D: 2.04 | MSD: 2.00 | VD: 0.00 | GP: 0.33 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 840/1000000 [6:16:43<7312:47:18, 26.35s/it]G: 1.01 | MSG: -0.26 | VG: 0.00 | D: 1.21 | MSD: 2.00 | VD: 0.00 | GP: 0.39 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 860/1000000 [6:25:40<7276:56:18, 26.22s/it]G: 1.62 | MSG: -0.29 | VG: 0.00 | D: 0.95 | MSD: 1.99 | VD: 0.00 | GP: 0.79 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 880/1000000 [6:34:36<7314:04:12, 26.35s/it]G: 4.80 | MSG: -0.30 | VG: 0.00 | D: 0.46 | MSD: 2.03 | VD: 0.00 | GP: 0.37 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%|▏ | 900/1000000 [6:43:33<7316:47:52, 26.36s/it]G: 4.87 | MSG: -0.31 | VG: 0.00 | D: 0.39 | MSD: 2.03 | VD: 0.00 | GP: 0.13 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 920/1000000 [6:52:37<7317:22:39, 26.37s/it]G: 1.75 | MSG: -0.22 | VG: 0.00 | D: 0.21 | MSD: 1.99 | VD: 0.00 | GP: 0.25 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 940/1000000 [7:01:32<7289:19:58, 26.27s/it]G: 1.24 | MSG: -0.29 | VG: 0.00 | D: 0.97 | MSD: 2.01 | VD: 0.00 | GP: 0.25 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▏ | 960/1000000 [7:10:28<7269:00:02, 26.19s/it]G: 3.23 | MSG: -0.25 | VG: 0.00 | D: 1.08 | MSD: 2.00 | VD: 0.00 | GP: 0.16 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 980/1000000 [7:19:25<7302:15:59, 26.31s/it]G: 1.63 | MSG: -0.26 | VG: 0.00 | D: 0.27 | MSD: 2.00 | VD: 0.00 | GP: 0.11 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 1000/1000000 [7:28:24<7407:28:51, 26.69s/it]G: 1.37 | MSG: -0.22 | VG: 0.00 | D: 0.95 | MSD: 2.00 | VD: 0.00 | GP: 0.27 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 1020/1000000 [7:37:28<7275:00:59, 26.22s/it]G: 1.34 | MSG: -0.27 | VG: 0.00 | D: 1.47 | MSD: 1.99 | VD: 0.00 | GP: 0.12 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 1040/1000000 [7:46:20<7239:09:52, 26.09s/it]G: 1.42 | MSG: -0.28 | VG: 0.00 | D: 0.17 | MSD: 1.99 | VD: 0.00 | GP: 0.28 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 1060/1000000 [7:55:15<7317:32:33, 26.37s/it]G: 1.36 | MSG: -0.27 | VG: 0.00 | D: 0.26 | MSD: 2.01 | VD: 0.00 | GP: 0.47 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 1080/1000000 [8:04:12<7312:33:22, 26.35s/it]G: 1.34 | MSG: -0.28 | VG: 0.00 | D: 0.72 | MSD: 2.01 | VD: 0.00 | GP: 0.21 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▏ | 1100/1000000 [8:13:09<7301:29:33, 26.31s/it]G: 1.90 | MSG: -0.28 | VG: 0.00 | D: 0.72 | MSD: 2.01 | VD: 0.00 | GP: 0.20 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 1120/1000000 [8:22:13<7267:25:29, 26.19s/it]G: 2.14 | MSG: -0.27 | VG: 0.00 | D: 3.22 | MSD: 2.00 | VD: 0.00 | GP: 0.14 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 1140/1000000 [8:31:08<7256:37:42, 26.15s/it]G: 0.96 | MSG: -0.28 | VG: 0.00 | D: 0.78 | MSD: 2.01 | VD: 0.00 | GP: 0.75 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 1160/1000000 [8:40:04<7317:56:48, 26.38s/it]G: 1.64 | MSG: -0.28 | VG: 0.00 | D: 0.59 | MSD: 1.99 | VD: 0.00 | GP: 0.16 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 1180/1000000 [8:49:02<7244:56:13, 26.11s/it]G: 1.75 | MSG: -0.27 | VG: 0.00 | D: 0.62 | MSD: 2.02 | VD: 0.00 | GP: 0.16 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%|▏ | 1200/1000000 [8:57:58<7240:24:30, 26.10s/it]G: 2.38 | MSG: -0.29 | VG: 0.00 | D: 0.49 | MSD: 2.00 | VD: 0.00 | GP: 0.15 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 1220/1000000 [9:07:02<7278:57:15, 26.24s/it]G: 1.52 | MSG: -0.28 | VG: 0.00 | D: 0.76 | MSD: 2.00 | VD: 0.00 | GP: 0.17 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 1240/1000000 [9:15:58<7255:45:47, 26.15s/it]G: 1.37 | MSG: -0.32 | VG: 0.00 | D: 0.62 | MSD: 1.99 | VD: 0.00 | GP: 0.46 | SSL: 0.07 | CL: 0.00 | MAL: 0.00
0%|▏ | 1260/1000000 [9:24:54<7267:03:04, 26.19s/it]G: 1.80 | MSG: -0.30 | VG: 0.00 | D: 0.81 | MSD: 2.00 | VD: 0.00 | GP: 0.20 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▏ | 1280/1000000 [9:33:49<7255:07:22, 26.15s/it]G: 1.55 | MSG: -0.29 | VG: 0.00 | D: 0.46 | MSD: 1.98 | VD: 0.00 | GP: 0.16 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▏ | 1300/1000000 [9:42:45<7302:47:21, 26.32s/it]G: 1.78 | MSG: -0.30 | VG: 0.00 | D: 0.44 | MSD: 2.01 | VD: 0.00 | GP: 0.19 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▏ | 1320/1000000 [9:51:48<7313:09:58, 26.36s/it]G: 0.71 | MSG: -0.31 | VG: 0.00 | D: 0.96 | MSD: 1.98 | VD: 0.00 | GP: 1.12 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▏ | 1340/1000000 [10:00:45<7322:31:09, 26.40s/it]G: 1.02 | MSG: -0.33 | VG: 0.00 | D: 0.78 | MSD: 2.01 | VD: 0.00 | GP: 0.82 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1360/1000000 [10:09:42<7289:33:37, 26.28s/it]G: 1.20 | MSG: -0.33 | VG: 0.00 | D: 0.80 | MSD: 2.01 | VD: 0.00 | GP: 0.31 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1380/1000000 [10:18:36<7214:40:26, 26.01s/it]G: 1.59 | MSG: -0.31 | VG: 0.00 | D: 0.73 | MSD: 1.99 | VD: 0.00 | GP: 0.14 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 1400/1000000 [10:27:33<7267:04:04, 26.20s/it]G: 1.20 | MSG: -0.33 | VG: 0.00 | D: 0.60 | MSD: 2.00 | VD: 0.00 | GP: 0.12 | SSL: 0.05 | CL: 0.00 | MAL: 0.00
0%|▎ | 1420/1000000 [10:36:34<7178:12:51, 25.88s/it]G: 1.39 | MSG: -0.34 | VG: 0.00 | D: 0.75 | MSD: 1.99 | VD: 0.00 | GP: 1.22 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1440/1000000 [10:45:32<7311:57:32, 26.36s/it]G: 1.39 | MSG: -0.33 | VG: 0.00 | D: 0.70 | MSD: 1.99 | VD: 0.00 | GP: 0.40 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 1460/1000000 [10:54:28<7244:35:26, 26.12s/it]G: 0.98 | MSG: -0.35 | VG: 0.00 | D: 0.89 | MSD: 2.00 | VD: 0.00 | GP: 1.45 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1480/1000000 [11:03:24<7291:00:55, 26.29s/it]G: 1.65 | MSG: -0.34 | VG: 0.00 | D: 0.32 | MSD: 2.01 | VD: 0.00 | GP: 0.52 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1500/1000000 [11:12:21<7232:54:12, 26.08s/it]G: 2.14 | MSG: -0.33 | VG: 0.00 | D: 0.45 | MSD: 1.99 | VD: 0.00 | GP: 0.25 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1520/1000000 [11:21:23<7324:17:03, 26.41s/it]G: 1.51 | MSG: -0.33 | VG: 0.00 | D: 0.81 | MSD: 2.00 | VD: 0.00 | GP: 0.66 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1540/1000000 [11:30:20<7246:17:08, 26.13s/it]G: 2.26 | MSG: -0.33 | VG: 0.00 | D: 0.58 | MSD: 2.03 | VD: 0.00 | GP: 0.11 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1560/1000000 [11:39:13<7219:56:08, 26.03s/it]G: 1.62 | MSG: -0.33 | VG: 0.00 | D: 0.88 | MSD: 2.00 | VD: 0.00 | GP: 0.22 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 1580/1000000 [11:48:10<7310:45:27, 26.36s/it]G: 1.38 | MSG: -0.35 | VG: 0.00 | D: 0.88 | MSD: 1.99 | VD: 0.00 | GP: 0.15 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1600/1000000 [11:57:04<7233:41:52, 26.08s/it]G: 0.88 | MSG: -0.35 | VG: 0.00 | D: 0.84 | MSD: 1.98 | VD: 0.00 | GP: 1.55 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 1620/1000000 [12:06:07<7216:09:00, 26.02s/it]G: 1.53 | MSG: -0.36 | VG: 0.00 | D: 0.67 | MSD: 2.01 | VD: 0.00 | GP: 0.10 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1640/1000000 [12:15:04<7283:14:03, 26.26s/it]G: 1.45 | MSG: -0.36 | VG: 0.00 | D: 0.90 | MSD: 2.00 | VD: 0.00 | GP: 0.11 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1660/1000000 [12:24:01<7223:11:07, 26.05s/it]G: 1.31 | MSG: -0.36 | VG: 0.00 | D: 0.76 | MSD: 1.98 | VD: 0.00 | GP: 6.51 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1680/1000000 [12:32:59<7284:35:36, 26.27s/it]G: 1.66 | MSG: -0.40 | VG: 0.00 | D: 0.74 | MSD: 2.03 | VD: 0.00 | GP: 0.08 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1700/1000000 [12:41:58<7268:55:41, 26.21s/it]G: 0.91 | MSG: -0.34 | VG: 0.00 | D: 1.19 | MSD: 2.05 | VD: 0.00 | GP: 0.70 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 1720/1000000 [12:51:02<7298:28:04, 26.32s/it]G: 1.88 | MSG: -0.35 | VG: 0.00 | D: 0.73 | MSD: 2.01 | VD: 0.00 | GP: 0.08 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1740/1000000 [12:59:58<7280:34:43, 26.26s/it]G: 1.50 | MSG: -0.34 | VG: 0.00 | D: 0.86 | MSD: 2.00 | VD: 0.00 | GP: 0.14 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1760/1000000 [13:08:53<7273:13:37, 26.23s/it]G: 1.50 | MSG: -0.37 | VG: 0.00 | D: 0.37 | MSD: 1.98 | VD: 0.00 | GP: 0.55 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1780/1000000 [13:17:50<7279:54:09, 26.25s/it]G: 0.63 | MSG: -0.37 | VG: 0.00 | D: 0.95 | MSD: 1.99 | VD: 0.00 | GP: 1.47 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1800/1000000 [13:26:46<7283:15:55, 26.27s/it]G: 1.93 | MSG: -0.36 | VG: 0.00 | D: 0.69 | MSD: 2.00 | VD: 0.00 | GP: 0.19 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1820/1000000 [13:35:44<7292:57:10, 26.30s/it]G: 0.91 | MSG: -0.36 | VG: 0.00 | D: 0.63 | MSD: 2.00 | VD: 0.00 | GP: 0.69 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1840/1000000 [13:44:40<7262:32:53, 26.19s/it]G: 1.40 | MSG: -0.36 | VG: 0.00 | D: 0.66 | MSD: 2.01 | VD: 0.00 | GP: 0.18 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1860/1000000 [13:53:36<7310:45:46, 26.37s/it]G: 2.67 | MSG: -0.33 | VG: 0.00 | D: 0.76 | MSD: 1.99 | VD: 0.00 | GP: 0.07 | SSL: 0.04 | CL: 0.00 | MAL: 0.00
0%|▎ | 1880/1000000 [14:02:32<7220:21:52, 26.04s/it]G: 2.36 | MSG: -0.37 | VG: 0.00 | D: 0.37 | MSD: 2.01 | VD: 0.00 | GP: 0.46 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1900/1000000 [14:11:28<7295:34:51, 26.31s/it]G: 3.30 | MSG: -0.31 | VG: 0.00 | D: 0.09 | MSD: 1.99 | VD: 0.00 | GP: 0.56 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1920/1000000 [14:20:31<7273:53:47, 26.24s/it]G: 1.58 | MSG: -0.37 | VG: 0.00 | D: 0.76 | MSD: 2.00 | VD: 0.00 | GP: 0.21 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1940/1000000 [14:29:26<7302:06:18, 26.34s/it]G: 2.07 | MSG: -0.37 | VG: 0.00 | D: 0.56 | MSD: 2.00 | VD: 0.00 | GP: 0.07 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 1960/1000000 [14:38:23<7356:45:25, 26.54s/it]G: 1.99 | MSG: -0.37 | VG: 0.00 | D: 0.58 | MSD: 2.00 | VD: 0.00 | GP: 0.08 | SSL: 0.06 | CL: 0.00 | MAL: 0.00
0%|▎ | 1980/1000000 [14:47:21<7271:23:30, 26.23s/it]G: 1.10 | MSG: -0.38 | VG: 0.00 | D: 0.74 | MSD: 2.01 | VD: 0.00 | GP: 0.28 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 2000/1000000 [14:56:18<7280:28:53, 26.26s/it]G: 1.60 | MSG: -0.35 | VG: 0.00 | D: 1.02 | MSD: 2.00 | VD: 0.00 | GP: 0.15 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▎ | 2020/1000000 [15:05:18<7240:14:33, 26.12s/it]G: 0.86 | MSG: -0.38 | VG: 0.00 | D: 0.92 | MSD: 2.00 | VD: 0.00 | GP: 0.38 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▍ | 2040/1000000 [15:14:15<7312:43:22, 26.38s/it]G: 1.69 | MSG: -0.38 | VG: 0.00 | D: 0.41 | MSD: 2.00 | VD: 0.00 | GP: 0.21 | SSL: 0.03 | CL: 0.00 | MAL: 0.00
0%|▍ | 2048/1000000 [15:17:49<7262:27:42, 26.20s/it]/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/autograd/__init__.py:200: UserWarning: Error detected in AddmmBackward0. Traceback of forward call that caused the error:
File "/home/ubuntu/giga/train.py", line 49, in <module>
gan(
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/gigagan_pytorch/gigagan_pytorch.py", line 2571, in forward
) = self.train_generator_step(
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/gigagan_pytorch/gigagan_pytorch.py", line 2401, in train_generator_step
logits, multiscale_logits, _ = self.D(
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/accelerate/utils/operations.py", line 632, in forward
return model_forward(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/accelerate/utils/operations.py", line 620, in __call__
return convert_to_fp32(self.model_forward(*args, **kwargs))
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
return func(*args, **kwargs)
File "<@beartype(gigagan_pytorch.gigagan_pytorch.Discriminator.forward) at 0x7ffa9dafa0c0>", line 94, in forward
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/gigagan_pytorch/gigagan_pytorch.py", line 1721, in forward
logits = self.to_logits(x)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
(Triggered internally at /opt/conda/conda-bld/pytorch_1682343995622/work/torch/csrc/autograd/python_anomaly_mode.cpp:114.)
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
0%|▍ | 2048/1000000 [15:18:07<7460:04:57, 26.91s/it]
Traceback (most recent call last):
File "/home/ubuntu/giga/train.py", line 49, in <module>
gan(
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/gigagan_pytorch/gigagan_pytorch.py", line 2571, in forward
) = self.train_generator_step(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/gigagan_pytorch/gigagan_pytorch.py", line 2441, in train_generator_step
self.accelerator.backward(total_loss / grad_accum_every, retain_graph = self.need_contrastive_loss)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/accelerate/accelerator.py", line 1921, in backward
self.scaler.scale(loss).backward(**kwargs)
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/_tensor.py", line 487, in backward
torch.autograd.backward(
File "/home/ubuntu/anaconda3/envs/giga/lib/python3.11/site-packages/torch/autograd/__init__.py", line 200, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: Function 'AddmmBackward0' returned nan values in its 2th output.
from gigagan-pytorch.
lucidrains
commented on May 5, 2024
@sansavision ah thank you, this helps a bit
all the numbers seem healthy up to that point
did you run this on the updated 0.2.12 or the same version as when you opened the issue?
from gigagan-pytorch.
lucidrains
commented on May 5, 2024
@sansavision it is a shot in the dark, but would narrow it down even if it fails
from gigagan-pytorch.
lucidrains
commented on May 5, 2024
@sansavision thanks, it is helpful, and indeed a NaN arose from the gradient penalty code like i initially thought. maybe there are two issues going on haha
if you have another stretch of time, try running it without the detect anomaly, as i protect against NaN for gradient penalty since v0.2.14 (but there may be another issue in the discriminator)
from gigagan-pytorch.
lucidrains
commented on May 5, 2024
yeah, it would help! thanks!
from gigagan-pytorch.
Ahmed3wais
commented on May 5, 2024
I faced the same issue, any fixes yet?
from gigagan-pytorch.
axel578
commented on May 5, 2024
Same issue for me
from gigagan-pytorch.
axel578
commented on May 5, 2024
@lucidrains any reasons it could lead to that issue ?
from gigagan-pytorch.
Related Issues (20)
- Possible Discrepancies HOT 3
- The training code not deal with paired data yet? HOT 2
- [Question] About the upscaler HOT 2
- Multi GPU training HOT 4
- Multi GPU with gradient accumulation
- [Request] Please provide a replicate.com version
- Confused about this project?
- How to implement this model to enhance my input images? Do I have to train the model to use? HOT 2
- Weights of Gigagan Upscaler HOT 1
- Turn on/off gradients computation between generator/discriminator HOT 2
- Wrong order of resolutions list HOT 1
- to_rgb branch has only 1 learnable kernel HOT 7
- Gradient Penalty is very high in the start HOT 10
- How to use this model for SR ?
- Has Anyone Trained This Model Yet? HOT 2
- The text-to-image tasks
- Config to reproduce paper
- question about code in unet_upsampler.py HOT 1
- the loss became nan after a few train steps HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gigagan-pytorch.