Simplifying weigh decay checking doesn't work about mingpt HOT 3 CLOSED

This is the test case matrix where the modified code behaves differently compared with the original code. The linear module maps to the whitelist and the rest belongs to the blacklist. This is generally the idea behind it.

from mingpt.

modifying the else should help fix it.

code:

decay = set()
no_decay = set()
blacklist_weight_modules = (torch.nn.LayerNorm, torch.nn.Embedding)
for mn, m in self.named_modules():
for pn, p in m.named_parameters():
fpn = f"{mn}.{pn}" if mn else pn # full param name
if pn.endswith("bias") or isinstance(m, blacklist_weight_modules):
no_decay.add(fpn)
elif pn.endswith("weight"):
decay.add(fpn)

from mingpt.

modifying the else should help fix it.

code:

decay = set() no_decay = set() blacklist_weight_modules = (torch.nn.LayerNorm, torch.nn.Embedding) for mn, m in self.named_modules(): for pn, p in m.named_parameters(): fpn = f"{mn}.{pn}" if mn else pn # full param name if pn.endswith("bias") or isinstance(m, blacklist_weight_modules): no_decay.add(fpn) elif pn.endswith("weight"): decay.add(fpn)

Thanks, missed the logic there.

from mingpt.