drsy / emo Goto Github PK

代码中提供了三种DEMD和MLE的融合方式, 请问论文中使用的是哪一种呐?

Dear authors:
Thank you for your great work advancing the frontier of language model training.

Learning with a Wasserstein Loss (arxiv)
Learning with a Wasserstein Loss (NeurIPS 2015)

This paper represents the first use of the Wasserstein distance (i.e. earth mover’s distance) as a loss for supervised learning.
It considers the problem of learning to predict a non-negative measure over a finite set.
Language Models are essentially learning to predict a non-negative measure over a finite set.

In summary, Learning with a Wasserstein Loss has used a similar method to solve a similar problem. The core idea and core technique are the same, the problem is the same in principle.

Nonetheless, your great work proposes a tractable and effective upper bound for EMD and verifies EMD's effectiveness in language model fine-tuning, which is nontrivial and impressive.

Could you please, by any chance, cite Learning with a Wasserstein Loss in the camera-ready version paper? I believe that will help readers to find related works.

本来想在OpenReview发个Public Comment，但现在不能发了😂

您好，请教一下。
我再qwen 上使用这个loss，进行的是下游任务的微调，原本的loss 是 mle。
使用的是： EMO/language_modeling/gpt2.py 中的emo 。

他在训练的过程中会出现这样的现象：

以及：

请问这个情况大概是为什么？

gt_q = (q_grad * one_hot).detach()
q_final = q_grad - gt_q

理论上改完之后loss的值应该不变，但现在使用公式$1 − ( EQ_{\theta})^TEP$算出来修改之前和之后的结果不同，又验证了公式$Q_{\theta}^TCP$，修改前后相同，应该是$Q_{\theta}$本身发生了变化导致$Q_{\theta}^TP$的值不等于1了，所以
emo_loss = (1 - torch.sum(p_contextual_repr * q_contextual_repr, dim=-1))是不是需要再减去一项 torch.sum(gt_q * stable_onehot, dim=-1)？

另外，在ground-truth token处因为$1-E^TE$对角线一直为0，梯度本身也会乘以这个值，相当于ground-truth token处的梯度为0，也不需要更新梯度，这两行代码是否还有存在的必要？

Hi, Thank you for sharing this work.

I wonder the applicability of this method for pretraining from-scratch.
To do that, one needs to build a good initial cost matrix $\mathbb{C} = [C(v_i, v_j)]_{i,j}$. I can come up with a trivial uniform initialization for this. Is there a better way than this?

If you ever tried this method for pretraining from scratch, it would be really helpful to know the results. Thanks!

gt_q = (q_grad * one_hot).detach()
q_final = q_grad - gt_q

这两行代码目的是什么，论文中未提到相关内容？

在我自己的代码中引入 emo, 使用的 bf16 训练, 训练过程中 loss 会变负, 可能是什么原因呢

loss: -2.334918e-02, loss_cur_dp: -2.334918e-02

你好，这篇论文很有启发性，从代码可以看出相当于加入了修正的cos loss。而这个是否会导致模型倾向于给相同语义的token类似的权重，比如前面的字是“今天是周三，明天是”， 后面会倾向于给出“周四”，而“星期四”和“周四”的语义类似，经过emo loss训练的模型给出“星期四”的logit和“周四”接近一致，这样语义相似且logit高的token占了大部分概率，导致模型倾向于输出单一的风格，是否会存在这个问题呢？
@DRSY