Pretrain pegasus about pegasus HOT 6 CLOSED

1. pegasus model masked several tokens in input and let encoder to predict masked tokens as MLM loss. However, the paper stated that the pegasus_large model deleted the MLM loss part. Does that mean the final model takes masked inputs to the encoder and decoder leveraged shifted target text and encoder outputs to generate target text only using the next token prediction loss? If so, what is the difference between the pegasus pretrained model and BART model (BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension). The BART model also takes noised text input to their encoder and uses the next token prediction loss for decoder outputs.

Yes, in the final model, only GSG is applied and there is no MLM.

Document: ABCDE

• BART: input: A_C_E, target: ABCDE
• PEGASUS: input: A_C_E, target: BD

2. What is the strategy for selecting target sentences as shown in Fig.1? Is that the Gap Sentence Generation procedure?

Fig 1 demonstrates both GSG and MLM. There is no specific sentence selection strategy for Fig 1 as the figure is only for demo purpose. Any strategy (e.g. random, lead or principal) would fit the procedure shown in Fig 1.

Hope this may answer your questions. Thanks!

from pegasus.

1. I do not understand " PEGASUS: input: A_C_E, target: BD".
   As you deleted MLM loss, if the input for the decoder is " ABCDE" then the target would be "ABCDE". Then what is your "target: BD"?

2. In fig1, I don't understand how to select input text for encoder and decoder?
   In this example, your text is "Pegasus if mythical. It is pure white. It names the model."
   The input for the encoder is masked "Pegasus if mythical. It names the model."
   The input for the decoder is " It is pure white."
   The target for the decoder is "It is pure while. "

If I have a paragraph, how to select sentences for encoder input and decoder input?

from pegasus.

1. Each letter actually represents one whole sentence. So the target "BD" has two sentences B and D, which are masked in the input.

2. How to select sentences for encoder input and decoder input? The sentence selection strategies (from a document) are described in Section 3.1 GSG in the paper.

from pegasus.

I still don't understand. If our paragraph is "ABCDE ":

From BART paper, their encoder input is "A_B_E", the decoder input is "ABCD", the decoder output is "ABCDE".

Let's take this for example, what is your encoder input and decoder input and output?

from pegasus.

As mentioned above,

Document: ABCDE

• BART: encoder input: A_C_E, decoder input/target: ABCDE (it seems BART masks tokens or span, not sentences)
• PEGASUS: encoder input: A_C_E, decoder input/target: BD (we mask sentences not tokens)

Decoder input and decoder output (target) are mostly same, except shifted right (i.e. insert token <START> in decoder input).

from pegasus.

Thanks

from pegasus.