Comments (6)
- pegasus model masked several tokens in input and let encoder to predict masked tokens as MLM loss. However, the paper stated that the pegasus_large model deleted the MLM loss part. Does that mean the final model takes masked inputs to the encoder and decoder leveraged shifted target text and encoder outputs to generate target text only using the next token prediction loss? If so, what is the difference between the pegasus pretrained model and BART model (BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension). The BART model also takes noised text input to their encoder and uses the next token prediction loss for decoder outputs.
Yes, in the final model, only GSG is applied and there is no MLM.
Document: ABCDE
- BART: input: A_C_E, target: ABCDE
- PEGASUS: input: A_C_E, target: BD
- What is the strategy for selecting target sentences as shown in Fig.1? Is that the Gap Sentence Generation procedure?
Fig 1 demonstrates both GSG and MLM. There is no specific sentence selection strategy for Fig 1 as the figure is only for demo purpose. Any strategy (e.g. random, lead or principal) would fit the procedure shown in Fig 1.
Hope this may answer your questions. Thanks!
from pegasus.
-
I do not understand " PEGASUS: input: A_C_E, target: BD".
As you deleted MLM loss, if the input for the decoder is " ABCDE" then the target would be "ABCDE". Then what is your "target: BD"? -
In fig1, I don't understand how to select input text for encoder and decoder?
In this example, your text is "Pegasus if mythical. It is pure white. It names the model."
The input for the encoder is masked "Pegasus if mythical. It names the model."
The input for the decoder is " It is pure white."
The target for the decoder is "It is pure while. "
If I have a paragraph, how to select sentences for encoder input and decoder input?
from pegasus.
-
Each letter actually represents one whole sentence. So the target "BD" has two sentences B and D, which are masked in the input.
-
How to select sentences for encoder input and decoder input? The sentence selection strategies (from a document) are described in Section 3.1 GSG in the paper.
from pegasus.
I still don't understand. If our paragraph is "ABCDE ":
From BART paper, their encoder input is "A_B_E", the decoder input is "ABCD", the decoder output is "ABCDE".
Let's take this for example, what is your encoder input and decoder input and output?
from pegasus.
As mentioned above,
Document: ABCDE
- BART: encoder input: A_C_E, decoder input/target: ABCDE (it seems BART masks tokens or span, not sentences)
- PEGASUS: encoder input: A_C_E, decoder input/target: BD (we mask sentences not tokens)
Decoder input and decoder output (target) are mostly same, except shifted right (i.e. insert token <START> in decoder input).
from pegasus.
Thanks
from pegasus.
Related Issues (20)
- delete it please
- Can anyone recommend a course or book
- Gap Sentences Generation
- Training Time HOT 1
- make it work keeping HTML tags from text
- unable to use pegasus-x checkpoints
- how does fine tuning change the original transformer language model?
- mask token id == 3 during pretraining?
- 2 HOT 1
- www.%20%20%20%20%20http.com//www.apache.org/licenses/LICENSE-2.0
- How to pre-train the pegasus language model?
- Usage as extractive summarizer
- PEGASUS-X in JAX/Flax requirements
- Version of Jax and Flax
- Is the pegasus-cnn_dailymail on Huggingface finetuned on all of CNN or only on the 'train' subset of the whole data
- Broken link to PegasusXLarge arXiv checkpoint HOT 1
- Missing CNN/DailyMail checkpoint for Pegasus-X HOT 1
- CheckSumError HOT 3
- Could you provide requirements file please?
- Getting error while following steps for training and evaluating
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pegasus.