Comments (7)
@kyquang97 Luong takes the last context vector and concatenates them with the last output vector as an input to RNN. The output from RNN will be passed to Attention layer to calculate the context vector in the current time step. The current context vector combines with the current output from RNN will be calculated to the current output for this time step. Please noted that the current context vector will be passed to the next time step.
from practical-pytorch.
@beebrain Please correct me if I'm wrong, but you are using the Lstm layer, instead of the lstm cell, so when each forward pass happens its a different sample not a different timestep on a single sample. You have no control over each timesteps separately here. what you get out of the RNN in this configuration is just a translation/sequence that has already gone through all timesteps!
from practical-pytorch.
@kyquang97 Luong takes the last context vector and concatenates them with the last output vector as an input to RNN. The output from RNN will be passed to Attention layer to calculate the context vector in the current time step. The current context vector combines with the current output from RNN will be calculated to the current output for this time step. Please noted that the current context vector will be passed to the next time step.
I think he just meant that in this implementation, the rnn_output
is fed into the attention layer instead of the current decoder hidden state, which is inconsistent with the description in the original paper.
from practical-pytorch.
@kyquang97 Luong takes the last context vector and concatenates them with the last output vector as an input to RNN. The output from RNN will be passed to Attention layer to calculate the context vector in the current time step. The current context vector combines with the current output from RNN will be calculated to the current output for this time step. Please noted that the current context vector will be passed to the next time step.
I think he just meant that in this implementation, the
rnn_output
is fed into the attention layer instead of the current decoder hidden state, which is inconsistent with the description in the original paper.
I think the rnn_output and hidden output of self.gru had the same value. You can use hidden or rnn_output.
from practical-pytorch.
@kyquang97 Luong takes the last context vector and concatenates them with the last output vector as an input to RNN. The output from RNN will be passed to Attention layer to calculate the context vector in the current time step. The current context vector combines with the current output from RNN will be calculated to the current output for this time step. Please noted that the current context vector will be passed to the next time step.
I think he just meant that in this implementation, the
rnn_output
is fed into the attention layer instead of the current decoder hidden state, which is inconsistent with the description in the original paper.I think the rnn_output and hidden output of self.gru had the same value. You can use hidden or rnn_output.
You do remind me! I'm also confused by the usage of outputs and hidden states in some attention implementations at first and they do actually share the same values. BTW, what about the LSTM? From Pytorch doc, the LSTM outputs hidden states as well as cell states. Are cell states used in attention or can I just consider using outputs and last hidden states equally?
from practical-pytorch.
@kyquang97 Luong takes the last context vector and concatenates them with the last output vector as an input to RNN. The output from RNN will be passed to Attention layer to calculate the context vector in the current time step. The current context vector combines with the current output from RNN will be calculated to the current output for this time step. Please noted that the current context vector will be passed to the next time step.
I think he just meant that in this implementation, the
rnn_output
is fed into the attention layer instead of the current decoder hidden state, which is inconsistent with the description in the original paper.I think the rnn_output and hidden output of self.gru had the same value. You can use hidden or rnn_output.
You do remind me! I'm also confused by the usage of outputs and hidden states in some attention implementations at first and they do actually share the same values. BTW, what about the LSTM? From Pytorch doc, the LSTM outputs hidden states as well as cell states. Are cell states used in attention or can I just consider using outputs and last hidden states equally?
In my opinion, You can use the hidden state output like GRU.
from practical-pytorch.
I am also confused about why we can calculate all the attention scores for the source sentence using the previous hidden state and current input embedding.
from practical-pytorch.
Related Issues (20)
- Issue on Windows
- Seq-seq not working for creating chatbot
- How to save and load train model and use it for evaluation HOT 2
- Link for Series 2 - RNNs for time-series data
- can't import torch HOT 2
- The link for Teacher Forcing in "Translation with a Sequence to Sequence Network and Attention" is broken
- Error in BahdanauAttnDecoderRNN HOT 1
- Issues in your tutorial on Classifying Names with a Character-Level RNN
- I can't calculate the score of attention in Seq2Seq Translation. HOT 2
- Error in practical-pytorch/seq2seq-translation/seq2seq-translation-batched.ipynb
- Question from character level RNN classifier, why not use the hidden state across epochs? HOT 1
- RuntimeError: 1D tensors expected, got 2D, 2D tensors HOT 1
- May I know how to support a new sentence translation?
- seq2seq: Replace the embeddings with pre-trained word embeddings such as word2vec
- About seq2seq-translation-batched.py RuntimeError HOT 1
- Wrong implementation of attention mechanism in pytorch tutorials
- FileNotFoundError: [Errno 2] No such file or directory: 'char-rnn-classification.pt'
- small format issue
- Implicit dimension choice for log_softmax has been deprecated while running 'python train.py'
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from practical-pytorch.