mindspore-nlp-tutorial

mindspore-nlp-tutorial is a tutorial for who is studying NLP(Natural Language Processing) using MindSpore. This repository is migrated from nlp-tutorial. Most of the models in NLP were migrated from Pytorch version with less than 100 lines of code.(except comments or blank lines)

Notice: All models are tested on CPU(Linux and macOS), GPU and Ascend.

Curriculum - (Example Purpose)

1. Basic Embedding Model

1-1. NNLM(Neural Network Language Model) - Predict Next Word
- Paper - A Neural Probabilistic Language Model(2003)
1-2. Word2Vec(Skip-gram) - Embedding Words and Show Graph
- Paper - Distributed Representations of Words and Phrases and their Compositionality(2013)

2. CNN(Convolutional Neural Network)

2-1. TextCNN - Binary Sentiment Classification
- Paper - Convolutional Neural Networks for Sentence Classification(2014)

3. RNN(Recurrent Neural Network)

3-1. TextRNN - Predict Next Step
- Paper - Finding Structure in Time(1990)
3-2. TextLSTM - Autocomplete
- Paper - LONG SHORT-TERM MEMORY(1997)
3-3. Bi-LSTM - Predict Next Word in Long Sentence

4. Attention Mechanism

4-1. Seq2Seq - Change Word
- Paper - Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation(2014)
4-2. Seq2Seq with Attention - Translate
- Paper - Neural Machine Translation by Jointly Learning to Align and Translate(2014)
4-3. Bi-LSTM with Attention - Binary Sentiment Classification

5. Model based on Transformer

5-1. The Transformer - Translate
- Paper - Attention Is All You Need(2017)
5-2. BERT - Classification Next Sentence & Predict Masked Tokens
- Paper - BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding(2018)

Dependencies

Python >= 3.7.5
MindSpore 1.9.0
Pytorch 1.7.1(for comparation)

Author

Yufeng Lyu
Author Email : [email protected]
Acknowledgements to graykode who opensource the Pytorch and Tensorflow version.

5-1.Transformer raise error

mindspore 1.8.1 how to slove the problem?

from mindspore import context
context.set_context(mode=context.GRAPH_MODE)

net_with_criterion = WithLossCell(model, criterion)
train_network = nn.TrainOneStepCell(net_with_criterion, optimizer)
train_network.set_train()

Training

for epoch in range(20):
# hidden : [num_layers * num_directions, batch, hidden_size]
loss = train_network(enc_inputs, dec_inputs, target_batch.view(-1))
print('Epoch:', '%04d' % (epoch + 1), 'cost =', '{:.6f}'.format(loss.asnumpy()))

RuntimeError Traceback (most recent call last)
/tmp/ipykernel_1841/2954476418.py in
9 for epoch in range(20):
10 # hidden : [num_layers * num_directions, batch, hidden_size]
---> 11 loss = train_network(enc_inputs, dec_inputs, target_batch.view(-1))
12 print('Epoch:', '%04d' % (epoch + 1), 'cost =', '{:.6f}'.format(loss.asnumpy()))

/usr/local/python-3.7.5/lib/python3.7/site-packages/mindspore/nn/cell.py in call(self, *args, **kwargs)
576 logger.warning(f"For 'Cell', it's not support hook function in graph mode. If you want to use hook "
577 f"function, please use context.set_context to set pynative mode.")
--> 578 out = self.compile_and_run(*args)
579 return out
580

/usr/local/python-3.7.5/lib/python3.7/site-packages/mindspore/nn/cell.py in compile_and_run(self, *inputs)
963 """
964 self._auto_parallel_compile_and_run = True
--> 965 self.compile(*inputs)
966
967 new_inputs = []

/usr/local/python-3.7.5/lib/python3.7/site-packages/mindspore/nn/cell.py in compile(self, *inputs)
936 if self._dynamic_shape_inputs is None or self._dynamic_shape_inputs[0] is None:
937 _cell_graph_executor.compile(self, *inputs, phase=self.phase, auto_parallel_mode=self._auto_parallel_mode,
--> 938 jit_config_dict=self._jit_config_dict)
939 else:
940 self._check_compile_dynamic_shape(*inputs)

/usr/local/python-3.7.5/lib/python3.7/site-packages/mindspore/common/api.py in compile(self, obj, phase, do_convert, auto_parallel_mode, jit_config_dict, *args)
1135 if jit_config_dict:
1136 self._graph_executor.set_jit_config(jit_config_dict)
-> 1137 result = self._graph_executor.compile(obj, args_list, phase, self._use_vm_mode())
1138 obj.compile_cache.add(phase)
1139 if not result:

RuntimeError: Parent func graph should be handled in advance, fg: ◀Equal.125, context: {FuncGraph: ◀Equal.125 Args: [0]: AbstractTensor(shape: (1, 5), element: AbstractScalar(Type: Float32, Value: AnyValue, Shape: NoShape), value_ptr: 0x5613f0692bd0, value: AnyValue), Parent: {FuncGraph: ▶Equal.53 Args: [0]: AbstractTensor(shape: (1, 5), element: AbstractScalar(Type: Int32, Value: AnyValue, Shape: NoShape), value_ptr: 0x5613f0692bd0, value: AnyValue), [1]: AbstractTensor(shape: (), element: AbstractScalar(Type: Int32, Value: AnyValue, Shape: NoShape), value_ptr: 0x561472ca8d70, value: Tensor(shape=[], dtype=Int32, value=0)), Parent: { Args: }}}, parent context: {FuncGraph: ▶Equal.53 Args: [0]: AbstractTensor(shape: (1, 5), element: AbstractScalar(Type: Int32, Value: AnyValue, Shape: NoShape), value_ptr: 0x5613f0692bd0, value: AnyValue), [1]: AbstractTensor(shape: (), element: AbstractScalar(Type: Int32, Value: AnyValue, Shape: NoShape), value_ptr: 0x561472ca8d70, value: Tensor(shape=[], dtype=Int32, value=0)), Parent: { Args: }}

C++ Call Stack: (For framework developers)

mindspore/ccsrc/pipeline/jit/static_analysis/program_specialize.cc:343 FuncGraphSpecializer

lvyufeng / mindspore-nlp-tutorial Goto Github PK