numpy implementation of Recurrent Neural Networks

• RNN_numpy based on iamtrask's github.io
• LSTM_numpy based on wiseodd's github.io

Training Vanilla RNN for binary addition.

largest = pow(2, BIN_DIM)
decimal = np.array([range(largest)]).astype(np.uint8).T
binary = np.unpackbits(decimal, axis=1)

Simply prepare the binary digit database. And propagate forward and backward to train binary addition.

hidden = sigmoid(np.dot(X, w0) + np.dot(hidden_values[-1], wh))
output = sigmoid(np.dot(hidden, w1))

pred[pos] = np.round(output[0][0])

Training was successful with 100% accuracy.

----------
Iter 9000
Error : [ 0.68126419]
Pred : [1 0 1 0 1 0 0 0]
True : [1 0 1 0 1 0 0 0]
108 + 60 = 168
----------

Training LSTM RNN for mnist handwritten digit recognition.

mnist = input_data.read_data_sets('MNIST_data', one_hot=True)

Prepare the mnist digit database with tensorflow.examples.tutorials.mnist.input_data.

caches, states = LSTM_Cell(Xt)
c, h = states[-1]

out = np.dot(h, wy) + by
pred = softmax(out)
entropy = cross_entropy(pred, Y)

Train the LSTM model with BPTT and it was successful to recognize handwritten digits.