maximumentropy / cudnn_rnn_theano_benchmarks Goto Github PK

Hello, firstly I want to thank you for providing the nice code samples and benchmark results!

I think it would be great to include compile time on the results, because I saw that using cuDNN decreased the compile time substantially. (over 20 seconds vs. about 5~6 seconds)

Using your codes, I ran benchmarks on my machine.
On small networks, there were no big differences, and for a very small network using cuDNN performed worse than the pure Theano version.

Some of my experimental results do not match your results, especially when the model is very simple (-b 32 -t 30 -o 128 -d 1) and when the model is very complex (-b 128 -t 200 -o 1024 -d 3).

Depth 1

Batch Size 32 x Seq Len 30

Hidden Size 128

Version	Train (µs)	Forward only (µs)
Theano	108.0	32.1
cuDNN Theano	108.5	44.4

Hidden Size 512

Version	Train (µs)	Forward only (µs)
Theano	401.7	106.1
cuDNN Theano	228.5	87.4

Hidden Size 1024

Version	Train (µs)	Forward only (µs)
Theano	888.6	235.6
cuDNN Theano	441.9	161.7

Batch Size 128 x Seq Len 30

Hidden Size 128

Version	Train (µs)	Forward only (µs)
Theano	105.8	29.1
cuDNN Theano	29.9	11.3

Hidden Size 512

Version	Train (µs)	Forward only (µs)
Theano	399.9	103.9
cuDNN Theano	102.8	29.1

Hidden Size 1024

Version	Train (µs)	Forward only (µs)
Theano	932.1	241.0
cuDNN Theano	218.1	78.9

Depth 3

Batch Size 128 x Seq Len 30

Hidden Size 512

Version	Train (µs)	Forward only (µs)
Theano	592.3	294.7
cuDNN Theano	254.3	70.8

Hidden Size 1024

Version	Train (µs)	Forward only (µs)
Theano	1386.4	690.8
cuDNN Theano	722.3	246.9

Batch Size 128 x Seq Len 200

Hidden Size 512

Version	Train (µs)	Forward only (µs)
Theano	1880.6	1065.9
cuDNN Theano	1678.3	447.8

Hidden Size 1024

Version	Train (µs)	Forward only (µs)
Theano	6133.5	3609.7
cuDNN Theano	4867.0	1593.2