Does the number of pairs (training data) impacts the step duration ? about nmt-chatbot HOT 3 CLOSED

12 seconds per step is a pretty big value, are you sure you're doing that training using GPU(s)? You should have tensorflow-gpu installed, not tensorflow.
Also, to utilize all 3 GPUs (won't give you much speed improvement tho) add num_gpus = 3 into hparams section of setup/settings.py

Your duration per step will not change when you increase number of paird in training set. Overall training time will increase. You can calculate how many steps do you need to train for a full epoch by dividing number of pairs by batch_size (128 by default). Also with newer version of code you'll be informed about that (green messages).

from nmt-chatbot.

I tried to put num_gpus = 3 in hparams as a propriety for the dictionary. However, I noticed that only 2 gpu are being used

As you can one the screenshot above, the 3rd GPU is not used at all. GPU0 is used at around 60% on average and GPU1 is at 100%.
Why aren't all of them used at 100% ? I made sure however to set num_gpus = 3

from nmt-chatbot.

As far as i know, NMT places different layers on different GPUs. If you are using only 2 layers, that might explain it.
By NMT i mean https://github.com/tensorflow/nmt - we are including that in our project. If you want to know more about how to utilize multiple GPUs, it's better to ask there. If you are going to do that, please point me to that topic, so i could learned as well how exactly it's done.

from nmt-chatbot.