I'm wondering if the chatbot could be trained with a CPU only.
I have an AMD GPU and therefore cannot run tensorflow, as it only supports NVIDIA GPUs.
Does it even make sense to train with a CPU, or would it take so much longer that I'll have to let my grandchildren check for the results? :)
I followed the Chatbot tutorial and just reached this point, so it would be great if there is a way to use CPU only. Even if it does not make sense to train on large datasets, but just to experiment with it.
Thanks for the great tutorial and the chatbot on github!

So while observing some bots training and fiddling around a bit, one thing I would like to do is add tags to the training data. Something like this Hi, my name is [FIRSTNAME] from [UNIVERSITY], nice to meet you. Today is [DAYOFWEEK].

Then I could use the tags to dynamically customize bots using the same training data.

Using brackets like this seems to work as the brackets and containing text does get output after training. But, I'm wondering if there is a better way to implement this tagging behavior.

i was wondering if its possible to stop the training process and run it again while retaining all the prior data. Im just scared that if i stop the train process itll restart from the beginning with no steps. im also wondering if there is a way to use multiple reddit years in the making of the test.to and the test.from

Am I missing GPU support or something?

Collecting tensorflow-gpu>=1.4.0 (from -r requirements.txt (line 1))
Downloading tensorflow_gpu-1.5.0-cp35-cp35m-manylinux1_x86_64.whl (201.9MB)
99% |████████████████████████████████| 201.9MB 21.8MB/s eta 0:00:01Exception:
Traceback (most recent call last):
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/commands/install.py", line 335, in run
wb.build(autobuilding=True)
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/req/req_set.py", line 620, in _prepare_file
session=self.session, hashes=hashes)
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/download.py", line 821, in unpack_url
hashes=hashes
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/download.py", line 659, in unpack_http_url
hashes)
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/download.py", line 882, in _download_http_url
_download_url(resp, link, content_file, hashes)
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/download.py", line 603, in _download_url
hashes.check_against_chunks(downloaded_chunks)
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/utils/hashes.py", line 46, in check_against_chunks
for chunk in chunks:
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/download.py", line 571, in written_chunks
for chunk in chunks:
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/utils/ui.py", line 139, in iter
for x in it:
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/download.py", line 560, in resp_read
decode_content=False):
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/_vendor/requests/packages/urllib3/response.py", line 357, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/_vendor/requests/packages/urllib3/response.py", line 314, in read
data = self._fp.read(amt)
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 63, in read
self._close()
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 50, in _close
self.__callback(self.__buf.getvalue())
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/_vendor/cachecontrol/controller.py", line 275, in cache_response
self.serializer.dumps(request, response, body=body),
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/pip/_vendor/cachecontrol/serialize.py", line 86, in dumps
data, separators=(",", ":"), sort_keys=True,
File "/usr/lib/python3.5/json/init.py", line 237, in dumps
**kw).encode(obj)
File "/usr/lib/python3.5/json/encoder.py", line 198, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python3.5/json/encoder.py", line 256, in iterencode
return _iterencode(o, 0)
MemoryError

I am training my model with 20 million paired rows in 'train.to' and 'train.from' file and after whole day of training it is showing that
"global steps 3100 lr 0.001 step-time 30.11s wps 0.34K ppl 5.95gN 24.58 bleu 0.00" and continuing.
can anyone tell me what does that above line means and how much time it will take more. I am new in this field, so sorry if my question is not appropriate.

I replaced everything in new_data dir.
Run the prepare.py.
Created the vocab successfully..
But when running it throws an error..
The image file is attached

Okay, I've done a little training of the chatbot, and I tried moving it around my folders structure, but it seems that it takes the physical paths for some of the dirs. Found out that hparams is one of the folders that holds the physical paths. Which other folders need to be modified in order to be able to run the model from anywhere?
Also, what are the minimum files to run the inference script?

So I keep hitting this error when SSH'ing into my boxes, and so I decided to follow along exactly with Sentdex's tutorial and did exactly as he did on paperspace and when I go to train.py I get the following error.

Before I reinstalled TensorFlow and it corrected the problem (On a Digital Ocean 32CPU box). Doesn't seem to work on Paperspace and not sure why I'm getting this error when Harrison doesn't in the tutorial :/

This is the position in the tutorial in which I get the following: https://youtu.be/040CfFRJ9Rs?t=879

Perhaps related: tensorflow/tensorflow#15604

paperspace@psh728wf0:~/Desktop/nmt-chatbot$ python train.py
Traceback (most recent call last):
  File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/home/paperspace/anaconda3/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/home/paperspace/anaconda3/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 7, in <module>
    from nmt import nmt
  File "/home/paperspace/Desktop/nmt-chatbot/nmt/nmt/nmt.py", line 26, in <module>
    import tensorflow as tf
  File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import *
  File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/home/paperspace/anaconda3/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/home/paperspace/anaconda3/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.
paperspace@psh728wf0:~/Desktop/nmt-chatbot$ 

python version = 3.6.3
tensorflow version = 1.4 - CPU

i get this error after created eval model with fresh parameters
tensorflow.python.framework.errors_impl.OutOfRangeError, End of sequence
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[?,?], [?,?], [?,?], [?], [?]], output_types=[DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Caused by op 'IteratorGetNext', defined at:
File "C:\Users\Rio.vscode\extensions\ms-python.python-0.9.1\pythonFiles\PythonTools\visualstudio_py_launcher.py", line 91, in
vspd.debug(filename, port_num, debug_id, debug_options, currentPid, run_as)
File "C:\Users\Rio.vscode\extensions\ms-python.python-0.9.1\pythonFiles\PythonTools\visualstudio_py_debugger.py", line 2624, in debug
exec_file(file, globals_obj)
File "C:\Users\Rio.vscode\extensions\ms-python.python-0.9.1\pythonFiles\PythonTools\visualstudio_py_util.py", line 119, in exec_file
exec_code(code, file, global_variables)
File "C:\Users\Rio.vscode\extensions\ms-python.python-0.9.1\pythonFiles\PythonTools\visualstudio_py_util.py", line 95, in exec_code
exec(code_obj, global_variables)
File "c:\Users\Rio\MLDL\nmt-chatbot\train.py", line 18, in
tf.app.run(main=nmt.main, argv=[os.getcwd() + '\nmt\nmt\nmt.py'] + unparsed)
File "C:\Users\Rio\AppData\Local\conda\conda\envs\tensorflow2\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "c:\Users\Rio\MLDL\nmt-chatbot/nmt\nmt\nmt.py", line 539, in main
run_main(FLAGS, default_hparams, train_fn, inference_fn)
File "c:\Users\Rio\MLDL\nmt-chatbot/nmt\nmt\nmt.py", line 532, in run_main
train_fn(hparams, target_session=target_session)
File "c:\Users\Rio\MLDL\nmt-chatbot/nmt\nmt\train.py", line 223, in train
eval_model = model_helper.create_eval_model(model_creator, hparams, scope)
File "c:\Users\Rio\MLDL\nmt-chatbot/nmt\nmt\model_helper.py", line 149, in create_eval_model
tgt_max_len=hparams.tgt_max_len_infer)
File "c:\Users\Rio\MLDL\nmt-chatbot/nmt\nmt\utils\iterator_utils.py", line 202, in get_iterator
tgt_seq_len) = (batched_iter.get_next())
File "C:\Users\Rio\AppData\Local\conda\conda\envs\tensorflow2\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 259, in get_next
name=name))
File "C:\Users\Rio\AppData\Local\conda\conda\envs\tensorflow2\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 705, in iterator_get_next
output_shapes=output_shapes, name=name)
File "C:\Users\Rio\AppData\Local\conda\conda\envs\tensorflow2\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\Rio\AppData\Local\conda\conda\envs\tensorflow2\lib\site-packages\tensorflow\python\framework\ops.py", line 2956, in create_op
op_def=op_def)
File "C:\Users\Rio\AppData\Local\conda\conda\envs\tensorflow2\lib\site-packages\tensorflow\python\framework\ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
OutOfRangeError (see above for traceback): End of sequence
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[?,?], [?,?], [?,?], [?], [?]], output_types=[DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

how can i train it with another language that is not English?
I want it to chat to me in my home language.

For the sample data provided, the maximum vocab size I could use was 15175. But for my own data set, that won't work. The shape of the tensor it wants to allocate is changed somehow. For the sample data it was something like [20000,vocab_size] and now it's much bigger [124288,vocab_size]. How exactly can I reduce that number? I tried vocab size of 12500 but even that won't fit and I don't want to go any lower than that. Any help would be great.

I'm using a GTX 960M with 4GB VRam on windows 10. Data I'm using contains all comments with score > 5 from months 6-10 of 2017.

Hey can we please have the link to the trained model checkpoint?
pretty please!!!

Vurrently your version of NMT requires a specific version of CUDA v9.0 I have 9.1 installed and in order to run train.py I have had to roll back CUDA to 9.0 This is annoying.

I dont foresee that the issue of versioning should be too terribly had to do I only ask that you look into making it more forward compatible for future users.

Hello.
I cannot seem to find modded-inference.py file and I really wanted to play around with it.
Can you please put up a link here if you have modified or replaced it with something ?
Thank you in advance.
Regards,
iKK.

Is the right way to close the terminal, change the parameters in the settings.py and then run train.py again?

I am using a Tesla P100 with 16GB VRAM for training, with the default settings only 4GB of the VRAM is consumed. Which parameter in the settings will help me use the full potential of the GPU.
. My bleu score is always 0.
I increased the batch size to 1024 still, it remains the same.
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.30 Driver Version: 390.30 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla P100-PCIE... Off | 00000000:00:04.0 Off | 0 |
| N/A 40C P0 37W / 250W | 4551MiB / 16280MiB | 5% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1930 G /usr/lib/xorg/Xorg 82MiB |
| 0 25720 C python3 4459MiB |
+-----------------------------------------------------------------------------+

loaded eval model parameters from /home/adamshamsudeen/nmt-chatbot/model/translate.ckpt-51000, time 0.06s
eval dev: perplexity 13.00, time 0s, Thu Mar 15 13:49:54 2018.
eval test: perplexity 13.00, time 0s, Thu Mar 15 13:49:55 2018.
global step 51100 lr 0.001 step-time 2.49s wps 4.34K ppl 4.43 gN 10.65 bleu 0.00
global step 51200 lr 0.001 step-time 2.52s wps 4.36K ppl 4.45 gN 10.24 bleu 0.00
global step 51300 lr 0.001 step-time 2.50s wps 4.34K ppl 4.46 gN 9.46 bleu 0.00
global step 51400 lr 0.001 step-time 2.52s wps 4.33K ppl 4.44 gN 9.61 bleu 0.0

Thanks

I run train.py for the first time with 3 * GeForce GTX 1060 6G and I get on average 12 seconds per step on a training size of 120,000 pairs.
I am planning to increase the training pairs to 2 millions. Will my duration per step decrease?
Is there a way to calculate the duration of step based on the training pairs?

Perhaps I somehow have my training files in the wrong location? After training my model and running inference.py I get the following error.

`Starting interactive mode (first response will take a while):

hello
Traceback (most recent call last):
File "inference.py", line 277, in
answers = process_questions(question)[0]
File "inference.py", line 238, in process_questions
answers_list = inference_helper(prepared_questions)
File "inference.py", line 174, in start_inference
return inference_helper(question)
File "inference.py", line 167, in
inference_helper = lambda question: do_inference(question, *inference_object)
File "inference.py", line 91, in do_inference
loaded_infer_model = nmt.inference.model_helper.load_model(infer_model.model, flags.ckpt, sess, "infer")
File "/home/science/tf-demo/models/nmt-chatbot/nmt/nmt/model_helper.py", line 465, in load_model
model.saver.restore(session, ckpt)
File "/home/science/tf-demo/tensorflow-dev/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1682, in restore
raise ValueError("Can't load save_path when it is None.")
ValueError: Can't load save_path when it is None.`

I've trained all the chat data and ran almost everything, except python won't import nmt at all, i have no idea what to do here, please help me out.

Hi , I came into this error when i run prepare_data.py from cmd:

Traceback (most recent call last):
File "prepare_data.py", line 5, in
from core.tokenizer import tokenize
ModuleNotFoundError: No module named 'core'

The error did not came out when I run it from python and is it normal to get this respond when training the chatbot:
src: ▁> ▁O ld ▁M a us ers ▁newlinechar ▁newlinechar ▁FTFY
ref: ▁Two ▁of ▁them ▁are ▁definitely ▁beat ers ! ▁LOL !
nmt: > Old Mausers newlinechar newlinechar FTFY

The underscore does not appear in the older version and I am using the latest nmt-chatbot.

My train loss suddenly increase during the 2nd epoch when I am training my 3.2 million pairs data. Should I continue training or retrain a new model? I am using the auto decay learn rate.

I am running two instances of nmt-chatbot in two separate setups. I have taken two deep learning VMs from azure: NC24 & NC12 with 24 & 12 vCPUs respectively.
In one instance I have used the dat as it is from you git repo. In the other instance I have created training data myself for reddit as instructed by Sentdex. The training program has been running on both the VMs since last 5 days.
Is there an indicator which can tell me that the training is going to complete?
Below is the latest status on both the VMs:

default data VM1

global step 178400 lr 0.001 step-time 2.46s wps 2.36K ppl 4.52 gN 16.79 bleu 1.08
global step 178500 lr 0.001 step-time 2.32s wps 2.58K ppl 4.46 gN 15.53 bleu 1.08
global step 178600 lr 0.001 step-time 2.31s wps 2.59K ppl 4.51 gN 15.54 bleu 1.08
global step 178700 lr 0.001 step-time 2.31s wps 2.59K ppl 4.64 gN 16.18 bleu 1.08
global step 178800 lr 0.001 step-time 2.30s wps 2.60K ppl 4.75 gN 16.67 bleu 1.08
global step 178900 lr 0.001 step-time 2.32s wps 2.60K ppl 4.91 gN 16.41 bleu 1.08

#VM2: with larger training data

 [[Node: IteratorGetNext = IteratorGetNext[output_shapes=[[?,?], [?,?], [?,?], [?], [?]], output_types=[DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"](Iterator)]]

eval dev: perplexity 41.61, time 0s, Sun Mar 4 15:59:49 2018.
eval test: perplexity 41.61, time 0s, Sun Mar 4 15:59:50 2018.
global step 311100 lr 0.001 step-time 1.10s wps 6.90K ppl 38.31 gN 25.26 bleu 0.00
global step 311200 lr 0.001 step-time 1.11s wps 6.90K ppl 37.57 gN 12.39 bleu 0.00
global step 311300 lr 0.001 step-time 1.12s wps 6.89K ppl 37.92 gN 17.00 bleu 0.00
global step 311400 lr 0.001 step-time 1.11s wps 6.93K ppl 37.97 gN 22.01 bleu 0.00
global step 311500 lr 0.001 step-time 1.12s wps 6.83K ppl 37.70 gN 15.15 bleu 0.00
global step 311600 lr 0.001 step-time 1.12s wps 6.92K ppl 37.76 gN 18.56 bleu 0.00
global step 311700 lr 0.001 step-time 1.11s wps 6.89K ppl 38.05 gN 22.62 bleu 0.00
global step 311800 lr 0.001 step-time 1.13s wps 6.91K ppl 37.41 gN 12.24 bleu 0.00
global step 311900 lr 0.001 step-time 1.10s wps 6.90K ppl 38.28 gN 19.09 bleu 0.00
global step 312000 lr 0.001 step-time 1.10s wps 6.95K ppl 37.82 gN 19.45 bleu 0.00

Save eval, global step 312000

I have used common settings.py in both the instances:

preprocessing = {

# Number of samples to save in training data set
# -1 means all available in source data set
'samples': -1,

# Vocab max size
'vocab_size': 15000,

# Whether to use joined (common) vocab for both source and destination
# (should work well with BPE/WPM-like tokenization for our chatbot - english-english translation)
'joined_vocab': True,

# Whether to use BPE/WPM-like tokenization, or standard one
'use_bpe': True,

# Whether to use:
# - embedded detokenizer (increases number of vocab tokens, but is more accurate)
# - external/rule-based detokenizer (based of a bunch of rules and regular expressions -
#   doesn't increase number of tokens in vocab, but it's hard to make a rule for every case)
# Note, that embedded detokenizer is forced to True while using BPE-like tokenizer
'embedded_detokenizer': True,

# Test sets' max size
'test_size': 100,


## You don't need to change anything below (internal settings)

# Source (raw) data folder
'source_folder': source_dir,

# Place to save preprocessed and tokenized training set
'train_folder': train_dir,

# File with protected phrases for standard tokenizer
'protected_phrases_standard_file': os.path.join(package_path, 'setup/protected_phrases_standard.txt'),

# File with protected phrases for BPE/WPM-like tokenizer
'protected_phrases_bpe_file': os.path.join(package_path, 'setup/protected_phrases_bpe.txt'),

# File with blacklisted answers
'answers_blacklist_file': os.path.join(package_path, 'setup/answers_blacklist.txt'),

# File with detokenizer rules
'answers_detokenize_file': os.path.join(package_path, 'setup/answers_detokenize.txt'),

# File with replace rules for answers
'answers_replace_file': os.path.join(package_path, 'setup/answers_replace.txt'),

# File with blacklisted answers
'vocab_blacklist_file': os.path.join(package_path, 'setup/vocab_blacklist.txt'),

# File with replace rules for vocab
'vocab_replace_file': os.path.join(package_path, 'setup/vocab_replace.txt'),

# Number of processes to be spawned during tokenization (leave None for os.cpu_count())
'cpu_count': None,

}

hparams

hparams = {
'attention': 'scaled_luong',
'num_train_steps': 10000000,
'num_layers': 2,
'num_units': 512,

'batch_size': 128,

'override_loaded_hparams': True,

'decay_scheme': 'luong234'

'residual': True,

'optimizer': 'adam',
'encoder_type': 'bi',
'learning_rate':0.001,
'beam_width': 20,
'length_penalty_weight': 1.0,
'num_translations_per_input': 20,

'num_keep_ckpts': 5,

## You don't need to change anything below (internal settings)
'src': 'from',
'tgt': 'to',
'vocab_prefix': os.path.join(train_dir, "vocab"),
'train_prefix': os.path.join(train_dir, "train"),
'dev_prefix': os.path.join(train_dir, "tst2012"),
'test_prefix': os.path.join(train_dir, "tst2013"),
'out_dir': out_dir,
'share_vocab': preprocessing['joined_vocab'],

}

######## DO NOT TOUCH ANYTHING BELOW ########

if preprocessing['use_bpe']:
preprocessing['embedded_detokenizer'] = True
hparams['subword_option'] = 'spm'

preprocessing['protected_phrases_file'] = preprocessing['protected_phrases_bpe_file'] if preprocessing['use_bpe'] else preprocessing['protected_phrases_standard_file']

if preprocessing['use_bpe']:
hparams['vocab_prefix'] += '.bpe'
hparams['train_prefix'] += '.bpe'
hparams['dev_prefix'] += '.bpe'
hparams['test_prefix'] += '.bpe'

if preprocessing['joined_vocab']:
hparams['share_vocab'] = True

I am currently on step 31K, vocab = 25000

want to increase vocab to 50000 now. Should I change vocab size in settings.py and run prepare_data.py again followed by train.py to continue from step 31K??

Hi, is it possible to train on new set of data and retain the current trained progress?
If it is possible, how should i go about doing it?

Thank you for your help, as i am still very new to this i would like to apologize if this is a stupid question.

Just sitting here trying to figure out if I can import inference and what to call from it to get responses for my discord bot. I would imagine I would need to make something similar to everything below if name == "main":. But I am trying to sort through it and I'm just getting all confused.

it would be really great if i could see your trained model as my gtx850m 4gb takes forever to train the model.
i have no intention of using your model for any purpose over the internet.

With ./utils as working directory, here is traceback, on Ubuntu 16.04.3 LTS.

Traceback (most recent call last):
  File "run_tensorboard.py", line 7, in <module>
    p = subprocess.Popen('tensorboard --port 22222 --logdir {}'.format(hparams['out_dir']), stdout=subprocess.PIPE, bufsize=1)
  File "/path/to/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/path/to/python3.6/subprocess.py", line 1344, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'tensorboard --port 22222 --logdir /path/to/nmt-chatbot/model': 'tensorboard --port 22222 --logdir /path/to/nmt-chatbot/model'

Appending shell=True inside Popen seems to fix it though.

I´m training my chatbot right now and read some of the results in the training terminal.
appears very often. It's not in the training data but it is in the vocabs. Why is it added there?

Hi all, can i remove/delete tst2012.(from|to)m tst2013(from|to)) completely, as i just have tran/test/.from/.to files.

i have errors when i tried to remove them :(

2018-04-10 11:11:54 PSTTraining model... 2018-04-10 11:11:54 PST 2018-04-10 11:11:54 PSTTraceback (most recent call last): 2018-04-10 11:11:54 PSTFile "train.py", line 90, in <module> 2018-04-10 11:11:54 PSTtrain() 2018-04-10 11:11:54 PSTFile "train.py", line 25, in train 2018-04-10 11:11:54 PSTwith open('{}/corpus_size'.format(preprocessing['train_folder']), 'r') as f: 2018-04-10 11:11:54 PSTFileNotFoundError: [Errno 2] No such file or directory: 'data//corpus_size' 2018-04-10 11:11:58 PST

Hello, I was trying to train the model on floydhub and I got this error. Do you know what I am doing wrong?

Can we please have a path to trained files
translate.ckpt-76000.data-00000-of-00001*.* etc
passed as an argument to the inference.py
Thanks!

is there any text to speech and speech to text codes or program to implement with this kind of like how the first episode was?

I have attached the screenshot above..

I have just finished training my 3.2million pairs chatbot but the reply from the chatbot is not really reliable. Should I continue training it or retrain a new model?

I need to create a chatbot which has the custom database. So I just wanted to know what all changes do I need to make to create the chatbot. Provided that I'm using the same set of data entries in the database.

I get the following error:

File: train.from (iteration = 10k lines)
1it [00:00, 4.19it/s]Exception in thread Thread-7:
Traceback (most recent call last):
File "C:\xxx\threading.py", line 916, in _bootstrap_inner
self.run()
File "C:\xxx\threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "prepare_data.py", line 155, in write_lines
file.write('\n'.join(lines) + ('' if last else '\n'))
UnboundLocalError: local variable 'last' referenced before assignment

Indeed, "last" gets first assigned in the if conditional:

if not len(lines) or lines[-1] == '':
    lines = list(filter(None, list(lines)))
    last = True

Shouldn't the score be summed rather than assignment at each step of this function, since currently it takes score from bad response function only.

Thanks,
Shashwat Aggarwal

I created my own training data using the scripts in sentdex's tutorials and now I'm trying to run the train.py script and I keep getting this error:

`Traceback (most recent call last):
File "train.py", line 18, in
tf.app.run(main=nmt.main, argv=[os.getcwd() + '\nmt\nmt\nmt.py'] + unparsed)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/home/paperspace/Desktop/nmt-chatbot/nmt/nmt/nmt.py", line 539, in main
run_main(FLAGS, default_hparams, train_fn, inference_fn)
File "/home/paperspace/Desktop/nmt-chatbot/nmt/nmt/nmt.py", line 532, in run_main
train_fn(hparams, target_session=target_session)
File "/home/paperspace/Desktop/nmt-chatbot/nmt/nmt/train.py", line 294, in train
step_result = loaded_train_model.train(train_sess)
File "/home/paperspace/Desktop/nmt-chatbot/nmt/nmt/model.py", line 245, in train
self.learning_rate])
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[512,100003]
[[Node: gradients/dynamic_seq2seq/decoder/output_projection/Tensordot/MatMul_grad/MatMul_1 = MatMul[T=DT_FLOAT, _class=["loc:@dynamic_seq2seq/decoder/output_projection/Tensordot/MatMul"], transpose_a=true, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](dynamic_seq2seq/decoder/output_projection/Tensordot/Reshape, gradients/dynamic_seq2seq/decoder/output_projection/Tensordot_grad/Reshape)]]
[[Node: gradients/dynamic_seq2seq/decoder/decoder/while/BasicDecoderStep/TrainingHelperNextInputs/cond/Merge_grad/cond_grad/_215 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1567_gradients/dynamic_seq2seq/decoder/decoder/while/BasicDecoderStep/TrainingHelperNextInputs/cond/Merge_grad/cond_grad", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'gradients/dynamic_seq2seq/decoder/output_projection/Tensordot/MatMul_grad/MatMul_1', defined at:
File "train.py", line 18, in
tf.app.run(main=nmt.main, argv=[os.getcwd() + '\nmt\nmt\nmt.py'] + unparsed)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/home/paperspace/Desktop/nmt-chatbot/nmt/nmt/nmt.py", line 539, in main
run_main(FLAGS, default_hparams, train_fn, inference_fn)
File "/home/paperspace/Desktop/nmt-chatbot/nmt/nmt/nmt.py", line 532, in run_main
train_fn(hparams, target_session=target_session)
File "/home/paperspace/Desktop/nmt-chatbot/nmt/nmt/train.py", line 222, in train
train_model = model_helper.create_train_model(model_creator, hparams, scope)
File "/home/paperspace/Desktop/nmt-chatbot/nmt/nmt/model_helper.py", line 108, in create_train_model
extra_args=extra_args)
File "/home/paperspace/Desktop/nmt-chatbot/nmt/nmt/attention_model.py", line 61, in init
extra_args=extra_args)
File "/home/paperspace/Desktop/nmt-chatbot/nmt/nmt/model.py", line 139, in init
colocate_gradients_with_ops=hparams.colocate_gradients_with_ops)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 581, in gradients
grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 353, in _MaybeCompile
return grad_fn() # Exit early
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 581, in
grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py", line 922, in _MatMulGrad
grad_b = math_ops.matmul(a, grad, transpose_a=True)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 1891, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 2437, in _mat_mul
name=name)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

...which was originally created as op 'dynamic_seq2seq/decoder/output_projection/Tensordot/MatMul', defined at:
File "train.py", line 18, in
tf.app.run(main=nmt.main, argv=[os.getcwd() + '\nmt\nmt\nmt.py'] + unparsed)
[elided 5 identical lines from previous traceback]
File "/home/paperspace/Desktop/nmt-chatbot/nmt/nmt/attention_model.py", line 61, in init
extra_args=extra_args)
File "/home/paperspace/Desktop/nmt-chatbot/nmt/nmt/model.py", line 97, in init
res = self.build_graph(hparams, scope=scope)
File "/home/paperspace/Desktop/nmt-chatbot/nmt/nmt/model.py", line 284, in build_graph
encoder_outputs, encoder_state, hparams)
File "/home/paperspace/Desktop/nmt-chatbot/nmt/nmt/model.py", line 407, in _build_decoder
logits = self.output_layer(outputs.rnn_output)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/layers/base.py", line 575, in call
outputs = self.call(inputs, *args, **kwargs)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/layers/core.py", line 156, in call
[0]])
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 2520, in tensordot
ab_matmul = matmul(a_reshape, b_reshape)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 1891, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 2437, in _mat_mul
name=name)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/paperspace/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[512,100003]
[[Node: gradients/dynamic_seq2seq/decoder/output_projection/Tensordot/MatMul_grad/MatMul_1 = MatMul[T=DT_FLOAT, _class=["loc:@dynamic_seq2seq/decoder/output_projection/Tensordot/MatMul"], transpose_a=true, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](dynamic_seq2seq/decoder/output_projection/Tensordot/Reshape, gradients/dynamic_seq2seq/decoder/output_projection/Tensordot_grad/Reshape)]]
[[Node: gradients/dynamic_seq2seq/decoder/decoder/while/BasicDecoderStep/TrainingHelperNextInputs/cond/Merge_grad/cond_grad/_215 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1567_gradients/dynamic_seq2seq/decoder/decoder/while/BasicDecoderStep/TrainingHelperNextInputs/cond/Merge_grad/cond_grad", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]`

I am using the exact same PaperSpace setup with the ML in a Box...

log.txt
hi, when i run prepare_data.py errors in log file appears.
i noticed this line:

FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\Sahand\Desktop\nmt-chatbot\prepare_data.py
it tries to find prepare_data.py in root directory but it's in setup.
Windows 10 x64 ( Anaconda / tf 1.7.0 )

Everything works great for me in your README, until I try to import inference.

The README says to use:

from nmt-chatbot.inference import inference

But I get a syntax error trying to use nmt-chatbot because of the hyphen. If I rename the nmt-chatbot folder to nmt_chatbot and change the PATHS in the hparams file I can import like from nt_chatbot.inference import inference but something else checks for the nmt-chatbot directory and I get this error:

Traceback (most recent call last):
  File "/var/www/scripts/test.py", line 8, in 
    print(inference("Hello!"))
  File "/var/www/scripts/nmt_chatbot/inference.py", line 186, in inference
    answers_list = process_questions(questions, include_blacklisted)
  File "/var/www/scripts/nmt_chatbot/inference.py", line 238, in process_questions
    answers_list = inference_helper(prepared_questions)
  File "/var/www/scripts/nmt_chatbot/inference.py", line 163, in start_inference
    inference_object = do_start_inference(out_dir, hparams)
  File "/var/www/scripts/nmt_chatbot/inference.py", line 59, in do_start_inference
    flags.ckpt = tf.train.latest_checkpoint(flags.out_dir)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1812, in latest_checkpoint
    if file_io.get_matching_files(v2_path) or file_io.get_matching_files(
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/lib/io/file_io.py", line 337, in get_matching_files
    for single_filename in filename
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: /var/www/scripts/nmt-chatbot/model; No such file or directory

nmt-chatbot/setup$ python prepare_data.py
Traceback (most recent call last):
File "prepare_data.py", line 8, in
from core.tokenizer import tokenize
File "../core/tokenizer.py", line 1, in
import HTML

How to fix? :(

So using the example, I should be able to use

from nmt-chatbot.inference import inference

But when I try, I get this:

>>> from nmt-chatbot.inference import inference
  File "<stdin>", line 1
    from nmt-chatbot.inference import inference
            ^
SyntaxError: invalid syntax
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 23, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

Original exception was:
  File "<stdin>", line 1
    from nmt-chatbot.inference import inference
            ^
SyntaxError: invalid syntax
>>> 

So is there a way to correctly import it?

Hello, would it be possible to run the trained model on a CPU?
I download the 'demo model' from dropbox but when I run the inference.py from my laptop I get:
ImportError: libcublas.so.8.0: cannot open shared object file: No such file or directory`

Does the model need a GPU also in production?

Thought we could start a project around the inference choices/scoring.

At the moment, there are 3 tiers (-1, 0, or 1). I think we should have a larger range, and I would like to start with some ideas here for that.

1. Finished thoughts/sentences OR is complete URL. From what I have seen so far, if the AI doesn't end a sentence with some ending punctuation, it's a problem

2. Penalize repetition. Example: "I don't know what you're talking about, but I don't know what you're talking about."

3. Penalize parroting: When his response to a question is basically either just repeating the question, or something very close to that. Example:

---

Legalize medical marijuana in 2 0 0 1 too ! Just goes to show how forward thinking this guy is .

Reply: legalize marijuana in 2 0 0 1

---

or..

---

I like the idea ... have an upvote !

Reply: I like the idea .

---

or...

---

Why am I so nervous ?

Reply: Why am I nervous ?

---

It looks like it's using GPU but not quite right.
I have a gtx 1070

settings.py

## Internal path settings

# Root package path (leave as is, or type full path)
package_path = os.path.realpath(os.path.dirname(__file__) + '/..')

# Model path
out_dir = os.path.join(package_path, "model")

# Training data path
train_dir = os.path.join(package_path, "data")

# Raw data path (data to be prepared and tokenized)
source_dir = os.path.join(package_path, "new_data")


## Settings you can adjust

# Preprocessing settings
preprocessing = {

    # Number of samples to save in training data set
    # -1 means all available in source data set
    'samples': -1,

    # Vocab max size
    'vocab_size': 30000,

    # Whether to use joined (common) vocab for both source and destination
    # (should work well with BPE/WPM-like tokenization for our chatbot - english-english translation)
    'joined_vocab': True,

    # Whether to use BPE/WPM-like tokenization, or standard one
    'use_bpe': True,
    
    # Whether to use:
    # - embedded detokenizer (increases number of vocab tokens, but is more accurate)
    # - external/rule-based detokenizer (based of a bunch of rules and regular expressions -
    #   doesn't increase number of tokens in vocab, but it's hard to make a rule for every case)
    # Note, that embedded detokenizer is forced to True while using BPE-like tokenizer
    'embedded_detokenizer': True,

    # Test sets' max size
    'test_size': 100,


    ## You don't need to change anything below (internal settings)

    # Source (raw) data folder
    'source_folder': source_dir,

    # Place to save preprocessed and tokenized training set
    'train_folder': train_dir,

    # File with protected phrases for standard tokenizer
    'protected_phrases_standard_file': os.path.join(package_path, 'setup/protected_phrases_standard.txt'),

    # File with protected phrases for BPE/WPM-like tokenizer
    'protected_phrases_bpe_file': os.path.join(package_path, 'setup/protected_phrases_bpe.txt'),

    # File with blacklisted answers
    'answers_blacklist_file': os.path.join(package_path, 'setup/answers_blacklist.txt'),

    # File with detokenizer rules
    'answers_detokenize_file': os.path.join(package_path, 'setup/answers_detokenize.txt'),

    # File with replace rules for answers
    'answers_replace_file': os.path.join(package_path, 'setup/answers_replace.txt'),

    # File with blacklisted answers
    'vocab_blacklist_file': os.path.join(package_path, 'setup/vocab_blacklist.txt'),

    # File with replace rules for vocab
    'vocab_replace_file': os.path.join(package_path, 'setup/vocab_replace.txt'),

    # Number of processes to be spawned during tokenization (leave None for os.cpu_count())
    'cpu_count': None,
}

# hparams
hparams = {
    'attention': 'scaled_luong',
    'num_train_steps': 500000,
    'num_layers': 2,
    'num_units': 512,
#    'batch_size': 128,
#    'override_loaded_hparams': True,
#    'decay_scheme': 'luong234'
#    'residual': True,
    'optimizer': 'adam',
    'encoder_type': 'bi',
    'learning_rate':0.001,
    'beam_width': 20,
    'length_penalty_weight': 1.0,
    'num_translations_per_input': 20,
#    'num_keep_ckpts': 5,

    ## You don't need to change anything below (internal settings)
    'src': 'from',
    'tgt': 'to',
    'vocab_prefix': os.path.join(train_dir, "vocab"),
    'train_prefix': os.path.join(train_dir, "train"),
    'dev_prefix': os.path.join(train_dir, "tst2012"),
    'test_prefix': os.path.join(train_dir, "tst2013"),
    'out_dir': out_dir,
    'share_vocab': preprocessing['joined_vocab'],
}


######## DO NOT TOUCH ANYTHING BELOW ########

if preprocessing['use_bpe']:
    preprocessing['embedded_detokenizer'] = True
    hparams['subword_option'] = 'spm'

preprocessing['protected_phrases_file'] = preprocessing['protected_phrases_bpe_file'] if preprocessing['use_bpe'] else preprocessing['protected_phrases_standard_file']

if preprocessing['use_bpe']:
    hparams['vocab_prefix'] += '.bpe'
    hparams['train_prefix'] += '.bpe'
    hparams['dev_prefix'] += '.bpe'
    hparams['test_prefix'] += '.bpe'

if preprocessing['joined_vocab']:
    hparams['share_vocab'] = True

When I launch the prepare_for_deployment.py script I am getting the following.
`paperspace@pseoih0j5:~/Desktop/nmt_chatbot/utils$ python prepare_for_deployment.py Traceback (most recent call last): File "prepare_for_deployment.py", line 9, in <module> from setup.settings import hparams, preprocessing ModuleNotFoundError: No module named 'setup'

When I move the script to the nmt_chatbot folder I get "There are no model checkpoints ready for deployment'.

Am I missing something obvious?

Hi i'm following the video tutorial of Sentdex https://www.youtube.com/watch?v=040CfFRJ9Rs&t=418s
but I don't have an nvidia gpu, how can i train my model "train.from" and "train.to" using a cpu? what do i need to download? I see he uses python scripts in this github project. but how can i do? where do i find these python scripts in order to train my models? :)

git clone --recursive https://github.com/daniel-kukiela/nmt-chatbot/tree/v0.1
Cloning into 'v0.1'...
remote: Not Found
fatal: repository 'https://github.com/daniel-kukiela/nmt-chatbot/tree/v0.1/' not found

Hi Daniel, I have a issue regarding about my output in out_dev file, the model produces responses as [ deleted ] for most samples that are logged in my out_dev file and I also had attached my out_dev file and tensorboard screenshots. Its been 34000 steps and still its the same for nearly all samples. Am I going wrong or do I need to stop the model? I am using following params:

1. num_units: 512
2. num_layers: 2
3. vocab_size: 100000
   Thank you