it seems to run fine after reducing batch_size and eval_batch_size in udc_hparams.py. going to close this as not an issue.
tf.flags.DEFINE_integer("batch_size", 64, "Batch size during training")
tf.flags.DEFINE_integer("eval_batch_size", 8, "Batch size during evaluation")

from chatbot-retrieval.

I still have this issue even after changing the batch sizes , any solution as of yet ?

from chatbot-retrieval.

came across the same issue, solved it by changing TEXT_FEATURE_SIZE from the length of my samples to 160(the default value). Wondering this as a bug of tensorflow.Example Protocol Buffer Object?

from chatbot-retrieval.

Thanks for pointing out how to address this error. I ran into the same problem. (GTX 960)

from chatbot-retrieval.

modifying batch_size and eval_batch_size fixed it for me. thanks!

from chatbot-retrieval.

how to solve this issue, does someone can point it out ?

from chatbot-retrieval.

I made a few changes but I think the one that did it is changing things to be correct in my .config file

eval_config: {
num_examples: 2

Note: The below line limits the evaluation process to 10 evaluations.

Remove the below line to evaluate indefinitely.

max_evals: 10
}

Make sure your num_examples matches the number of images (chips) you have to eval against.

model {
faster_rcnn {
num_classes: 3
image_resizer {
keep_aspect_ratio_resizer {
min_dimension: 600
max_dimension: 1024
}
}

Was training on 3 labels, and previously only had one.

from chatbot-retrieval.

@muruga where is the location of udc_hparams.py??

from chatbot-retrieval.

I have the same issue!
I tried to run the code of a tutorial about using tfRecords with keras. link:
https://medium.com/@moritzkrger/speeding-up-keras-with-tfrecord-datasets-5464f9836c36

at the end of the code, it gives the error:
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [<Batch_size>,2] vs. [<2*<Batch_size>>,2]
no matter what the Batch_size is, it gives the error. for example for a Batch_size=7
[7,2] vs. [14,2]

from chatbot-retrieval.

Ok I don't know if it helps but I have found the answer to my own issue.
The problem was that I have defined the data simply by:
train_data = np.random.normal(size=(SUM_OF_ALL_DATASAMPLES,FEATURES_NUM))
which in a 64bit OS will generate data with type float64. Then this data was stored in tfRecords file as a 32bit data. And it was loaded later as a 32 bit data. This would cause every one 64bit data to convert to two 32bit data. Thus, duplicating the batch_size in tensors and causing error.
To correct it, I just added:
train_data = np.random.normal(size=(SUM_OF_ALL_DATASAMPLES,FEATURES_NUM)).astype(np.float32)

please fill free to delete this comment if you find it too simple. I am still newbie to tf. :-)

from chatbot-retrieval.

Ok I don't know if it helps but I have found the answer to my own issue.
The problem was that I have defined the data simply by:
train_data = np.random.normal(size=(SUM_OF_ALL_DATASAMPLES,FEATURES_NUM))
which in a 64bit OS will generate data with type float64. Then this data was stored in tfRecords file as a 32bit data. And it was loaded later as a 32 bit data. This would cause every one 64bit data to convert to two 32bit data. Thus, duplicating the batch_size in tensors and causing error.
To correct it, I just added:
train_data = np.random.normal(size=(SUM_OF_ALL_DATASAMPLES,FEATURES_NUM)).astype(np.float32)

please fill free to delete this comment if you find it too simple. I am still newbie to tf. :-)

I have the same issue!
InvalidArgumentError (see above for traceback): indices[7,16] = 99296 is not in [0, 91620)
[[node embed_utterance (defined at F:\05.study\code\chatbot-retrieval\models\dual_encoder.py:39) = GatherV2[Taxis=DT_INT32, Tindices=DT_INT64, Tparams=DT_FLOAT, _class=["loc:@OptimizeLoss/train/update_word_embeddings/AssignSub"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](word_embeddings/read, read_batch_features_train/fifo_queue_Dequeue:3, embed_context/axis)]]

from chatbot-retrieval.

I am using tensorflow=1.12.0 on win7 64. i am running into this error during training. thanks.
NFO:tensorflow:global_step/sec: 0.762605
INFO:tensorflow:loss = 3.4692888e-17, step = 1901 (133.367 sec)
Traceback (most recent call last):
File "F:/09.Practice/chatbot-deeplearning-retrieval/udc_train.py", line 72, in
tf.app.run()
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
_sys.exit(main(argv))
File "F:/09.Practice/chatbot-deeplearning-retrieval/udc_train.py", line 68, in main
estimator.fit(input_fn=input_fn_train, steps=None, monitors=[eval_monitor])
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\estimators\estimator.py", line 525, in fit
loss = self._train_model(input_fn=input_fn, hooks=hooks)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\estimators\estimator.py", line 1095, in _train_model
_, loss = mon_sess.run([model_fn_ops.train_op, model_fn_ops.loss])
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 671, in run
run_metadata=run_metadata)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1156, in run
run_metadata=run_metadata)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1255, in run
raise six.reraise(*original_exc_info)
File "C:\01.develop\tool\python3\lib\site-packages\six.py", line 693, in reraise
raise value
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1240, in run
return self._sess.run(*args, **kwargs)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1320, in run
run_metadata=run_metadata))
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\monitors.py", line 1294, in after_run
induce_stop = m.step_end(self._last_step, result)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\monitors.py", line 368, in step_end
return self.every_n_step_end(step, output)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\monitors.py", line 753, in every_n_step_end
validation_outputs = self._evaluate_estimator()
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\monitors.py", line 718, in _evaluate_estimator
name=self.name)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\estimators\estimator.py", line 618, in evaluate
log_progress=log_progress)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\estimators\estimator.py", line 911, in _evaluate_model
model_fn_results = self._get_eval_ops(features, labels, metrics)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\estimators\estimator.py", line 1293, in _get_eval_ops
model_fn_lib.ModeKeys.EVAL, metrics)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\estimators\estimator.py", line 1228, in _call_model_fn
model_fn_results = self._model_fn(features, labels, **kwargs)
File "F:\09.Practice\chatbot-deeplearning-retrieval\udc_model.py", line 83, in model_fn
tf.concat(0, all_contexts),
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\ops\array_ops.py", line 1122, in concat
tensor_shape.scalar())
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\framework\tensor_shape.py", line 848, in assert_is_compatible_with
raise ValueError("Shapes %s and %s are incompatible" % (self, other))
ValueError: Shapes (10, ?, 160) and () are incompatible

from chatbot-retrieval.

I am using tensorflow=1.12.0 on win7 64. i am running into this error during training. thanks.
raise ValueError("Shapes %s and %s are incompatible" % (self, other))
ValueError: Shapes (10, ?, 160) and () are incompatible
The detail error is :
NFO:tensorflow:global_step/sec: 0.762605
INFO:tensorflow:loss = 3.4692888e-17, step = 1901 (133.367 sec)
Traceback (most recent call last):
File "F:/09.Practice/chatbot-deeplearning-retrieval/udc_train.py", line 72, in
tf.app.run()
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
_sys.exit(main(argv))
File "F:/09.Practice/chatbot-deeplearning-retrieval/udc_train.py", line 68, in main
estimator.fit(input_fn=input_fn_train, steps=None, monitors=[eval_monitor])
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\estimators\estimator.py", line 525, in fit
loss = self._train_model(input_fn=input_fn, hooks=hooks)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\estimators\estimator.py", line 1095, in _train_model
_, loss = mon_sess.run([model_fn_ops.train_op, model_fn_ops.loss])
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 671, in run
run_metadata=run_metadata)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1156, in run
run_metadata=run_metadata)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1255, in run
raise six.reraise(*original_exc_info)
File "C:\01.develop\tool\python3\lib\site-packages\six.py", line 693, in reraise
raise value
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1240, in run
return self._sess.run(*args, **kwargs)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1320, in run
run_metadata=run_metadata))
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\monitors.py", line 1294, in after_run
induce_stop = m.step_end(self._last_step, result)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\monitors.py", line 368, in step_end
return self.every_n_step_end(step, output)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\monitors.py", line 753, in every_n_step_end
validation_outputs = self._evaluate_estimator()
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\monitors.py", line 718, in _evaluate_estimator
name=self.name)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\estimators\estimator.py", line 618, in evaluate
log_progress=log_progress)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\estimators\estimator.py", line 911, in _evaluate_model
model_fn_results = self._get_eval_ops(features, labels, metrics)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\estimators\estimator.py", line 1293, in _get_eval_ops
model_fn_lib.ModeKeys.EVAL, metrics)
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\contrib\learn\python\learn\estimators\estimator.py", line 1228, in _call_model_fn
model_fn_results = self._model_fn(features, labels, **kwargs)
File "F:\09.Practice\chatbot-deeplearning-retrieval\udc_model.py", line 83, in model_fn
tf.concat(0, all_contexts),
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\ops\array_ops.py", line 1122, in concat
tensor_shape.scalar())
File "C:\01.develop\tool\python3\lib\site-packages\tensorflow\python\framework\tensor_shape.py", line 848, in assert_is_compatible_with
raise ValueError("Shapes %s and %s are incompatible" % (self, other))
ValueError: Shapes (10, ?, 160) and () are incompatible

from chatbot-retrieval.

I am using tensorflow-0.10.0rc0-cp35-cp35m-linux_x86_64.whl, Cuda Toolkit 7.5 and cuDNN v4. i am running into this error during training. thanks.

INFO:tensorflow:Results after 1220 steps (0.081 sec/batch): recall_at_10 = 1.0, recall_at_2 = 0.333043032787, recall_at_1 = 0.184579918033, recall_at_5 = 0.663370901639, loss = 0.698521.
W tensorflow/core/framework/op_kernel.cc:968] Invalid argument: Incompatible shapes: [80,1] vs. [160,1]
[[Node: prediction/logistic_loss/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](prediction/Squeeze, prediction/ToFloat)]]
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors.InvalidArgumentError'>, Incompatible shapes: [80,1] vs. [160,1]
     [[Node: prediction/logistic_loss/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](prediction/Squeeze, prediction/ToFloat)]]
     [[Node: recall_at_10/ToInt64/_145 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_145_recall_at_10/ToInt64", tensor_type=DT_INT64, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Caused by op 'prediction/logistic_loss/mul', defined at:
  File "udc_train.py", line 70, in <module>
    tf.app.run()
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv[:1] + flags_passthrough))
  File "udc_train.py", line 67, in main
    estimator.fit(input_fn=input_fn_train, steps=None, monitors=[eval_monitor])
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 252, in fit
    max_steps=max_steps)
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 584, in _train_model
    max_steps=max_steps)
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/graph_actions.py", line 281, in _monitored_train
    None)
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 333, in run
    run_metadata=run_metadata)
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 490, in run
    run_metadata=run_metadata)
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 537, in run
    return self._sess.run(*args, **kwargs)
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 602, in run
    hook in outputs else None))
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 1148, in after_run
    induce_stop = m.step_end(self._last_step, result)
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 398, in step_end
    return self.every_n_step_end(step, output)
  File "udc_train.py", line 64, in every_n_step_end
    steps=None)
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 318, in evaluate
    name=name)
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 636, in _evaluate_model
    eval_dict = self._get_eval_ops(features, targets, metrics)
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 855, in _get_eval_ops
    predictions, loss, _ = self._call_model_fn(features, targets, ModeKeys.EVAL)
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 809, in _call_model_fn
    return self._model_fn(features, targets, mode=mode)
  File "/home/cent/play/chatbot-retrieval/udc_model.py", line 84, in model_fn
    tf.concat(0, all_targets))
  File "/home/cent/play/chatbot-retrieval/models/dual_encoder.py", line 81, in dual_encoder_model
    losses = tf.nn.sigmoid_cross_entropy_with_logits(logits, tf.to_float(targets))
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/nn.py", line 445, in sigmoid_cross_entropy_with_logits
    return math_ops.add(relu_logits - logits * targets,
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 760, in binary_op_wrapper
    return func(x, y, name=name)
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 909, in _mul_dispatch
    return gen_math_ops.mul(x, y, name=name)
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 1464, in mul
    result = _op_def_lib.apply_op("Mul", x=x, y=y, name=name)
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
    op_def=op_def)
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2334, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1253, in __init__
    self._traceback = _extract_stack()

Traceback (most recent call last):
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 972, in _do_call
    return fn(*args)
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 954, in _run_fn
    status, run_metadata)
  File "/usr/local/lib/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/home/cent/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/errors.py", line 450, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors.InvalidArgumentError: Incompatible shapes: [80,1] vs. [160,1]
     [[Node: prediction/logistic_loss/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](prediction/Squeeze, prediction/ToFloat)]]
     [[Node: recall_at_10/ToInt64/_145 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_145_recall_at_10/ToInt64", tensor_type=DT_INT64, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

I have the same problem? how can I to slove? thanks!

from chatbot-retrieval.

hi
I had a problem of incompatible shape sizes when fitting the model.
I had it running when i specified the batch size as 1.
But when I tried to evaluate with test data, it returned the same error.
What do I with this?
Please do help me.
The error message is displayed::::::::::

InvalidArgumentError Traceback (most recent call last)
in ()
----> 1 test_loss,test_accuracy = model_RNN.evaluate(X_test,y_test)

4 frames
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py in call(self, *args, **kwargs)
1470 ret = tf_session.TF_SessionRunCallable(self._session._session,
1471 self._handle, args,
-> 1472 run_metadata_ptr)
1473 if run_metadata:
1474 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Incompatible shapes: [32,5,5] vs. [32,5]
[[{{node metrics/mean_absolute_error/sub}}]]
[[metrics/mean_absolute_error/Identity/_153]]
(1) Invalid argument: Incompatible shapes: [32,5,5] vs. [32,5]
[[{{node metrics/mean_absolute_error/sub}}]]
0 successful operations.
0 derived errors ignored.

from chatbot-retrieval.