InvalidArgumentError (see above for traceback): Incompatible shapes: [80,1] vs. [160,1] about chatbot-retrieval HOT 3 CLOSED

INFO:tensorflow:Results after 1220 steps (0.096 sec/batch): loss = 3.94108, recall_at_5 = 0.509272540984, recall_at_10 = 1.0, recall_at_2 = 0.200973360656, recall_at_1 = 0.11080942623.
W tensorflow/core/framework/op_kernel.cc:968] Invalid argument: Incompatible shapes: [80,1] vs. [160,1]
     [[Node: prediction/logistic_loss/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](prediction/Squeeze, prediction/ToFloat)]]
Traceback (most recent call last):
  File "udc_train.py", line 70, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv[:1] + flags_passthrough))
  File "udc_train.py", line 67, in main
    estimator.fit(input_fn=input_fn_train, steps=None, monitors=[eval_monitor])
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 333, in fit
    max_steps=max_steps)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 708, in _train_model
    max_steps=max_steps)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/graph_actions.py", line 285, in _monitored_train
    None)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 368, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 521, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 488, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 625, in run
    hook in outputs else None))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 1215, in after_run
    induce_stop = m.step_end(self._last_step, result)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 411, in step_end
    return self.every_n_step_end(step, output)
  File "udc_train.py", line 64, in every_n_step_end
    steps=None)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 399, in evaluate
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 771, in _evaluate_model
    max_steps=steps)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/graph_actions.py", line 738, in evaluate
    session.run(update_op, feed_dict=feed_dict)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 717, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 915, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 965, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 985, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: Incompatible shapes: [80,1] vs. [160,1]
     [[Node: prediction/logistic_loss/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](prediction/Squeeze, prediction/ToFloat)]]
     [[Node: recall_at_2/ToInt64/_107 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_456_recall_at_2/ToInt64", tensor_type=DT_INT64, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op u'prediction/logistic_loss/mul', defined at:
  File "udc_train.py", line 70, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv[:1] + flags_passthrough))
  File "udc_train.py", line 67, in main
    estimator.fit(input_fn=input_fn_train, steps=None, monitors=[eval_monitor])
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 333, in fit
    max_steps=max_steps)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 708, in _train_model
    max_steps=max_steps)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/graph_actions.py", line 285, in _monitored_train
    None)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 368, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 521, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 488, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/monitored_session.py", line 625, in run
    hook in outputs else None))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 1215, in after_run
    induce_stop = m.step_end(self._last_step, result)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 411, in step_end
    return self.every_n_step_end(step, output)
  File "udc_train.py", line 64, in every_n_step_end
    steps=None)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 399, in evaluate
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 760, in _evaluate_model
    eval_dict = self._get_eval_ops(features, targets, metrics)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 991, in _get_eval_ops
    predictions, loss, _ = self._call_model_fn(features, targets, ModeKeys.EVAL)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 946, in _call_model_fn
    return self._model_fn(features, targets, mode=mode)
  File "/home/dmitry/go/src/github.com/lepikhin/chatbot-retrieval/udc_model.py", line 83, in model_fn
    tf.concat(0, all_targets))
  File "/home/dmitry/go/src/github.com/lepikhin/chatbot-retrieval/models/dual_encoder.py", line 85, in dual_encoder_model
    losses = tf.nn.sigmoid_cross_entropy_with_logits(logits, tf.to_float(targets))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn.py", line 448, in sigmoid_cross_entropy_with_logits
    return math_ops.add(relu_logits - logits * targets,
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 751, in binary_op_wrapper
    return func(x, y, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 910, in _mul_dispatch
    return gen_math_ops.mul(x, y, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 1519, in mul
    result = _op_def_lib.apply_op("Mul", x=x, y=y, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2380, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1298, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Incompatible shapes: [80,1] vs. [160,1]
     [[Node: prediction/logistic_loss/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](prediction/Squeeze, prediction/ToFloat)]]
     [[Node: recall_at_2/ToInt64/_107 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_456_recall_at_2/ToInt64", tensor_type=DT_INT64, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

dmitry@dmitry:~/go/src/github.com/lepikhin/chatbot-retrieval$ python udc_train.py