Hi,

I am having trouble figuring out how you are dealing with the Consistency updates, described in section 4.1 of the paper.
Can you help me clarify this?

Best regards.
René

According to appendix A and fig 7 in the original paper, the value net is operated on s^k (the bottom level 0 squares in fig 7), while the gamma net, reward net and lambda net are operated on the first conv layer's output (level 1 squares in fig 7).

In this implementation, the value net is operated on the first conv layer's output, and other 3 nets are operated on the 2nd conv layer's output. (https://github.com/zhongwen/predictron/blob/master/predictron.py#L73) This seems a mismatch with the paper. In short, I believe the value net should not share any conv layers with other branches.

It might worth to take a second look at those details. Thanks.

are you by any chance going to add evaluation for the predictions of the maze ?

#self.rewards = tf.concat(1, [tf.zeros(shape=[bs, 1, self.maze_size], dtype=tf.float32), self.rewards], 'rewards')
self.rewards = tf.concat_v2(1, [tf.zeros(shape=[bs, 1, self.maze_size], dtype=tf.float32, self.rewards]), 'rewards')

throws an error in tensorflow r0.12 and python 3.5. I have managed to almost make it compatible with python 3.5 but i am running in a small error

File "/developer/Downloads/predictron-master/predictron.py", line 134, in build_model
self.rewards = tf.concat_v2(1, [tf.zeros(shape=[bs, 1, self.maze_size], dtype=tf.float32), self.rewards], 'rewards')
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 1052, in concat_v2
dtype=dtypes.int32).get_shape(
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 645, in convert_to_tensor
as_ref=False)
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 710, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/constant_op.py", line 176, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/constant_op.py", line 165, in constant
tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape))
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/tensor_util.py", line 367, in make_tensor_proto
_AssertCompatible(values, dtype)
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/tensor_util.py", line 302, in _AssertCompatible
(dtype.name, repr(mismatch), type(mismatch).name))
TypeError: Expected int32, got list containing Tensors of type '_Message' instead.

Note: This will not run in Python 3.X or tensorflow 0.12

I used train.py and that works without any issues but train_multigpu.py throws an error at the following line as indicated in my earlier issue

Apply the gradients to adjust the shared variables.

apply_gradient_op = opt.apply_gradients(grad_vars, global_step=global_step)

ERROR#

Traceback (most recent call last):
File "./train_multigpu.py", line 253, in
tf.app.run()
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 44, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "./train_multigpu.py", line 249, in main
train()
File "./train_multigpu.py", line 183, in train
apply_gradient_op = opt.apply_gradients(grad_vars, global_step=global_step)
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/training/optimizer.py", line 409, in apply_gradients
self._create_slots(var_list)
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/training/adam.py", line 119, in _create_slots
self._zeros_slot(v, "m", self._name)
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/training/optimizer.py", line 609, in _zeros_slot
named_slots[var] = slot_creator.create_zeros_slot(var, op_name)
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/training/slot_creator.py", line 123, in create_zeros_slot
colocate_with_primary=colocate_with_primary)
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/training/slot_creator.py", line 101, in create_slot
return _create_slot_var(primary, val, '')
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/training/slot_creator.py", line 55, in _create_slot_var
slot = variable_scope.get_variable(scope, initializer=val, trainable=False)
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/variable_scope.py", line 988, in get_variable
custom_getter=custom_getter)
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/variable_scope.py", line 890, in get_variable
custom_getter=custom_getter)
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/variable_scope.py", line 348, in get_variable
validate_shape=validate_shape)
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/variable_scope.py", line 333, in _true_getter
caching_device=caching_device, validate_shape=validate_shape)
File "/home/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/variable_scope.py", line 657, in _get_single_variable
"VarScope?" % name)
ValueError: Variable state/conv1/weights/Adam/ does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=None in VarScope?

Conda latest is running python 3.5.2

example

import Queue ==> Python 2.7

i am not sure if you tested it with python 3.X , it has a couple of errors

Please can you make sure it is compatible with TF 0.12.1 and Python 3.5 at least .