batch_size for off-policy learning about acer HOT 4 CLOSED

The increase in compute time based on batch size won't be linear because of the ability to do parallel computation (same is true for GPUs and CPUs), so this shouldn't be an issue unless you use a very large batch size. If you consider the DQN, the original paper used a batch size of 32 - consider it the same situation.

Note that other people have worked on batch versions of A3C (see Training Agent for First-Person Shooter Game with Actor-Critic Curriculum Learning), so it is also possible to do the on-policy step with batches too.

from acer.

Hey, thanks for the reply. But for DQN it is running on GPU so large batch_size wouldn't be an issue. For the original A3C, it wouldn't get too much speed-ups were it run on GPU since for each forward pass it only passes one sample, while if a larger batch is fed in, I suspect that it wouldn't be worthy for the authors to claim that it can be run on purely CPUs, especially when your input is high-dimensional. And for the paper you refered to, I roughly skipped over it, but I think it's running on GPU? Would be gla to hear about your insights :)

from acer.

It is true that Batch-A3C (what I referenced) and GA3C are batch versions of A3C that run training on GPU, and I would indeed expect the benefits of transferring to GPU to increase as batch size increases. However, if the network is smaller or the batch size is smaller (not necessarily 1), the time to transfer data to and from the GPU can become the bottleneck, and the benefit from GPU computation is diminished. Does the variance reduction from using a batch help at this point? Who knows.

from acer.

Thanks a lot for the reply :)

from acer.