The initial handshake to exchange information works nicely, but after getting the public IP and ports, when the peers try to fire a packet to one another the packet was seemingly going nowhere.

I completely missed the fact that on giving the packet a new address(the remote peer address rather than the relay address) a new binding was created in the NAT even when sending the packet from the same port. So the problem was both the peers were sending packets to ports which were being used for the relay connection and thus due to symmetric NAT and not matching the remote address in the connection the packets were essentially getting discarded on the side of the receiver.