Hoping that this project is not abandoned and you're willing to patch this:
When trying to load a saved model using the Agent.load_model method it will throw the exception: 'keras load ValueError: Unknown loss function:loss'

The regular solution is to change the line:
self.actor_network = load_model(self.dic_path["PATH_TO_MODEL"], "%s_actor_network.h5")
(which I already changed to): self.actor_network = load_model("ppo/actor_network.h5")
to:
self.actor_network = load_model("ppo/actor_network.h5", custom_objects={'loss': self.loss})

However because the loss function is an inner function that cannot be called. When trying to use use proximal_policy_optimization_loss(which generates the loss function) instead it'll throw the exception: 'AttributeError: 'function' object has no attribute 'get_shape'

I've been trying to fix this by:
loading weights rather then the model
creating a lose loss function using self.parameters within
creating a lose loss function and use lampda https://stackoverflow.com/a/54177997/8579225

but I can't seem to fix things.
Hope you are willing to help me out with this.

Hi,
I think this function: _shared_network_structure is not doing what we expect
actually this adds different layers with different parameters each time we call it, not shared layer

After trying, gym 0.26.2 does not work, I reduce the use of 0.25.2 is feasible