[REQUEST] Adding Cal-QL about d3rlpy HOT 2 OPEN

@zxp567 Hi, thank you for your request. I'm sure this can be supported in d3rlpy. I'm currently working on the next major update release in nightly branch. I might support Cal-CQL there.

from d3rlpy.

The Cal-QL code has been released by the author: https://github.com/nakamotoo/Cal-QL/.
However, it was implemented in JAXCQL and seems more robotics oriented, while I would just need something simpler that work with an offline MDPDataset like this d3rlpy library offers, and I am looking for discrete actions only as well.

Would you have some capacity to include the calculation of reference policy Q-values and the logic of cal-ql could be added to this repo's discreteCQL algorithm?

It seems we only need to revise "_compute_conservative_loss" in "algos/torch/cql_impl.py" by:

1. adding the "return_to_go" similar to the calculation as in "https://github.com/nakamotoo/Cal-QL/blob/main/JaxCQL/replay_buffer.py" as another input;
2. and including the lower bound logic similar to the logic as in "https://github.com/nakamotoo/Cal-QL/blob/main/JaxCQL/conservative_sac.py"

Right? Would be helpful if you could add this into this repo as Cal-QL seems to work significantly better than CQL once the online fine-tuning begins.

Thank you very much.

from d3rlpy.