superjeary / uof-paper-code Goto Github PK

This is the official implementation of the codes that produced the results in the 2021 IEEE TNNLS paper titled "Hierarchical Reinforcement Learning with Universal Policies for Multi-Step Robotic Manipulation". Link to video demo. Feel free to play with the codes and raise issues.

@ARTICLE{9366328,
  author={X. {Yang} and Z. {Ji} and J. {Wu} and Y. -K. {Lai} and C. {Wei} and G. {Liu} and R. {Setchi}},
  journal={IEEE Transactions on Neural Networks and Learning Systems}, 
  title={Hierarchical Reinforcement Learning With Universal Policies for Multistep Robotic Manipulation}, 
  year={2021},
  volume={},
  number={},
  pages={1-15},
  doi={10.1109/TNNLS.2021.3059912}}

1. Ubuntu 16.04
   • Higher version Ubuntu systems should work as well.
   • The project was developed and tested on Linux, not sure how it works on Windows.
2. Python 3
3. Mujoco150
4. Mujoco-py==1.50.1.68 python -m pip install mujoco-py==1.50.1.68
5. Pytorch
6. Others python -m pip install -r requirements.txt

1. Clone the repository to wherever you like.
2. On a terminal: export PYTHONPATH=$PYTHONPATH:$PATH_OF_THE_PROJECT_ROOT. Replace $PATH_OF_THE_PROJECT_ROOT with something like /home/someone/UOF-paper-code.
3. (Optional) Activate your conda environment if desired.
4. From the project root:
   • Evaluate the pre-trained UOF agent python run_uof.py --task-id 0
   • Evaluate the pre-trained HAC agent python run_hac.py --task-id 0
   • Train your own UOF agent python run_uof.py --task-id 0 --train
   • Train your own HAC agent python run_hac.py --task-id 0 --train
5. If you know what you are doing, just modify the arguments in the script as you like.
6. More algorithm-related parameters can be found in the config files.

This table gives the correspondence between the pre-trained policies provided in this repo and the performance given in the paper figures. The given UOF policies were trained with AAES and 0.75 demonstration proportion.

run_uof.py

Arguments	Description
--task-id i	Task id, where,
--render	Use this flag if you want to render the task
--train	Use this flag for training an agent from scratch
--multi-inter	Use this flag to train separate high-level policies for each goal
--no-aaes	Use this flag to turn off the AAES exploration strategy
--no-demo	Use this flag to turn off the Abstract Demonstrations
--demo-proportion j	Use this flag to set the proportion of episodes that use demonstrations, where,

For run_hac.py, ignore the --multi-inter and --no-aaes arguments.