This repository contains the implementation of the paper Model-aided Federated Reinforcement Learning for Multi-UAV Trajectory Planning in IoT Networks.

• python 3.7.15 or newer
• numpy 1.21.5 or newer
• torch 1.12.1 or newer
• matplotlib 3.5.3 or newer
• pyswarms 1.3.0 or newer

We gratefully acknowledge the work Model-aided Deep Reinforcement Learning for Sample-efficient UAV Trajectory Design in IoT Networks, which we have expanded upon to accommodate multi-UAV scenarios. Our implementation of the QMIX and IQL algorithms was substantially based on starry-sky6688/marl-algorithms. Additionally, our simulation environment design drew inspiration from the SMAC.

$ python main.py --map --federated --model --alg --n_agents --tag --total_episodes --device

--map: the map name, RBM/RDM
--federated: whether to use federated learning
--model: whether to use model-aided learning
--alg: the algorithm name, qmix/iql
--n_agents: the number of agents, which is equal to the number of workers in federated learning
--tag: the tag of the experiment
--total_episodes: the total number of collected episodes
--device: the device to run the experiment, cpu/cuda

For example, if you want to train a model-aided FedQMIX on the RDM map with 3 agents, you can run the following command:

$ python main.py --map=RDM --federated=True --model=True --alg=qmix --n_agents=3 --tag=model_aided_fedqmix --total_episodes=30000

If you want to evaluate the above trained model, you can run the following command:

$ python evaluate.py --map=RDM --model=True --alg=qmix --tag=model_aided_fedqmix --n_agents=3 

Once the model training is complete, you can visualize the results using the Plot.py script.

We also provide an algorithm to improve the model learning performance by using k-means algorithm, you can try it by setting --model=True --sample_method=kmeans.

If you find this repository useful, please cite our paper:

@article{chen2023model,
  title={Model-aided Federated Reinforcement Learning for Multi-UAV Trajectory Planning in IoT Networks},
  author={Chen, Jichao and Esrafilian, Omid and Bayerlein, Harald and Gesbert, David and Caccamo, Marco},
  journal={arXiv preprint arXiv:2306.02029},
  year={2023}
}