A Review of Personalized Federated Reinforcement Learning

Authors

  • Gaofeng Chen
  • Qingtao Wu

DOI:

https://doi.org/10.62051/ijcsit.v3n1.01

Keywords:

Federated reinforcement learning, Heterogeneity factors, Personalized model training

Abstract

Reinforcement learning and federated learning both provide strong theoretical support for the study of artificial intelligence. In recent years, an emerging federated reinforcement learning paradigm has been proposed and widely studied and applied. However, in the federated reinforcement learning architecture, the environment, data type and device performance of different agents may be different, which is called heterogeneity. The existence of heterogeneity factors may lead to slow convergence speed of the algorithm, poor generalization, and suboptimal quality of the trained model. Therefore, how to solve the negative impact of the heterogeneity problem on model training has become a hot content of research, and the most important method is to train personalized models for clients. This paper introduces the theory of federated reinforcement learning, as well as methods to cope with heterogeneity in federated reinforcement learning, and provides an overview of the applications of federated reinforcement learning. We conclude the paper with a summary and future perspectives.

Downloads

Download data is not yet available.

References

Liu X, Fu X, Du X, et al. Machine learning based non-intrusive digital forensic service for smart homes [J]. IEEE Transactions on Network and Service Management, 2023, 20(2): 945-960.

Kiran B R, Sobh I, Talpaert V, et al. Deep reinforcement learning for autonomous driving: A survey [J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 23(6): 4909-4926.

Liu B, Wang L, Liu M. Lifelong federated reinforcement learning: A learning architecture for navigation in cloud robotic systems [J]. IEEE Robotics and Automation Letters, 2019, 4(4): 4555-4562.

Zhou Z H, Yu Y, Qian C. Evolutionary learning: Advances in theories and algorithms [M]. Singapore: Springer Singapore, 2019.

Mjolsness E, DeCoste D. Machine learning for science: State of the art and future prospects [J]. Science, 2001, 293(5537): 2051-2055.

McMahan B, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data [C]. Proceedings of the 20th International Conference on Artificial In-telligence and Statistics, 2017: 1273-1282.

Konečný J, McMahan H B, Ramage D, et al. Federated optimization: Distributed machine learning for on-device intelligence [J]. arXiv preprint arXiv:1610.02527, 2016.

Yang Q, Liu Y, Chen T, et al. Federated machine learning: Concept and applications [J]. ACM Transactions on Intelligent Systems and Technology (TIST), 2019, 10(2): 1-19.

Konečný J, McMahan H B, Yu F X, et al. Federated learning: Strategies for improving com-munication efficiency [J]. arXiv preprint arXiv:1610.05492, 2016.

Sutton R S, Barto A G. Reinforcement learning: An introduction [M]. MIT press, 2018.

Den Hengst F, Grua E M, el Hassouni A, et al. Reinforcement learning for personalization: A systematic literature review [J]. Data Science, 2020, 3(2): 107-147.

Jin H, Peng Y, Yang W, et al. Federated reinforcement learning with environment heterogenei-ty [C]. International Conference on Artificial Intelligence and Statistics, 2022: 18-37.

Lin Q, Ling Q. Byzantine-robust federated deep deterministic policy gradient [C]. IEEE Inter-national Conference on Acoustics, Speech and Signal Processing, 2022: 4013-4017.

Wu Q, He K, Chen X. Personalized federated learning for intelligent IoT applications: A cloud-edge based framework [J]. IEEE Open Journal of the Computer Society, 2020, 1: 35-44.

Kulkarni V, Kulkarni M, Pant A. Survey of personalization techniques for federated learn-ing [C]. IEEE Fourth World Conference on Smart Trends in Systems, Security and Sustainability, 2020: 794-797.

Sheng X, Gao Z, Cui X, et al. Federated reinforcement learning technology and application in edge intelligence scene [C]. International Conference on Emerging Internetworking, Data & Web Technologies. Cham: Springer International Publishing, 2023: 284-291.

Lauer M, Riedmiller M A. An algorithm for distributed reinforcement learning in cooperative multi-agent systems [C]. Proceedings of the 17th International Conference on Machine Learning. 2000: 535-542.

Jiang Y, Konečný J, Rush K, et al. Improving federated learning personalization via model agnostic meta learning [J]. arXiv preprint arXiv:1909.12488, 2019.

Nadiger C, Kumar A, Abdelhak S. Federated reinforcement learning for fast personalization [C]. IEEE Second International Conference on Artificial Intelligence and Knowledge Engineering . 2019: 123-127.

Wu J, Liu X, Liu J, et al. DPFed: Toward fair personalized federated learning with fast convergence [C]. Proceedings of the 18th IEEE International Conference on Mobility, Sensing and Networking, 2022: 510-517.

Lim W Y B, Luong N C, Hoang D T, et al. Federated learning in mobile edge networks: A comprehensive survey [J]. IEEE Communications Surveys & Tutorials, 2020, 22(3): 2031-2063.

Wang X, Han Y, Wang C, et al. In-edge AI: Intelligentizing mobile edge computing, caching and communication by federated learning [J]. IEEE Network, 2019, 33(5): 156-165.

Zhang M, Jiang Y, Zheng F C, et al. Cooperative edge caching via federated deep reinforcement learning in fog-rans [C]. IEEE International Conference on Communications Workshops. 2021: 1-6.

Tianqing Z, Zhou W, Ye D, et al. Resource allocation in IoT edge computing via concurrent federated reinforcement learning [J]. IEEE Internet of Things Journal, 2021, 9(2): 1414-1426.

Wang J, Hu J, Mills J, et al. Federated ensemble model-based reinforcement learning in edge computing [J]. IEEE Transactions on Parallel and Distributed Systems, 2023.

Wang C, Yao T, Fan T, et al. Modeling on resource allocation for age-sensitive mobile edge computing using federated multi-agent reinforcement learning [J]. IEEE Internet of Things Journal, 2023.

Liang X, Liu Y, Chen T, et al. Federated transfer reinforcement learning for autonomous driving [M]. Federated and Transfer Learning. Cham: Springer International Publishing, 2022: 357-371.

Na S, Rouček T, Ulrich J, et al. Federated reinforcement learning for collective navigation of robotic swarms [J]. IEEE Transactions on cognitive and developmental systems, 2023.

Ho T M, Nguyen K K, Cheriet M. Federated deep reinforcement learning for task scheduling in heterogeneous autonomous robotic system [J]. IEEE Transactions on Automation Science and Engineering, 2022.

Huang H, Zeng C, Zhao Y, et al. Scalable orchestration of service function chains in NFV-enabled networks: A federated reinforcement learning approach [J]. IEEE Journal on Selected Areas in Communications, 2021, 39(8): 2558-2571.

Cao Y, Lien S Y, Liang Y C, et al. Federated deep reinforcement learning for user access control in open radio access networks [C]. IEEE International Conference on Communications. 2021: 1-6.

Krouka M, Elgabli A, Issaid C B, et al. Communication-efficient and federated multi-agent reinforcement learning [J]. IEEE Transactions on Cognitive Communications and Networking, 2021, 8(1): 311-320.

Gupta A, Fernando X. Co-operative edge intelligence for C-V2X communication using federated reinforcement learning [C]. IEEE 34th Annual International Symposium on Personal, Indoor and Mobile Radio Communications. 2023: 1-6.

Downloads

Published

15-06-2024

Issue

Section

Articles

How to Cite

Chen, G., & Wu, Q. (2024). A Review of Personalized Federated Reinforcement Learning. International Journal of Computer Science and Information Technology, 3(1), 1-9. https://doi.org/10.62051/ijcsit.v3n1.01