Dynamic computation offloading in time-varying environment for ultra-dense networks: a stochastic game approach

doi:10.19682/j.cnki.1005-8885.2021.1003

中国邮电高校学报(英文) ›› 2021, Vol. 28 ›› Issue (2): 24-37.doi: 10.19682/j.cnki.1005-8885.2021.1003

Dynamic computation offloading in time-varying environment for ultra-dense networks: a stochastic game approach

Xie Renchao, Liu Xu, Duan Xuefei, Tang Qinqin, Yu Fei Richard, Huang Tao

1. State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China
2. Guangdong Communications and Networks Institute, Guangzhou 510700, China
3. Department of Future Networks, Purple Mountain Laboratory, Nanjing 211111, China
4. Department of Systems and Computer Engineering, Carleton University, Ottawa, Canada

收稿日期:2020-08-19 修回日期:2021-02-24 出版日期:2021-04-30 发布日期:2021-04-30
通讯作者: Xie Renchao E-mail:renchao_xie@bupt.edu.cn

Dynamic computation offloading in time-varying environment for ultra-dense networks: a stochastic game approach

Xie Renchao, Liu Xu, Duan Xuefei, Tang Qinqin, Yu Fei Richard, Huang Tao

1. State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China
2. Guangdong Communications and Networks Institute, Guangzhou 510700, China
3. Department of Future Networks, Purple Mountain Laboratory, Nanjing 211111, China
4. Department of Systems and Computer Engineering, Carleton University, Ottawa, Canada

Received:2020-08-19 Revised:2021-02-24 Online:2021-04-30 Published:2021-04-30
Contact: Xie Renchao E-mail:renchao_xie@bupt.edu.cn

摘要/Abstract

摘要： To meet the demands of large-scale user access with computation-intensive and delay-sensitive applications,
combining ultra-dense networks (UDNs) and mobile edge computing (MEC)are considered as important solutions.
In the MEC enabled UDNs, one of the most important issues is computation offloading. Although a number of work
have been done toward this issue, the problem of dynamic computation offloading in time-varying environment,
especially the dynamic computation offloading problem for multi-user, has not been fully considered. Therefore, in
order to fill this gap, the dynamic computation offloading problem in time-varying environment for multi-user is
considered in this paper. By considering the dynamic changes of channel state and users queue state, the dynamic
computation offloading problem for multi-user is formulated as a stochastic game, which aims to optimize the delay
and packet loss rate of users. To find the optimal solution of the formulated optimization problem, Nash Q-learning
(NQLN) algorithm is proposed which can be quickly converged to a Nash equilibrium solution. Finally, extensive
simulation results are presented to demonstrate the superiority of NQLN algorithm. It is shown that NQLN algorithm
has better optimization performance than the benchmark schemes.

关键词: dynamic computation offloading, time-varying environment, stochastic game, ultra-dense networks (UDNs), mobile edge computing (MEC)

Abstract: To meet the demands of large-scale user access with computation-intensive and delay-sensitive applications,
combining ultra-dense networks (UDNs) and mobile edge computing (MEC)are considered as important solutions.
In the MEC enabled UDNs, one of the most important issues is computation offloading. Although a number of work
have been done toward this issue, the problem of dynamic computation offloading in time-varying environment,
especially the dynamic computation offloading problem for multi-user, has not been fully considered. Therefore, in
order to fill this gap, the dynamic computation offloading problem in time-varying environment for multi-user is
considered in this paper. By considering the dynamic changes of channel state and users queue state, the dynamic
computation offloading problem for multi-user is formulated as a stochastic game, which aims to optimize the delay
and packet loss rate of users. To find the optimal solution of the formulated optimization problem, Nash Q-learning
(NQLN) algorithm is proposed which can be quickly converged to a Nash equilibrium solution. Finally, extensive
simulation results are presented to demonstrate the superiority of NQLN algorithm. It is shown that NQLN algorithm
has better optimization performance than the benchmark schemes.

Key words: dynamic computation offloading, time-varying environment, stochastic game, ultra-dense networks (UDNs), mobile edge computing (MEC)

中图分类号:

TN915.01

Xie Renchao, Liu Xu, Duan Xuefei, Tang Qinqin, Yu Fei Richard, Huang Tao. Dynamic computation offloading in time-varying environment for ultra-dense networks: a stochastic game approach[J]. The Journal of China Universities of Posts and Telecommunications, 2021, 28(2): 24-37.

参考文献

1. Ge X H, Tu S, Mao G Q, et al. 5G ultra-dense cellular networks. IEEE Wireless Communications, 2016, 23(1): 72-79

2. Kamel M, Hamouda W, Youssef A. Ultra-dense networks: a survey. IEEE Communications Surveys & Tutorials, 2016, 18(4): 2522-2545

3. An J P, Yang K, Wu J S, et al. Achieving sustainable ultra-dense heterogeneous networks for 5G. IEEE Communications Magazine, 2017, 55(12): 84-90

4. Mobile edge computing (MEC); framework and reference architecture. ETSI GS MEC 003 V1.1.1. Sophia Antipolis, Franc: ETSI, 2016

5. Mach P, Becvar Z. Mobile edge computing: a survey on architecture and computation offloading. IEEE Communications Surveys & Tutorials, 2017, 19(3): 1628-1656

6. Wang X F, Han Y W, Wang C Y, et al. In-edge AI: intelligentizing mobile edge computing, caching and communication by federated learning. IEEE Network, 2019, 33(5): 156-165

7. Ren J K, Yu G D, He Y H, et al. Collaborative cloud and edge computing for latency minimization. IEEE Trans on Vehicular Technology, 2019, 68(5): 5031-5044

8. Zhang H B, Wang R, Liu J J. Mobility management for ultra-dense edge computing: a reinforcement learning approach. Proceedings of the IEEE 90th Vehicular Technology Conference (VTC’19-Fall), Sept 22-25, 2019, Honolulu, HI, USA. Piscataway, NJ, USA: IEEE, 2019: 1-5

9. Chen S Y, Zhao T Y, Chen H H. et al. Downlink coordinated multi-point transmission in ultra-dense networks with mobile edge computing. IEEE Network, 2019, 33(2): 152-159

10. Wang Q, Zhou F H. Fair resource allocation in an MEC-enabled ultra-dense IoT network with NOMA. Proceedings of the 2019 IEEE International Conference on Communications Workshops (ICC Workshops’19), May 20-24, 2019, Shanghai, China. Piscataway, NJ, USA: IEEE, 2019: 1-6

11. Seng S M, Li X, Luo C Q, et al. A D2D-assisted MEC computation offloading in the blockchain-based framework for UDNs. Proceedings of the 2019 IEEE International Conference on Communications (ICC’19), May 20-24, 2019, Shanghai, China. Piscataway, NJ, USA: IEEE, 2019: 1-6

12. Guo H Z, Zhang J, Liu J J, et al. Energy-efficient task offloading and transmit power allocation for ultra-dense edge computing. Proceedings of the 2018 IEEE Global Communications Conference (GLOBECOM’18), Dec 9-13, 2018, Abu Dhabi, United Arab Emirates. Piscataway, NJ, USA: IEEE, 2018: 1-6

13. Li L J, Zhou H M, Xiong S X, et al. Compound model of task arrivals and load-aware offloading for vehicular mobile edge computing networks. IEEE Access, 2019, 7: 26631-26640

14. Guo F X, Zhang H L, Ji H, et al. An efficient computation offloading management scheme in the densely deployed small cell networks with mobile edge computing. IEEE/ACM Trans on Networking, 2018, 26(6): 2651-2664

15. Liu C B, Li K L, Liang J, et al. COOPER-MATCH: job offloading with a cooperative game for guaranteeing strict deadlines in MEC. IEEE Trans on Mobile Computing, 2019, DOI: 10.1109/TMC.2019.2921713

16. Asheralieva A, Niyato D. Hierarchical game-theoretic and reinforcement learning framework for computational offloading in UAV-enabled mobile edge computing networks with multiple service providers. IEEE Internet of Things Journal, 2019, 6(5): 8753-8769

17. Chen X, Liu Z Y, Chen Y, et al. Mobile edge computing based task offloading and resource allocation in 5G ultra-dense networks. IEEE Access, 2019, 7: 184172-184182

18. Xie R C, Tang Q Q, Liang C H, et al. Dynamic computation offloading in IoT fog systems with imperfect channel-state information: a POMDP approach. IEEE Internet of Things Journal, 2021, 8(1): 345-356

19. Van Le D, Tham C. Quality of service aware computation offloading in an ad-hoc mobile cloud. IEEE Trans on Vehicular Technology, 2018, 67(9): 8890-8904

20. Guo H Z, Liu J J, Qin H L. Collaborative mobile edge computation offloading for IoT over fiber-wireless networks. IEEE Network, 2018, 32(1): 66-71

21. Fu F W, van der Schaar M. Learning to compete for resources in wireless stochastic games. IEEE Trans on Vehicular Technology, 2009, 58(4): 1904-1919

22. Bao X C, Liang H, Liu Y, et al. A stochastic game approach for collaborative beamforming in SDN-based energy harvesting wireless sensor networks. IEEE Internet of Things Journal, 2019, 6(6): 9583-9595

23. Wei Z L, Zhao B K, Su J S, et al. Dynamic edge computation offloading for internet of things with energy harvesting: a learning method. IEEE Internet of Things Journal, 2019, 6(3): 4436-4447

24. Wang K L, Tan Y Y, Shao Z Y, et al. Learning-based task offloading for delay-sensitive applications in dynamic fog networks. IEEE Trans on Vehicular Technology, 2019, 68(11): 11399-11403

25. Chen L X, Zhou S, Xu J. Computation peer offloading for energy-constrained mobile edge computing in small-cell networks. IEEE/ACM Trans on Networking, 2018, 26(4): 1619-1632

26. Zhang H J, Duan Y N, Long K P, et al. Energy Efficient resource allocation in terahertz downlink NOMA systems. IEEE Trans on Communications, 2021, 69(2): 1375-1384

27. Zheng J C, Cai Y M, Wu Y, et al. Dynamic computation offloading for mobile cloud computing: a stochastic game-theoretic approach. IEEE Trans on Mobile Computing, 2019, 18(4): 771-786

28. Zhang L, Cao B. Stochastic programming method for offloading in mobile edge computing based internet of vehicle. Proceedings of the 2019 IEEE International Conference on Communications (ICC’19), May 20-24, 2019, Shanghai, China. Piscataway, NJ, USA: IEEE, 2019: 1-6

29. Wang B B, Wu Y L, Ray Liu K J. Game theory for cognitive radio networks: an overview. Computer Networks, 2010, 54(14): 2537-2561

30. Chen X, Jiao L, Li W Z, et al. Efficient multi-user computation offloading for mobile-edge cloud computing. IEEE/ACM Trans on Networking, 2016, 24(5): 2795-2808

31. Sun Y X, Zhou S, Xu J. EMM: energy-aware mobility management for mobile edge computing in ultra-dense networks. IEEE Journal on Selected Areas in Communications, 2017, 35(11): 2637-2646

32. Huang Y F, Tan T H, Wang N C, et al. Resource allocation for D2D communications with a novel distributed Q-learning algorithm in heterogeneous networks. Proceedings of the 2018 International Conference on Machine Learning and Cybernetics (ICMLC’18): Vol 2, Jul 15-18, 2018, Chengdu, China. Piscataway, NJ, USA: IEEE, 2018: 533-537

33. Dab B, Aitsaadi N, Langar R. Q-learning algorithm for joint computation offloading and resource allocation in edge cloud. Proceedings of the 2019 IFIP/IEEE Symposium on Integrated Network and Service Management (IM’19), Apr 8-12, 2019, Arlington, VA, USA. Piscataway, NJ, USA: IEEE, 2019: 45-52

34. Hu J L, Wellman M P. Nash Q-learning for general-sum stochastic games. Journal of Machine Learning Research, 2003, 4: 1039-1069

35. Shams F, Bacci G, Luise M. Energy-efficient power control for multiple-relay cooperative networks using Q-learning. IEEE Trans on Wireless Communications, 2015, 14(3): 1567-1580

36. Ke H C, Wang J, Wang H, et al. Joint optimization of data offloading and resource allocation with renewable energy aware for IoT devices: a deep reinforcement learning approach. IEEE Access, 2019, 7: 179349-179363

37. Guo K, Yang M C, Zhang Y B, et al. An efficient dynamic offloading approach based on optimization technique for mobile edge computing. Proceedings of the 6th IEEE International Conference on Mobile Cloud Computing, Services, and Engineering (MobileCloud’18), Mar 26-29, 2018, Bamberg, Germany. Piscataway, NJ, USA: IEEE, 2018: 29-36

38. Xiao L, Xie C X, Chen T H, et al. A mobile offloading game against smart attacks. IEEE Access, 2016, 4: 2281-2291

39. Liu X L, Yu J D, Wang J, et al. Resource allocation with edge computing in IoT networks via machine learning. IEEE Internet of Things Journal, 2020, 7(4): 3415-3426

[1]	曹建玲王敏陈葱任智. High-throughput low-delay MAC protocol for TeraHertz ultra- high data-rate wireless networks[J]. 中国邮电高校学报(英文版), 2016, 23(4): 17-24.
[2]	崔琪楣，刘俊，陶小峰，张平. CRLB analysis of wireless cognitive location with different short-range measurements[J]. Acta Metallurgica Sinica(English letters), 2008, 15(2): 1-6.
[3]	宋兴华，吴伟陵. Downlink scheduling of multiuser MIMO systems with transmit beamforming[J]. Acta Metallurgica Sinica(English letters), 2008, 15(2): 45-49.
[4]	张奇支. On the M/G/1 queueing system with multiclass customers and fixed feedback[J]. Acta Metallurgica Sinica(English letters), 2008, 15(2): 114-121.
[5]	朱晓暄;石明洋;张静;陶小峰. Non-line-of-sight error mitigating algorithm based on scattering models via multi-antenna system[J]. Acta Metallurgica Sinica(English letters), 2008, 15(1): 18-22.
[6]	曾宪;马涛;林粤伟;冯志勇. Genetic algorithm for autonomic joint radio resource management in end-to-end reconfigurable systems[J]. Acta Metallurgica Sinica(English letters), 2008, 15(1): 11-17.
[7]	胡南;吴伟陵. Call admission control scheme for real-time services in packet-switched OFDM wireless networks[J]. Acta Metallurgica Sinica(English letters), 2008, 15(1): 80-84.
[8]	胡甜;赵保华. Load-balanced broadcast routing in wireless Ad-hoc networks[J]. Acta Metallurgica Sinica(English letters), 2007, 14(3): 1-4.
[9]	谢正程;邢俊威;武丽;纪越峰. Shared p-cycles design for dual link failure restorability in optical WDM networks[J]. Acta Metallurgica Sinica(English letters), 2007, 14(3): 74-78.
[10]	刘俊; 崔琪楣;陶晓峰. Improved taylor-series location algorithm for cooperation networks[J]. Acta Metallurgica Sinica(English letters), 2007, 14(2): 38-44.
[11]	XIE Sheng-dong, WU Meng. Adaptive vertical handoff algorithm in heterogeneous networks[J]. Acta Metallurgica Sinica(English letters), 2007, 14(1): 22-26.
[12]	FAN Li-min, LIAO Jian-xin. Discrete micropayment protocol based on master-slave payword chain[J]. Acta Metallurgica Sinica(English letters), 2007, 14(1): 58-60.
[13]	WEN Dan-hui, CHEN Rong-bang, LU Ting-jie. Factors for wireless operators’ collusion and competition[J]. Acta Metallurgica Sinica(English letters), 2007, 14(1): 61-64.
[14]	ZOU Ting, DENG Gang, WANG Ying, ZHANG Ping. Subcarrier and power allocation algorithm based on inter-cell interference mitigation for OFDMA system[J]. Acta Metallurgica Sinica(English letters), 2007, 14(1): 90-95.
[15]	LIU Bing;TANG Shou-lian. Revenue-sharing analysis in the mobile value-added services[J]. Acta Metallurgica Sinica(English letters), 2006, 13(4): 102-105.

Dynamic computation offloading in time-varying environment for ultra-dense networks: a stochastic game approach

Dynamic computation offloading in time-varying environment for ultra-dense networks: a stochastic game approach

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价