MapReduce optimization algorithm based on machine learning in heterogeneous cloud environment

doi:10.1016/S1005-8885(13)60112-0

中国邮电高校学报(英文) ›› 2013, Vol. 20 ›› Issue (6): 77-87.doi: 10.1016/S1005-8885(13)60112-0

MapReduce optimization algorithm based on machine learning in heterogeneous cloud environment

林文辉¹,雷振明²,刘军¹,杨洁³,刘芳³,何刚¹

1. Beijing Key Laboratory of Network System Architecture and Convergence, Beijing University of Posts and Telecommunications, Beijing 100876, China 2. School of Information Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China 3. Technology Research Institute, Aisino Corporation, Beijing 100195, China 4. School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China

收稿日期:2013-04-15 修回日期:2013-10-10 出版日期:2013-12-31 发布日期:2013-12-27
通讯作者: 林文辉 E-mail:linwh16@gmail.com
基金资助:
This work was supported by the Important National Science & Technology Specific Projects (2012ZX03002008), the 111 Project of China (B08004), and the Fundamental Research Funds for the Central Universities (2012RC0121).

MapReduce optimization algorithm based on machine learning in heterogeneous cloud environment

1. Beijing Key Laboratory of Network System Architecture and Convergence, Beijing University of Posts and Telecommunications, Beijing 100876, China 2. School of Information Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China 3. Technology Research Institute, Aisino Corporation, Beijing 100195, China 4. School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China

Received:2013-04-15 Revised:2013-10-10 Online:2013-12-31 Published:2013-12-27
Contact: Wenhui Lin E-mail:linwh16@gmail.com
Supported by:
This work was supported by the Important National Science & Technology Specific Projects (2012ZX03002008), the 111 Project of China (B08004), and the Fundamental Research Funds for the Central Universities (2012RC0121).

摘要/Abstract

摘要： We present an approach to optimize the MapReduce architecture, which could make heterogeneous cloud environment more stable and efficient. Fundamentally different from previous methods, our approach introduces the machine learning technique into MapReduce framework, and dynamically improve MapReduce algorithm according to the statistics result of machine learning. There are three main aspects: learning machine performance, reduce task assignment algorithm based on learning result, and speculative execution optimization mechanism. Furthermore, there are two important features in our approach. First, the MapReduce framework can obtain nodes’ performance values in the cluster through machine learning module. And machine learning module will daily calibrate nodes’ performance values to make an accurate assessment of cluster performance. Second, with the optimization of tasks assignment algorithm, we can maximize the performance of heterogeneous clusters. According to our evaluation result, the cluster performance could have 19% improvement in current heterogeneous cloud environment, and the stability of cluster has greatly enhanced.

关键词: cloud computing, MapReduce, machine learning, heterogeneity

Abstract: We present an approach to optimize the MapReduce architecture, which could make heterogeneous cloud environment more stable and efficient. Fundamentally different from previous methods, our approach introduces the machine learning technique into MapReduce framework, and dynamically improve MapReduce algorithm according to the statistics result of machine learning. There are three main aspects: learning machine performance, reduce task assignment algorithm based on learning result, and speculative execution optimization mechanism. Furthermore, there are two important features in our approach. First, the MapReduce framework can obtain nodes’ performance values in the cluster through machine learning module. And machine learning module will daily calibrate nodes’ performance values to make an accurate assessment of cluster performance. Second, with the optimization of tasks assignment algorithm, we can maximize the performance of heterogeneous clusters. According to our evaluation result, the cluster performance could have 19% improvement in current heterogeneous cloud environment, and the stability of cluster has greatly enhanced.

Key words: cloud computing, MapReduce, machine learning, heterogeneity

中图分类号:

TP393

[1]	肖雅郑世慧孙斌. Trusted GPSR protocol without reputation faking in VANET[J]. Acta Metallurgica Sinica(English letters), 2015, 22(5): 22-31.
[2]	石文孝吴丹许银龙王继红. Routing metric of interference-aware link quality: an improved ETX in wireless mesh networks[J]. Acta Metallurgica Sinica(English letters), 2014, 21(5): 61-67.
[3]	肖丁石川. Autonomic discovery of subgoals in hierarchical reinforcement learning[J]. Acta Metallurgica Sinica(English letters), 2014, 21(5): 94-104.
[4]	童俊杰鄂海红宋美娜宋俊德. Host load prediction in cloud based on classification methods[J]. Acta Metallurgica Sinica(English letters), 2014, 21(4): 40-46.
[5]	徐嬴颖陈常嘉赵永祥陈一帅. Cloud download system optimizing by job and notification scheduling[J]. Acta Metallurgica Sinica(English letters), 2014, 21(4): 47-53.
[6]	王若愚刘珍张凌. Method of data cleaning for network traffic classification[J]. Acta Metallurgica Sinica(English letters), 2014, 21(3): 35-45.
[7]	李文霁郑康锋张冬梅 YE-Qing YANG Yi-xian. Efficient identity-based signature scheme with batch authentication for delay tolerant mobile sensor network [J]. Acta Metallurgica Sinica(English letters), 2013, 20(4): 80-86.
[8]	付雄 ZHU Xin-xin, HAN Jing-yu, WANG Ru-chuan. QoS-aware replica placement for data intensive applications[J]. Acta Metallurgica Sinica(English letters), 2013, 20(3): 43-47.
[9]	姚玉坤王冠任智李鹏翔陈永超. Efficient distributed address assignment algorithm based on topology maintenance in ZigBee networks[J]. Acta Metallurgica Sinica(English letters), 2013, 20(3): 53-59.
[10]	张招亮李栋黄庭培崔莉. Leveraging data fusion to improve barrier coverage in wireless sensor networks[J]. Acta Metallurgica Sinica(English letters), 2013, 20(1): 26-36.
[11]	沙超王汝传. Energy-efficient node deployment strategy forwireless sensor networks[J]. Acta Metallurgica Sinica(English letters), 2013, 20(1): 54-57.
[12]	姚玉坤温亚迪任智刘智虎. High efficient multipacket decoding approach for network coding in wireless networks[J]. Acta Metallurgica Sinica(English letters), 2013, 20(1): 95-100.
[13]	詹振球朱杰孙宜进. Further results on local stability of LRC-RED algorithm in Internet[J]. Acta Metallurgica Sinica(English letters), 2012, 19(5): 99-103.
[14]	詹振球朱杰徐迪. Stability analysis in an AVQ model of Internet congestion control algorithm[J]. Acta Metallurgica Sinica(English letters), 2012, 19(4): 22-28.
[15]	吴吉祥夏靖波. Enhanced calculation of necessary QoS for user satisfaction with a QoS mapping matrix[J]. Acta Metallurgica Sinica(English letters), 2012, 19(4): 29-33.

MapReduce optimization algorithm based on machine learning in heterogeneous cloud environment

MapReduce optimization algorithm based on machine learning in heterogeneous cloud environment

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价