中国邮电高校学报(英文) ›› 2014, Vol. 21 ›› Issue (1): 60-66.doi: 10.1016/S1005-8885(14)60269-7

• Networks • 上一篇    下一篇

Sensor data compression based on MapReduce

于雨,郭忠文   

  1. Department of Computer Science and Engineer, Ocean University of China, Qingdao 266100, China
  • 收稿日期:2013-08-09 修回日期:2014-01-05 出版日期:2014-02-28 发布日期:2014-02-28
  • 通讯作者: 郭忠文 E-mail:guozhw2007@163.com
  • 基金资助:

    This work was supported by the National Natural Science Foundation of China (60933011, 61170258).

Sensor data compression based on MapReduce

Yu Yu1,   

  1. Department of Computer Science and Engineer, Ocean University of China, Qingdao 266100, China
  • Received:2013-08-09 Revised:2014-01-05 Online:2014-02-28 Published:2014-02-28
  • Supported by:

    This work was supported by the National Natural Science Foundation of China (60933011, 61170258).

摘要:

A compression algorithm is proposed in this paper for reducing the size of sensor data. By using a dictionary-based lossless compression algorithm, sensor data can be compressed efficiently and interpreted without decompressing. The correlation between redundancy of sensor data and compression ratio is explored. Further, a parallel compression algorithm based on MapReduce [1] is proposed. Meanwhile, data partitioner which plays an important role in performance of MapReduce application is discussed along with performance evaluation criteria proposed in this paper. Experiments demonstrate that random sampler is suitable for highly redundant sensor data and the proposed compression algorithms can compress those highly redundant sensor data efficiently.

关键词:

data compression, sensor data, MapReduce, surveillance application, measurement system

Abstract:

A compression algorithm is proposed in this paper for reducing the size of sensor data. By using a dictionary-based lossless compression algorithm, sensor data can be compressed efficiently and interpreted without decompressing. The correlation between redundancy of sensor data and compression ratio is explored. Further, a parallel compression algorithm based on MapReduce [1] is proposed. Meanwhile, data partitioner which plays an important role in performance of MapReduce application is discussed along with performance evaluation criteria proposed in this paper. Experiments demonstrate that random sampler is suitable for highly redundant sensor data and the proposed compression algorithms can compress those highly redundant sensor data efficiently.

Key words:

data compression, sensor data, MapReduce, surveillance application, measurement system

中图分类号: