Research on Improved Method of Storage and Query of Large-Scale Remote Sensing Images

Research on Improved Method of Storage and Query of Large-Scale Remote Sensing Images

Jing Weipeng (Northeast Forestry University, Harbin, China), Tian Dongxue (Northeast Forestry University, Harbin, China), Chen Guangsheng (Northeast Forestry University, Harbin, China) and Li Yiyuan (China Aerospace Construction Group Co., LTD, Beijing, China)
Copyright: © 2018 |Pages: 16
DOI: 10.4018/JDM.2018070101

Abstract

The traditional method is used to deal with massive remote sensing data stored in low efficiency and poor scalability. This article presents a parallel processing method based on MapReduce and HBase. The filling of remote sensing images by the Hilbert curve makes the MapReduce method construct pyramids in parallel to reduce network communication between nodes. Then, the authors design a massive remote sensing data storage model composed of metadata storage model, index structure and filter column family. Finally, this article uses MapReduce frameworks to realize pyramid construction, storage and query of remote sensing data. The experimental results show that this method can effectively improve the speed of data writing and querying, and has good scalability.
Article Preview
Top

In the past few years, some efforts have been made to achieve efficient storage and processing through high performance cloud computing technology. Zheng, and Fu (2013) used the Hadoop platform to store vector space data. This method effectively solves the problems of single node failure and lack of extensibility. However, the problem of generating a lot of small files in the process of storing data is not considered, which results in a large amount of memory being used to increase the burden on the primary node. This is because HDFS is a bottleneck caused by managing the metadata of multiple slaves (DataNodes) based on a single master server (NameNode). Especially in processing a large number of small files, in order to improve processing efficiency, the NameNode stores the entire metadata of the HDFS in the main memory. Zhong et al. (2011) proposed a distributed key-value storage model to manage massive image data and store small files in large data files, which effectively solved a large number of small file problems. But, it doesn’t describe of each layer of image metadata, thus increasing the data retrieval time. Chi et al. (2013) designed metadata for each layer of image and used MapFile to store the images generated during the data merge process, thus improving the accessibility of the data. Patel, and Mehta (2015) proposed approach combines the correlated files into one single file to reduce the metadata storage on Namenode to improve the access efficiency of small files. The above methods all store remote sensing data directly on HDFS, and HDFS supports update operations on files is poor. Because remote sensing images have the characteristics of multiple phases, it is necessary to update the data information at different times in the same region in real time. Therefore, the above method cannot directly update the data information of the corresponding location.

Complete Article List

Search this Journal:
Reset
Open Access Articles
Volume 31: 4 Issues (2020): Forthcoming, Available for Pre-Order
Volume 30: 4 Issues (2019)
Volume 29: 4 Issues (2018)
Volume 28: 4 Issues (2017)
Volume 27: 4 Issues (2016)
Volume 26: 4 Issues (2015)
Volume 25: 4 Issues (2014)
Volume 24: 4 Issues (2013)
Volume 23: 4 Issues (2012)
Volume 22: 4 Issues (2011)
Volume 21: 4 Issues (2010)
Volume 20: 4 Issues (2009)
Volume 19: 4 Issues (2008)
Volume 18: 4 Issues (2007)
Volume 17: 4 Issues (2006)
Volume 16: 4 Issues (2005)
Volume 15: 4 Issues (2004)
Volume 14: 4 Issues (2003)
Volume 13: 4 Issues (2002)
Volume 12: 4 Issues (2001)
Volume 11: 4 Issues (2000)
Volume 10: 4 Issues (1999)
Volume 9: 4 Issues (1998)
Volume 8: 4 Issues (1997)
Volume 7: 4 Issues (1996)
Volume 6: 4 Issues (1995)
Volume 5: 4 Issues (1994)
Volume 4: 4 Issues (1993)
Volume 3: 4 Issues (1992)
Volume 2: 4 Issues (1991)
Volume 1: 2 Issues (1990)
View Complete Journal Contents Listing