Article Preview
Top1. Introduction
In recent years, with the rapid development of the Internet, network service has become one of the most frequently used computer applications. Search engine, webmail, and social network services are presently crucial data intensive applications. Search engine, webmail, and social network services are presently crucial data intensive applications. Because increasingly more people use web services, processing a big amount of data efficiently can be a significant problem. The very word “Big” indicates size. Big data have rapidly developed into a hotspot that attracts great attention to academia, industry, and even governments around the world (Mayer-Schonberger, & Cukier, 2013). Big data is of great value, which is beyond all doubt. From the perspective of the information industry, big data is a strong impetus to the next generation of IT industry, which is essentially built on the third platform, mainly referring to big data, cloud computing, mobile Internet, and social business. IDC predicted that by 2020 the market size of the third IT platform will reach US$ 5.3 trillion; and from 2013 to 2020, 90% of the growth in the IT industry would be driven by the third IT platform. From the socio-economic point of view, big data is the core connotation and critical support of the so-called second economy, a concept proposed by the American economist (W.B. Arthur, 2011), which refers to the economic activities running on processors, connectors, sensors, and executors. It is estimated that at 2030 the size of the second economy will approach that of the first economy (namely, the traditional physical economy). The main support of the second economy is big data, as it is an inexhaustible and constantly enriching resource. In the future, by virtue of big data, the competence under the second economy will no longer be that of labor productivity but of knowledge productivity.
Compare with traditional datasets, big data typically includes masses of unstructured data that need more real-time analysis. In addition, big data also bring about new opportunities for discovering new values, helps us to gain an in-depth understanding of the hidden values, and also incurs new challenges, e.g. how effectively organized and manage such datasets. Recently, industries become interested in the high potential of big data, and many government agencies announced major plans to accelerate big data research and applications (Federal,2012). Nowadays, big data related to the service of Internet companies grow rapidly. For example, Google processes data of hundreds of Petabyte (PB), Facebook generates log data of over 10 PB per month, Baidu, a Chinese company, processes data of 10 PB per day, and Taobao, a subsidiary of Alibaba, generates data of tens of Terabyte (TB) for online trading per day. The features of big data can be characterized by 5V, namely, huge Volume, high Velocity, high Variety, low Veracity, and high Value. The main difficulty in coping with big data does not only lie in its huge volume, as we may alleviate to some extent this issue by reasonably expanding or extending our computing systems. Actually, the real challenges center on the diversified data types (Variety), timely response requirements (Velocity), and uncertainties in the data (Veracity). Because of the diversified data types, an application often needs to deal with not only traditional structured data, but also semi-structured or unstructured data (including text, images, video, and voice). Timely responses are also challenging because there may not be enough resources to collect, store, and process the big data within a reasonable amount of time.
Big document summarization method is an important technique for data management of Internet of Things (IoT). Document summarization is the process of reducing the sizes of documents while maintaining their basic outlines. That is, it should extract the most significant information from the document. The summarization method can involve either generic summaries or query-based summaries. A generic summary distills an overall wisdom of a document’s contents, whereas a query-based summary distills only the contents of a document that is relevant to a user’s query. Traditional document summarization methods are restricted for summarizing suitable information from the big document data, since it has been proposed for enhancing the summarization precision which it uses various statistical or natural language processing methods based on single node computer environment. The objective of this paper is proposed big document summarization method which the information is summarized from a big document data. The proposed method uses the novel clustering algorithm and extracted semantic feature of document using Haddop.