Article Preview
TopIntroduction
In recent scenario the modern social media, mobile or web strategy involved in communication has been technology concentric that makes data to grow rapidly, ultimately creates large noisy and unstructured data. This gave sudden rise (Felt, 2016) to concept of Bigdata. (Kitchin, 2014) describe Bigdata by 3 V’s: large volume, uninterruptable velocity, different data structure as variety which can be extensive in opportunity, interactive in nature, and springy in quality. Many theories in social science like correlation has been proven to be pertinent to social media. As per social correlation theory (Tang, Tan, & Liu, 2014), contiguous users in a social media have similar behaviors or attributes. These phenomena clarify user’s inclination to connect or follow with others having certain similarity or sharing the same surroundings. The quantity of information available for harnessing in social media is massive and growing every second. Increasing volumes of data (Tang et al., 2014), has been a major challenge for the data oriented companies like Google, Yahoo, LinkedIn, Twitter, and Facebook for which different solutions are proposed and implemented. Managing the voluminous and evolutionary real-world data demands the scalable data management system. Emerging distributed storage system like Hadoop, NoSQL and Cloud based infrastructure aids to reduce the cost for data storage. Researchers and practitioners (Beyer & Laney, 2012; Chen & Zhang, 2014; Wang, Kung, & Byrd, 2018) work dedicated to such huge volume of data shows constant growing interest. Similarly Open source software communities like Apache comes with opinion that due to massive increase in size of dataset it has been quiet difficult to acquire, store and analysed such large volume of data.
Social media (Zafarani, Abbasi, & Liu, 2014) is indeed a way to communicate virtually, in terms of opinions and sentiments of people that can be used by businesses and governments organisations to act accordingly. The process of collecting, integrating, storing and processing of Big social media data to gain information is highly tedious task which yet to be solved fully. In addition, such data needs pre-processing as it contains outliers and noisy data. Similarly, post, opinions or replies (Tang, Tan, & Liu, 2009) from various user on same or different topics have sentiments attached to it. Twitter is a microblogging sites which become one of the main platforms for capturing data to do further analysis. These analysis can be useful for finding out polarity in terms of positive or negative (Tang et al., 2009), detecting trends (Alsaedi, Burnap, & Rana, 2017; Lambrecht, Tucker, & Wiertz, 2018), community detections (Wen et al., 2017), recommendation of product and services (Abbas, Zhang, & Khan, 2015). This paper mainly focuses on stream data management and data analytics for finding similarity and sentiment analysis of user using tweets from Twitter data. Also, investigates and implement various imminent technologies for acquiring, storing and analysing Big social media data. The contributions of proposed work are highlighted below:
- •
Configured a highly reliable system i.e. Hadoop to store very large files in distributed environment. To ingest data as stream we configure Apache Flume to fetch Twitter data;
- •
To store Twitter data in structure format, we need to pre-process by storing data in Hive tables. On Hive table we implement a process to calculate sentiments of tweets using HiveQL based on AFFINE Dictionary;
- •
Proposed architecture for extracting information from large number of tweets to cluster similar user. For calculating Similarity between Tweets, we designed a MapReduce process for efficiency and fault-tolerance which uses text mining approach such as TF-IDF and cosine similarity measure to calculate values for similar tweets and users;
- •
Finally, results generates output clusters based on sentiments and similarity score.