Article Preview
Top1. Introduction
Social media brings people together so that they can generate ideas or share their experiences with each other. The information generated through such sites can be utilized in many ways to discover fruitful patterns. But, accumulation of data via such sources create a huge unstructured textual data with numerous unwanted formats. Henceforth, the first step of text mining involves pre-processing of gathered reviews.
The journey of transforming dataset into a form, an algorithm may digest, takes a complicated road. The task embraces four differentiable phases: Cleaning, Annotation, Normalization and Analysis. The step of cleaning comprehends extrication of worthless text, tackling with capitalization and other similar details. Stop words, Punctuations marks, URLs, numbers are some of the instances which can be discarded at this phase. Annotation is a step of applying some scheme over text. In context to natural language processing, this includes part-of-speech tagging. Normalization demonstrates reduction of linguistic. In other words, it is a process that maps terms to a scheme. Basically, standardization of text through lemmatization and stemming are the part of normalization. Finally, text undergoes manipulation, generalization and statistical probing to interpret features.
For this study, pre-processing is accomplished in three major steps, as signified in Figure 1, keeping process of sentiment analysis in consideration. Foremost step included collection of tweets from Twitter by means of Twitter API. Captured data is then stored in a NoSQL database known to be MongoDB. Thereafter, collected tweets underwent cleaning (Zainol et al., 2018) process. Cleaning phase incorporated removal of user name, URLs, numbers, punctuations, special characters along in addition to lower casing and emoji decoding. The first two phases of data collection and clean ing were demonstrated in previous research. Also, it was shown that application of cleaning process still left data with anomalies and that is why the endmost stage of data transformation is introduced in this research. Data transformation comprise of tokenization (Mullen et al., 2018), stop word removal (Effrosynidis et al., 2017), part-of-speech tagging (Belinkov et al., 2018) and lemmatization (Liu et al., 2012).
The remaining paper is organized as follows: Section 2 includes discussion of various author’s work in concerned arena. Further, entire methodology for preprocessing of data opted for this research is postulated in Section 4. Then, the results generated through implementation of algorithms mentioned in Section 4 are scrutinized utterly in Section 5. Thereafter, Section 6 provides conclusion of entire work.
TopMany studies centered around the issue of preprocessing for text mining are scrutinized in this section.
Figure 2.
Errors left in cleaned data