Mining Big Data for Marketing Intelligence

Data Collection And Cleaning

Taking an example of twitter, following section shows how data collection and cleaning can be done. Once the communities on the twitter have been identified, user data and tweet information can be collected by the twitter APIs like rest or stream. Every user has unique id through which his profile details can be extracted. The usage pattern of a twitter user can be either personal, group, aggregator, satire or marketing related (Cheong & Lee, 2009). Users can also be categorized as tourist, minglers, devotees and insiders (Kozinets, 2002). Tweet search is generally based on time period or is a keyword search (Twitter Developer). Data about location, user mentions, friends, and followers can also be extracted with the help of these API. Example for pattern detection, retweet, replied and trend keyword can be used (Cheong & Lee, 2009). Re-tweet are special posts where one twitter user forwards the tweet of another user, this kind of tweets start with RT. Similarly, reply to a tweet can be detected as a reply starts with @username where username is the twitter user whom another user is replying too. Hashtags are another feature which starts with #keyword such that it creates an index for the keyword. This user information tweets and metadata can be stored in a relational database for further analysis. Further, not all the extracted tweets might to useful or related to the marketing research question. Classifying tweets as primarily on topic or off topic or primarily social or informational will help to get tweets related to the scope of the question (Kozinets, 2002).