Object Architected Design and Efficient Dynamic Adjustment Mechanism of Distributed Web Crawlers

2. Related Work

According to the survey of eMarketer at 2013, the number of global social media users has reached 1.73 billion, which covers the 67.7% of the population of internet users. Among them the monthly active users of Facebook are 1.2 billion, the ones of Twitter are over 0.55 billion, the ones of Weibo and Tencent originated in China are over 0.4 billion (IDEAS, III, 2013). Currently the social media users are mostly young people. There are over 90% of internet users aging from 18 to 29 use social networking sites, but the penetration rate of those over 65 years old is only 40%. However, the penetration rate of senior citizens on social networking sites is increasing rapidly.

According to the statistics of III (2013), domestic enterprises have only 15% structured data, yet the other 85% exists as an unstructured data, such as email, web links, logs/journals of information systems. On the other hand, according to the research of IBM (2013), after the internet becomes universal, the data produced on the internet every two days at present is equal to the sum of data from the year of invention of computer to 2003. Under the rapid and massive increase of digital data, formal information systems/platforms are unable to stand. This causes the big data to become the widely concerned issue.