Web Mining for Strategic Competitive Intelligence: South African Experiences and a Practical Methodology

Web Mining for Strategic Competitive Intelligence: South African Experiences and a Practical Methodology

Lynnda Wagner (University of Cape Town, South Africa) and Jean-Paul Van Belle (University of Cape Town, South Africa)
DOI: 10.4018/978-1-61692-886-5.ch001
OnDemand PDF Download:


This chapter explains the use of web mining for gathering strategic competitive intelligence. It first discusses some concepts relating to web mining. This is followed by the findings of an exploratory study on how web mining for strategic intelligence is actually used by competitive intelligence professionals in some South African organizations. The findings are discussed under the three major themes, namely general intelligence practices; Web-based intelligence; and skills development and education. The bulk of the chapter proposes a practical methodology for using web mining to gather strategic business intelligence. Some practical comments about the impact of particular circumstances of emerging African countries on this methodology are also given.
Chapter Preview


In today’s rapidly changing global business environment, it is important for companies to conduct systematic local and global environmental scans to equip executive management with the necessary knowledge to plan, lead, organize, and control their business. Ideally, organizations employ mechanisms for efficient strategic intelligence collection to empower management to forecast and influence the companies’ future direction. This is especially important for smaller businesses in emerging countries, which increasingly have to compete globally but are faced with infrastructural limitations such as high bandwidth costs and more difficult access to information about global technology and market trends (Viviers, Saayman, Muller & Calof, 2002).

Although the Web can be an excellent source of current, insightful, and easily accessible information, the useful, relevant and high-quality information is often hard to find systematically in the ocean of lower quality information. Mining the Web for nuggets of intelligence may give an organization competitive advantage, and should form an integral part of the strategic planning process. Companies now have the opportunity to devise a web mining system to rapidly assess its external environment. This will enable companies to be aware of trends and/or changes outside of their operations that can negatively or positively impact their business. This empowers businesses to become proactive towards emerging market issues. Also, it allows management to detect opportunities and threats, and evaluate how to adapt their operations accordingly. Perhaps most of all, it facilitates decision-making with regards to the future direction of the company. Overall, the information gained can be used to formulate corporate strategies.

Not surprisingly, the Internet has become one of the top sources of intelligence for intelligence gathering professionals. For example, the Internet Intelligence Index (I3 available from www.fuld.com), is an index of about 600 intelligence-related websites divided into three areas: 1) general business Internet resources; 2) industry-specific Internet resources; and 3) international Internet resources (Kennedy, 1998).

Unfortunately, the amount of information on the Web is staggering and growing fast. As such, it is not practical to cover the full extent of the Web manually. Although search engines are powerful tools, they are still incapable of providing tailored information, which means that users must still sift through copious search results before finding something within the right scope. Technologies are being developed to enable online data analysis with the ultimate goal being to gain insight from online data. Thus, Web mining has emerged as an active area of research and development.

Web mining provides extremely useful and automated tools and techniques to gather the information required for competitive intelligence, business intelligence and for the strategic information processes which can be readily incorporated as part of any knowledge management process. Thus any knowledge management strategy should carefully consider competitive analysis and web mining as an integral part of the knowledge management process (Liebowitz, 2006). Many commercially available tools already exist (see Table 1) and most offer easy integration points into larger knowledge management systems.

Table 1.
Mapping of some commercially available web mining tools on the proposed process steps
ToolTaxonomyMeta-searcherWeb CrawlingWeb ClusteringInformation ExtractionText Miner
Temis Luxid···
Convera NewOwl··
SAS TextMiner··
RapidMiner (open source)····

Complete Chapter List

Search this Book: