Web Mining for Strategic Competitive Intelligence: South African Experiences and a Practical Methodology

Introduction

In today’s rapidly changing global business environment, it is important for companies to conduct systematic local and global environmental scans to equip executive management with the necessary knowledge to plan, lead, organize, and control their business. Ideally, organizations employ mechanisms for efficient strategic intelligence collection to empower management to forecast and influence the companies’ future direction. This is especially important for smaller businesses in emerging countries, which increasingly have to compete globally but are faced with infrastructural limitations such as high bandwidth costs and more difficult access to information about global technology and market trends (Viviers, Saayman, Muller & Calof, 2002).

Although the Web can be an excellent source of current, insightful, and easily accessible information, the useful, relevant and high-quality information is often hard to find systematically in the ocean of lower quality information. Mining the Web for nuggets of intelligence may give an organization competitive advantage, and should form an integral part of the strategic planning process. Companies now have the opportunity to devise a web mining system to rapidly assess its external environment. This will enable companies to be aware of trends and/or changes outside of their operations that can negatively or positively impact their business. This empowers businesses to become proactive towards emerging market issues. Also, it allows management to detect opportunities and threats, and evaluate how to adapt their operations accordingly. Perhaps most of all, it facilitates decision-making with regards to the future direction of the company. Overall, the information gained can be used to formulate corporate strategies.

Not surprisingly, the Internet has become one of the top sources of intelligence for intelligence gathering professionals. For example, the Internet Intelligence Index (I3 available from www.fuld.com), is an index of about 600 intelligence-related websites divided into three areas: 1) general business Internet resources; 2) industry-specific Internet resources; and 3) international Internet resources (Kennedy, 1998).

Unfortunately, the amount of information on the Web is staggering and growing fast. As such, it is not practical to cover the full extent of the Web manually. Although search engines are powerful tools, they are still incapable of providing tailored information, which means that users must still sift through copious search results before finding something within the right scope. Technologies are being developed to enable online data analysis with the ultimate goal being to gain insight from online data. Thus, Web mining has emerged as an active area of research and development.

Web mining provides extremely useful and automated tools and techniques to gather the information required for competitive intelligence, business intelligence and for the strategic information processes which can be readily incorporated as part of any knowledge management process. Thus any knowledge management strategy should carefully consider competitive analysis and web mining as an integral part of the knowledge management process (Liebowitz, 2006). Many commercially available tools already exist (see Table 1) and most offer easy integration points into larger knowledge management systems.