IT and Business Can Succeed in BI by Embracing Agile Methodologies

Introduction

Business Intelligence (BI) is defined as “a set of methodologies, processes, architectures, and technologies that transform raw data into meaningful and useful information” (Evelson, 2008). Companies today generate and store vast amounts of raw data. What BI proposes is that any company that can effectively leverage this naturally-occurring business resource to measure performance, guide decisions, and implement strategies stands to create a sustainable competitive advantage out of these otherwise dormant stores of transactional, financial, and demographic records. Most companies have recognized the potential value of BI and have made various attempts to realize it; however, with only 21% of surveyed companies rating their BI deployment as “Very Successful” (Howson, 2009) it is clear that translating the vision into reality is much easier said than done.

What is reality for a twenty-first century company hoping to reap the benefits that BI presents? For many, budgets have been slashed, as a recent economic downturn and increased global market competition forces companies to “do more with less.” Schedules are being crashed, with executive-level pressures pushing on management to sustain a dozen “top priority” initiatives at once, and flavor-of-the-week issues result in frequent “student body left” engagement tactics. With this environment, it is no wonder that teams are overwhelmed with their workloads while customers remain underwhelmed with the results.

What can be done to succeed in such an environment? Business Intelligence lies at the intersection of applied technical skillsets and business function understanding, so achieving it cannot be the singular responsibility of either IT or Business; it will always require a highly-coordinated effort between the two. Exploring the why, how, who, and where of bringing IT and Business together, and examining a history of software development approaches, will ultimately reveal a path forward for BI success through leveraging a selection of methodologies from the well-known “Agile” movement (most notably, Scrum).

This paper begins with a study of four research questions, the answers to which will guide the end analysis. The research questions explored are the following: Why do IT and Lines of Business have to effectively cooperate for BI to succeed? How did the relationship between IT and Business become so strained? Whom should the company look for when forming the BI team? Where should the members of the BI team be located? These questions are each thoroughly addressed in independent sections.

Having reviewed the findings for these four BI organizational questions, a brief timeline leading up to Agile methodologies is presented. The solutions to the potentially disparate issues examined have common themes that move throughout. These themes ultimately come together in the form of a single, comprehensive approach to BI, found under the umbrella of Agile methodologies. After providing a proper exposition for the recommended approach, this paper concludes with a more detailed prescription of Agile applied to Data Warehousing and Business Intelligence (DWBI), reviewing probable benefits and possible pitfalls.

Agile is defined as “a group of software development methodologies based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams” (Wikipedia, 2010). It is argued herein that IT and Business together can succeed by embracing Agile methodologies. An attempt to argue that IT and Business will succeed would be naïve. There is no “silver bullet” prescription to rescue an untalented or uncommitted team. For every glowing success story of effectively applied Scrum, XP, or other “Agile” methodology, one can find several documented failures. Opponents of Agile – perhaps justifiably wearied of wading through the immensely broad yet shallow pool of buzz-word-laden attention that the movement has garnered – may be quick to point to these failures in blatant FUD (fear, uncertainty, and doubt) tactics at the very mention of the term. However, thoughtful practitioners should remain ever mindful of the Ted Sturgeon’s (1958) maxim: