Research has shown that IT project failure rates increase in line with project size, and by the time you reach projects over $10M, the failure rate hits 100% (The Standish Group International, Inc., 1999). Hughes and Stodder (2012) go on to show that at an average cost of $12.8M, large data warehousing projects were failing 65% of the time for internally built apps and 86% of the time for purchased packages. The outtake here is that small projects are by far more successful. “We have long been convinced that shorter time frames, with delivery of software components early and often, increase the success rate” (The Standish Group International, Inc., 1999, p. 3). How can this be achieved on large projects? It is possible to be successful at the enterprise level, but it requires a fresh approach to project execution, an ability to deliver projects in small, incremental chunks while maintaining business involvement and support, and setting clear, achievable objectives throughout the program.
TopIntroduction
While the agile approach has been around for many years - indeed, 2011 was the ten year anniversary of the agile manifesto (http://agilemanifesto.org/) - the adaptation to a data-driven application is relatively recent. Rather than the traditional use case or general systems integration, agile data warehousing became topical in August 2008 when Ralph Hughes, inventor of the concept, published his first book, “Agile Data Warehousing: Delivering World-Class Business Intelligence Systems Using Scrum and XP”.
Whatever term you use - information management (IM), data warehousing (DW) or business intelligence (BI) - an agile approach is able to successfully connect the business and the outcomes sought, to the development teams doing the actual work. It’s about synthesising the ‘contract’ which details the business requirements between the two areas.
The importance of this ‘contract’ is heightened in IM, DW and BI as it’s all about the data, not the technology being used or the process being created. The aim is to uncover those truly valuable nuggets of information hidden in masses of data within an organisation. Even when you successfully combine the right technical tools and experience with business knowledge and desire, you still need a bit of luck to uncover data gold. No surprise that IM is a field littered with implementation failures, budget overruns and timeline blowouts.
I believe an agile approach to IM is more successful because it forces the business and IT to connect regularly throughout the program – something many enterprises pay lip service to, but rarely achieve.
An agile IM approach is about making the complex, simple; stripping back to bare minimum with a singular goal in mind. The goal is what’s important not the documentation and bureaucracy. Agile IM by its pure nature is a highly adaptive approach; it’s light on process and heavy on outcome.
If you try to overlay heavy bureaucracy, common in enterprise size organisations, you confuse the project and the business outcomes you seek. Overlay that with the myriad of individuals who have competing agendas and potentially divisional or departmental conflicts of interest, and you have exponentially complicated your project and your outcomes. The fundamental problem is that when an enterprise wants to scale, it imposes more process and governance on the activity which results in non-agile activity such as service level agreements between organisational business units.
Everything gets heavier, more complex and goes against the core philosophy of the agile manifesto thereby removing its agility. Enterprises need to scale without the bureaucratic weight of process and governance.
Don’t get me wrong, process and governance are critical in an agile IM methodology; but they don’t rule the program; they are elements within the program.
Agile is simple in its construct and definitely achievable at enterprise level but it is by no means easy to implement and the larger the organisation, the harder it is to ‘go agile’. Ultimately, if the CEO (private sector) or the Secretary (public sector) wants to constantly deliver value to their customers/citizens, eliminate wasted effort and significantly increase accountability, they need to adopt an agile approach to IM and be able to scale it accordingly.
In this chapter, we will take you through how an agile information management delivery approach can be used to successfully deliver projects within the enterprise.