Using Knowledge Management and Aggregation Techniques to Improve Web Effort Estimation

Introduction

Effort estimation, the process by which effort is forecasted and used as basis to predict costs and to allocate resources effectively, is one of the main pillars of sound project management, given that its accuracy can affect significantly whether projects will be delivered on time and within budget (Mendes et al., 2001). However, because it is a complex domain where corresponding decisions and predictions require reasoning with uncertainty, there are countless examples of companies that underestimate effort. Jørgensen and Grimstad (2009) reported that such estimation error can be of 30%-40% on average, thus leading to serious project management problems.

Similarly to software effort estimation, most research in Web effort estimation has to date focused on solving companies’ inaccurate effort predictions via investigating techniques that are used to build formal effort estimation models, in the hope that such formalization will improve the accuracy of estimates. They do so by assessing, and often also comparing, the prediction accuracy obtained from applying numerous statistical and artificial intelligence techniques to datasets of completed Web projects developed by industry, and sometimes also developed by students. A recent systematic literature review of Web resource estimation studies is given in (Azhar et al. 2012).

The variables characterizing such datasets are determined in different ways, such as via surveys (Mendes et al. 2005a), interviews with experts (Ruhe et al., 2003), expertise from companies (Ferrucci et al. 2008), a combination of research findings (Mendes et al. 2001), or even a researcher’s own consulting experience (Reifer, 2000). In all of these instances, once variables are defined, a data gathering exercise takes place, obtaining data (ideally) from industrial projects volunteered by companies. Except when using research findings to inform variables’ identification, invariably the mechanism employed to determine variables relies on experts’ recalling, where the subjective measure of an expert’s certainty is often their amount of experience estimating effort.

However, in addition to eliciting the important effort predictors (and optionally also their relationships), such mechanism does not provide the means to also quantify the uncertainty associated with these relationships and to validate the knowledge obtained. Why should these be important?

Our experience developing and validating several single-company expert-based Web effort prediction models that use a knowledge management methodology to incorporate the uncertainty inherent in this domain (Mendes, 2012a) showed that the use of a structured iterative process in which factors and relationships are identified, quantified and validated (Mendes, 2011a; Mendes 2011b; Mendes et al., 2009) leads the participating companies to a much more thorough and deep understanding of their mental processes and their decisions when estimating effort, when compared to just the recalling of factors and their relationships. The iterative process we use employs Bayesian inference, which is one of the techniques employed in root cause analysis (Ammerman, 1998); therefore, it aims at a detailed analysis and understanding of a particular phenomenon of interest.

In all the case studies we conducted, the original set of factors and relationships initially elicited was always modified as the model evolved; this occurred as a result of applying a root cause analysis approach comprising a Bayesian inference mechanism and feedback into the analysis process via a model validation. In addition, post-mortem interviews with the participating companies showed that the understanding companies gained by being actively engaged in building those models led to both improved estimates and estimation processes (Mendes, 2012a, 2011a 2011b; Mendes et al., 2009).

We therefore contend that the aggregation of the diverse factors and relationships from these models into a single knowledge map brings several advantages over a simple compilation of factors from different lists provided by several companies, or the non-aggregation of factors and relationships, as follows: