This paper examines the Data Envelopment Analysis (DEA) methodology from a cognitive perspective. Specifically, it analyzes (a) the role of DEA scores as an overall efficiency measure and (b) to what extent the presence of DEA scores for a non-financial performance appraisal influences a posterior financial performance assessment. The study confirms that the efficiency score acts as a strong performance marker when deciding on which decision making units (DMUs) should be awarded for their non-financial performance. Furthermore, it shows that the results of the non-financial performance evaluation may act as an anchor which significantly influences a posterior financial assessment. These insights have practical consequences for planning, reporting, and controlling processes that incorporate DEA efficiency scores.
Top1. Introduction
While performance measurement provides data about the performance of decision making units (DMUs), performance assessment can be described as the subsequent step of performance data interpretation. Both tasks, summarized here under the term of performance evaluation, are core elements of management control and aim at providing valuable information for decision facilitating and decision influencing (Sprinkle & Williamson, 2007). Performance evaluation has traditionally been based on accounting measures as financial performance criteria. As this can tempt managers to focus only on short-term monetary results, non-financial criteria have become more important (Ittner, Larcker, & Randall, 2003b).
The additional consideration of non-financial performance criteria offers a series of benefits to the decision maker, but it also poses significant challenges (Luft, 2009). It has been shown that more comprehensive measurement systems reinforce managers’ existing points of view (Hall, 2010) and bear the risk of failure due to their complexity (Ittner & Larcker, 1998). Other issues that have been mentioned include data availability (Neely, Gregory, & Platts, 2005), criteria weights (Krishnan, Luft, & Shields, 2005), and the usefulness of aggregated data (Hall, 2010). The last two aspects are closely connected, especially because subjective criteria weighting can cause undesired behavior.
To meet some of these challenges, different instruments have been developed. One of these instruments is the Data Envelopment Analysis (DEA), which calculates overall (relative) efficiency scores for DMUs based on financial and non-financial performance criteria. Numerous articles dealing with the theory of DEA and its application have been published. Nevertheless, DEA remains unknown to most practitioners. This may be due to methodological aspects or to cognitive difficulties experienced by the decision maker when interpreting the results. In this study, we specifically analyze the use of DEA scores for aggregating non-financial measures and its influence on a posterior assessment of financial performance.
Drawing upon previous studies on heuristics and biases and the use of financial and non-financial performance criteria, we test four exploratory hypotheses. The results show that the DEA score used for aggregating the relative non-financial performance acts as a strong performance marker, therefore affecting the choice among several good performing DMUs. Decision makers reduce their cognitive effort by concentrating their attention on the DEA score, possibly neglecting the rest of the available information. This effect was to be expected and seems to be fairly unproblematic from the perspective of (control-oriented) decision facilitating. However, when considering its decision influencing role in planning processes, an important pitfall of DEA comes to light: awards based on the DEA score are likely to cause the usually undesirable incentive to concentrate management efforts on a few performance criteria.
The second main finding of our experimental study shows that the financial performance assessment is influenced by the results of a prior non-financial performance assessment, which in turn is affected by the presence of DEA scores. This effect is of major practical relevance and shows the importance of a thorough analysis regarding the incorporation of DEA scores into the reporting and controlling process.
The rest of the paper is organized as follows: Section 2 introduces the original DEA model used in the experiment and provides a brief overview of the heuristics and biases associated to multiple attribute choice and their manifestations. On this basis, the hypotheses of the study are developed. Section 3 describes the methodological aspects of the experiments conducted to test the hypotheses. Section 4 presents the results of the experiments. Section 5 discusses the findings and outlines further research questions.