An Evaluation of a Test-Driven Security Risk Analysis Approach Based on Two Industrial Case Studies

Introduction

Security risk analysis is carried out in order to identify and assess security specific risks. Traditional risk analyses often rely on expert judgment for the identification of risks, their causes, as well as risk estimation in terms of likelihood and consequence. The outcome of these kinds of risk analyses is therefore dependent on the background, experience, and knowledge of the participants, which in turn reflects uncertainty regarding the validity of the results.

In order to mitigate this uncertainty, security risk analysis can be complemented by other ways of gathering information of relevance. One such approach is to combine security risk analysis with security testing, in which the testing is used to validate and correct the risk analysis results. This is referred to as test-driven security risk analysis.

The authors have developed an approach to test-driven security risk analysis, and as depicted in Figure 1, the approach is divided into three phases. Phase 1 expects a description of the target of evaluation. Then, based on this description, the security risk assessment is planned and carried out. The output of Phase 1 is security risk models, which is used as input to Phase 2. In Phase 2, security tests are identified based on the risk models and executed. The output of Phase 2 is security test results, which is used as input to the third and final phase. In the third phase, the risk models are validated and corrected with respect to the security test results.

Figure 1.

Overview of the test-driven security risk analysis approach

Source: Authors' work

Although strongly related, it is important to note that test-driven risk analysis is different from the more common combination of risk analysis and testing, which is referred to as risk-driven (or risk-based) testing. The purpose of risk-driven testing is to makes use of risk assessment within the testing process to support risk-driven test planning, risk-driven test design and implementation, and risk-driven test reporting. Großmann and Seehusen (2015) provide a detailed explanation of these two approaches by combining the well-known and widely used standards ISO 31000 (ISO, 2009) and ISO/IEC/IEEE 29119 (ISO, 2013a), with a focus on security.

In this chapter, the authors present an evaluation of the test-driven security risk analysis approach based on two industrial case studies. The objective of the case studies was to assess how useful testing is for validating and correcting security risk models. The basis of the evaluation is to compare the risk models produced before and after testing. That is, the authors compare the difference in risk models produced in Phase 1 with the updated risk models produced in Phase 3.

The first case study was carried out between March 2011 and July 2011, while the second case study was carried out between June 2012 and January 2013. In the first case study the authors analyzed a multilingual financial Web application, and in the second case study the authors analyzed a mobile financial application. The systems analyzed in both case studies serve as the backbone for the system owner's business goals and are used by many users every day. The system owners, which are also the customers that commissioned the case studies, required full confidentiality. The results presented in this chapter are therefore limited to the experiences from applying the test-driven security risk analysis approach.

The reminder of the chapter is structured as follows. First, the chapter describes the background concepts relevant for this chapter and then describes the test-driven security risk analysis approach, followed by a description of the research method. Then, the chapter gives an overview of the two case studies as well as the results obtained which are the basis of the evaluation. Then, the chapter provides a discussion of the results with respect to three research questions and an overall hypothesis defined as part of the research method. Finally, the chapter discusses related work and highlights the key findings before providing the conclusion.

Background

This section provides the main background concepts in security risk assessment, security testing, and their combination. The key terms and definitions are summarized at the end of this chapter.