Assessing the Usefulness of Testing for Validating and Correcting Security Risk Models Based on Two Industrial Case Studies

1. Introduction

Security risk analysis is carried out in order to identify and assess security specific risks. Traditional risk analyses often rely on expert judgment for the identification of risks, their causes, as well as risk estimation in terms of likelihood and consequence. The outcome of these kinds of risk analyses is therefore dependent on the background, experience, and knowledge of the participants, which in turn reflects uncertainty regarding the validity of the results.

In order to mitigate this uncertainty, security risk analysis can be complemented by other ways of gathering information of relevance. One such approach is to combine security risk analysis with security testing, in which the testing is used to validate and correct the risk analysis results. We refer to this as test-driven security risk analysis.

We have developed an approach to test-driven security risk analysis, and as depicted in Figure 1, our approach is divided into three phases. Phase 1 expects a description of the target of evaluation. Then, based on this description, the security risk assessment is planned and carried out. The output of Phase 1 is security risk models, which is used as input to Phase 2. In Phase 2, security tests are identified based on the risk models and executed. The output of Phase 2 is security test results, which is used as input to the third and final phase. In the third phase, the risk models are validated and corrected with respect to the security test results.

Figure 1.

Overview of the test-driven security risk analysis approach

In this paper, we present an evaluation of our test-driven security risk analysis approach based on two industrial case studies. The objective of the case studies was to assess how useful testing is for validating and correcting security risk models. The basis of our evaluation is to compare the risk models produced before and after testing. That is, we compare the difference in risk models produced in Phase 1 with the updated risk models produced in Phase 3.

The first case study was carried out between March 2011 and July 2011, while the second case study was carried out between June 2012 and January 2013. In the first case study we analyzed a multilingual financial Web application, and in the second case study we analyzed a mobile financial application. The systems analyzed in both case studies serve as the backbone for the system owner's business goals and are used by a large number of users every day. The system owners, which are also the customers that commissioned the case studies, required full confidentiality. The results that are presented in this paper are therefore limited to the experiences from applying the test-driven security risk analysis approach.

The reminder of the paper is structured as follows. Section 2 describes our test-driven security risk analysis approach. Section 3 describes our research method. Section 4 gives an overview of the two case studies. Section 5 describes the results obtained in the case studies which are the basis of our evaluation. Section 6 provides a discussion of the results with respect to our research questions and overall hypothesis. Section 7 discusses related work. Finally, Section 8 highlights our key findings and concludes the paper.

2. Test-Driven Security Risk Analysis

As already illustrated in Figure 1, the test-driven security risk analysis process is divided into three phases. Each phase is further divided into various steps. Figure 2 shows the steps within each phase. In the following, we give a more detailed description of each phase.

Figure 2.

The steps in test-driven security risk analysis