Designing Sound Security Metrics

Introduction

This work extends Yee (2019) by including a section on the feasibility of obtaining sound security metrics based on science, and adding clarifications and references.

Today’s world is precarious, in which barely a day goes by without headlines appearing about the latest data breaches, the most recent webs sites brought down by DDoS (Distributed Denial of Service) attacks, or the latest victim held hostage by a ransomware attack. In response, organizations have poured large sums of money into various security countermeasures (e.g. firewalls, biometrics, encryption, security training) and re-structured systems and workflows in attempts to resist these attacks. However, the return on these efforts or the resultant increase in the level of security has been largely unknown. The organization may then have to deal with the following dilemmas:

• •

  Has the organization invested enough funds in securing its systems to be “safe” from attack?

• •

  Will our software changes to improve security be effective?

• •

  Is our workflows and processes sufficiently secure?

• •

  How will our security be impacted by the addition of that third-party software component?

• •

  How can the organization be accountable for keeping its systems at the required level of security if it is unable to determine its level of security?

These dilemmas can be resolved by having a way to assess the level of security of a computer system or an organization. In other words, properly defined, effective security metrics can help. Security metrics have been defined and are being used. However, many of them are poorly defined for their intended purposes. They are far from giving the results needed to assess security and are therefore ineffective. For example, a traditional security metric is the number of viruses detected and eliminated at a firewall. This metric has been used to gauge the effectiveness of a firewall at filtering out viruses, which impacts the organization’s level of security. Unfortunately, this metric fails its mission because it says nothing about the viruses that were not detected and got through. If 50 viruses were detected and eliminated but 100 got through, basing the firewall’s effectiveness on the 50 viruses that were detected would falsely inflate the effectiveness and the level of security.

The security metric mentioned in the previous paragraph is neither meaningful nor effective. This leads to the following questions: Can one define security metrics that are meaningful and effective? What are the conditions that security metrics must satisfy in order to be considered sound metrics? The author does not claim that all existing security metrics are bad. Rather, he wishes to shed some light into how to define sound security metrics.

After showing how to design sound security metrics, the author tackles the question of scientifically-based security metrics. Are such metrics feasible and sound? A scientifically-based security metric is one that is based on science. For example, a computer system’s performance metrics such as throughput and service time are based on physics, and are therefore scientifically based.

The objectives of this paper are to a) introduce the reader to the problems and challenges exemplified by several typical traditional security metrics, b) propose a method that can be used to design a sound security metric or to test whether a particular security metric passes in terms of the characteristics that a sound security metric must have, c) discuss the feasibility of scientifically-based security metrics and whether or not such metrics are sound, and d) illustrate the above method by applying it to design a sound security metric, and to test existing security metrics to see if they are sound.