A Database of Existing Vulnerabilities to Enable Controlled Testing Studies

A Database of Existing Vulnerabilities to Enable Controlled Testing Studies

Sofia Rei, Rui Abreu
Copyright: © 2017 |Pages: 23
DOI: 10.4018/IJSSE.2017070101
(Individual Articles)
No Current Special Offers


From holding worldwide companies' information hostage to keeping several distributed systems down for hours, the last years were marked by several security attacks which are the result of complex software and its fast production. There are already tools which can be used to help companies detect vulnerabilities responsible for such attacks. However, their reliability is still not the best and well discriminated. In software testing, researchers tend to use hand-seeded test cases or mutations due to the challenges involved in the extraction or reproduction of real test cases which might not be suitable for testing techniques, since both approaches can create samples that inadvertently differ from the real vulnerabilities and thus might lead to misleading assessments of the tools' capabilities. The lack of databases of real security vulnerabilities is an issue since it hampers the tools' evaluation and categorization. To study these tools, the researchers created a database of 682 real test cases which is the outcome of mining 248 repositories for 16 different vulnerability patterns.
Article Preview

1. Introduction

Under the rush of companies trying to outdo each other, fast updates and software supporting new features in every new release, testing tends to be one of the most forgotten phases of the software development life cycle (Figure 1). Not only due to the lack of reliable automated tools to test software but also due to the high costs associated with the manual process. Some companies do not even pay attention to the process of finding security vulnerabilities before shipping software and the ones that do care, typically have two choices:

  • They try to tackle the issue hiring testers and the best developers they can; perform extensive manual code reviews and outsource to find bugs (bug bounty programs)

  • Or, they ship the product based on a possible “well-thought-out” balance between the damages of a vulnerable version and the fact that the vulnerable code will never be disclosed

Companies tend to embrace the second one since the first option is too time-consuming and monetarily unbearable. Although dangerous, corporations are used to escape attacks with the second approach, but now, and more than ever, it rarely happens, and the proof is the increase of security vulnerabilities reported by annual security reports (IBM Security Department USA, 2017; European Union Agency for Network and Information Security, 2017).

Figure 1.

Software development life cycle (SDLC)


According to IBM's X-Force Threat Intelligence 2017 Report (IBM Security Department USA, 2017), the number of vulnerabilities per year has been significantly increasing over the past six years. IBM's database counts with more than 10K vulnerabilities in 2016 alone. The most common ones are cross-site scripting and SQL injection vulnerabilities - these are two of the main classes that incorporate the Open Web Application Security Project (OWASP)'s 2017 Top-10 security risks (The OWASP Foundation, 2017). The past years have been flooded by news from the cybersecurity world: exposure of large amounts of sensitive data (e.g., 17M of zomato accounts stolen in 2015 which were put up for sale on a dark web marketplace only now in 2017), phishing attacks (e.g., Google Docs in 2017), denial-of-service attacks such as the one experienced last year by Twitter, The Guardian, Netflix, CNN and many other companies around the world; or, the one that possibly stamped the year, the ransomware attack which is still very fresh and kept hostage many companies, industries and hospitals information. All of these attacks were able to succeed due to the presence of security vulnerabilities in the software that were not tackled before someone exploit them. Another interesting point reported by IBM is the large number of undisclosed attacks, i.e., attacks through exploits that do not yet belong to a specific attack pattern or cannot be remediated by a software patch - the so-called zero-day exploits (IBM Redbooks, 2011) - which can be harmful since developers have been struggling already with the disclosed ones.

The identification and correction of defects take more than 50% of the software development costs (Tassey, 2002) which is extremely high given all the phases involved in the SDLC (Figure 1). Several static analysis tools (e.g., Infer, Find Security Bugs, Symbolic PathFinder, WAP, Brakeman, Dawnscanner and more) can detect security vulnerabilities through a source code scan which may help to reduce the time spent on those two activities. Unfortunately, their detection capability is not the best yet (i.e., the number of false-negatives and false-positives is still high) and sometimes is even comparable to random guessing (Goseva-Popstojanova & Perhinschi, 2015).

Complete Article List

Search this Journal:
Open Access Articles: Forthcoming
Volume 8: 4 Issues (2017)
Volume 7: 4 Issues (2016)
Volume 6: 4 Issues (2015)
Volume 5: 4 Issues (2014)
Volume 4: 4 Issues (2013)
Volume 3: 4 Issues (2012)
Volume 2: 4 Issues (2011)
Volume 1: 4 Issues (2010)
View Complete Journal Contents Listing