A Systematic Review of Web Accessibility Metrics

Introduction

Currently web sites have become a main dissemination mechanism of information and services. People do most of their daily transactions through the web sites. On the other hand, 15% of world’s population is estimated to be disabled by World Health Organization (Chan & Zoellick, 2011). In addition, the aging of the population would affect to increase the numbers because elderly people begin to have problems in vision or their cognitive skills. Therefore, the audiences of websites are various people with different abilities or disabilities (Masri & Luján-Mora, 2011). Thus, providing inclusive and accessible websites for all people is necessary.

Since web sites have become indispensable, some countries make use of general guidelines such as WCAG (Web Content Accessibility Guidelines) 1.0 (W3.org, 1999) or 2.0 (W3.org, 2008) or some rules or regulations such as ISO 9241-20 (ISO, 2008) or Section 508 (US Access Board, 2000) to make the web contents more accessible. Recently, WCAG 3.0 (W3C, 2021) was publicized as a working draft but WCAG 2.0 [3] is still used as a general guideline for accessibility evaluations by developers, designers, etc. to provide a certain level of accessibility for all users.

There are various evaluation methods to assure web accessibility in the literature and these methods can be categorized as qualitative and quantitative methods in general (Masri & Luján-Mora, 2011). Qualitative methods are used to investigate a website’s compliance to guidelines by the use of automatic tools (Zaphiris & Ellis, 2001) or to some heuristics by experts manually (Brajnik, 2011). Other than these two analytical methods, user testing conducted with people with disabilities is an empirical method which reveals accessibility problems more accurately (Masri & Luján-Mora, 2011). Many recent research (Akgül, 2021; Alajarmeh, 2021; Baeza-Yates, 2020) also focuses on website accessibility evaluations as well. On the other hand, quantitative methods show the level of accessibility of the website by the use of accessibility metrics which have been proposed in the literature (Freire et al., 2008). These metric-based methods are important to help understand products, processes, and control and improve them in software development (Fenton & Bieman, 2014). In terms of accessibility, metrics are useful for increasing accessibility of existing products and identifying accessibility problems in a product in the development process. Various studies (Sullivan & Matson, 2000) have been carried out on metric definition and application of these metrics in measuring web accessibility .

There are several studies (Hackett et al., 2003; Sirithumgul et al., 2009a; Vigo et al., 2007) that propose WAMs (Web Accessibility Metrics) and these are very valuable for developers and researchers studying in this domain. There are some reviews on WAMs in the literature. Freire et al. (2008) and Brajnik and Vigo (2011) report WAMs which were proposed until 2011. Vigo et al. (2007) also offers a comprehensive list of metrics in their work, where they proposed a framework for comparing WAMs. In a recent study, Brajnik and Vigo (2019) updated the information on accessibility metrics proposed until 2018.

Above mentioned review studies were generally conducted as traditional reviews. Alternatively, within the scope of this study, it is aimed to identify current trends and analyze WAMs via a systematic literature review (SLR) (Kitchenham & Charters, 2007). This study is implemented as a secondary study based on evidence-based software engineering (Kitchenham et al., 2004) and involves gathering the primary studies especially on WAMs. The scope of the SLR is determined to be started from 2008 which is the year of publication of WCAG 2.0. The aim is to provide a complete picture of WAMs by revealing the trends in WAMs, characteristics of these metrics, how they were developed and validated. The results are believed to provide insight to the researchers who are studying or would like to study on WAMs and to evaluators who would like to use these metrics for their assessments.