Usability Evaluation Meets Design: The Case of bisco Office™

Usability Evaluation Meets Design: The Case of bisco Office™

Judith Symonds (AUT University, New Zealand)
DOI: 10.4018/978-1-60566-040-0.ch010
OnDemand PDF Download:


Usability Evaluation Methods (UEM) are plentiful in the literature. However, there appears to be a new interest in usability testing from the viewpoint of the industry practitioner and renewed effort to reflect usability design principles throughout the software development process. In this chapter we examine one such example of usability testing from the viewpoint of the industry practitioner and reflect upon how usability evaluation methods are perceived by the software developers of a content driven system and discuss some benefits that can be derived from bringing together usability theory and usability evaluation methods protocols used by practitioners. In particular, we use the simulated prototyping method and the “Talk Aloud” protocol to assist a small software development company to undertake usability testing. We propose some issues that arise from usability testing from the perspectives of the researchers and practitioners and discuss our understanding of the knowledge transfer that occurs between the two.
Chapter Preview

Usability Evaluation Methods

Usability is a multidisciplinary field that falls under the larger umbrella of Human-Computer Interaction (HCI) where there are essentially two camps: design and evaluation (Wania, Atwood, & McCain, 2006). In their bibliographic citation analysis of the HCI literature, Wania et al. (2006) suggest that the two camps can learn from each other implying that they are separate and noninclusive. But, in industry, are usability design and evaluation really so far apart? Even relatively early research (Sullivan, 1989) called for usability to be considered throughout the entire software design process and suggested that considering only usability evaluation or testing is a “narrow conception” of usability. More recently, a study of Dutch IT companies found that in the software industry and the IT industry in general, there is variance in whether design is a consideration throughout the development process. In the study it was found that the usability of the system is often only addressed in the latter stages of the process (Gemser, Jacobs, & Ten Cate, 2006). The study also showed that developers of content-driven systems were more likely to consider usability throughout the whole design process. The authors suggest that this is related to the amount of influence customers have over the design process.

Hartson, Andre, and Williges (2003) conducted a thorough review of User Evaluation Methods (UEMs) for usability and even went so far as to establish a criteria for selection of a most suited UEM and provide guidance on the number of tests needed to ensure the reliability of the process. Carter (2007) advocates the simplicity of usability testing and suggests that the academic community have become too wrapped up in the protocols of UEMs and have lost sight of the usefulness of usability testing. The expectations of the practitioner and the usability expert differ. Certainly, away from the theory, there is an interest in how usability testing can be undertaken in the field (Waterson, Landay, & Matthews, 2002).

The three most widely used, robust, easy-to-use UEMs are cognitive walkthrough (CW), heuristic evaluation (HE), and thinking aloud (TA) (Hertzum & Jacobsen, 2003). The thinking (or talking) aloud (TA) verbal protocol analysis has been widely used, for example, in health care Web site design (Zimmerman, Akerelrea, Buller, Hau, & Leblanc, 2003) and in software development (Hohmann, 2003). Roberts and Fels (2006) extend the think aloud protocol to include procedures to follow when the participant is deaf. Krahmer and Ummelen (2004) in their comparison of the think aloud protocols developed by Ericsson and Simon (1998) and Boren and Ramey (2000) argue that the correct use of a UEM is important for two reasons. First, if UEMs used in practice are not reliable and valid, it becomes difficult to compare and replicate studies and therefore to redesign or improve previous versions. However, this reason pales into significance with the second reason which is simply that if UEMs used in practice are not reliable and valid, it becomes difficult to identify problems signalled by the test user from those that might be evoked by the test setting or indeed other intervening factors. Given these implications for incorrect use of the protocols and a reported lack of understanding between the groups, we wanted to understand the knowledge transfer process better.

Complete Chapter List

Search this Book:
Table of Contents
Richard Baskerville
Aileen Cater-Steel, Latif Al-Hakim
Chapter 1
Panagiotis Kanellis, Thanos Papadopoulos
This chapter offers a journey through the spectrum of epistemological and ontological perspectives in IS (IS), offering the necessary background to... Sample PDF
Conducting Research in Information Systems: An Epistemological Journey
Chapter 2
Francis Chia Cua, Tony C. Garrett
This chapter introduces ontological and epistemological elements in information systems research. It argues that ontology, epistemology, and... Sample PDF
Understanding Ontology and Epistemology in Information Systems Research
Chapter 3
John Loonam, Joe McDonagh
Enterprise systems (ES) promise to integrate all information flowing across the organisation. They claim to lay redundant many of the integration... Sample PDF
A Grounded Theory Study of Enterprise Systems Implementation: Lessons Learned from the Irish Health Services
Chapter 4
Khalid Al-Mabrouk
This chapter reviews some of the existing Information Technology Transfer (ITT) literature and suggests that it has fallen victim to the well-known... Sample PDF
A Critical Theory Approach to Information Technology Transfer to the Developing World and a Critique of Maintained Assumptions in the Literature
Chapter 5
João Porto de Albuquerque, Edouard J. Simon, Jan-Hendrik Wahoff, Arno Rolf
Research in the Information Systems (IS) field has been characterised by the use of a variety of methods and theoretical underpinnings. This fact... Sample PDF
The Challenge of Transdisciplinarity in Information Systems Research: Towards an Integrative Platform
Chapter 6
Paul D. Witman
This chapter provides a set of guidelines to assist information systems researchers in creating, negotiating, and reviewing nondisclosure... Sample PDF
A Guide to Non-Disclosure Agreements for Researchers Using Public and Private Sector Sources
Chapter 7
Slinger Jansen
Even though information systems is a maturing research area, information systems case study reports generally lack extensive method descriptions... Sample PDF
Applied Multi-Case Research in a Mixed-Method Research Project: Customer Configuration Updating Improvement
Chapter 8
Erja Mustonen-Ollila, Jukka Heikkonen
This chapter gives important methodological, theoretical, and practical guidelines to the information system (IS) researchers to carry out a... Sample PDF
Historical Research in Information System Field: From Data Collection to Theory Creation
Chapter 9
Paivi Ovaska
Large-scale systems development is a complex activity involving number of dependencies that people working together face. Only a few studies... Sample PDF
A Multi-Methodological Approach to Study Systems Development in a Software Organization
Chapter 10
Judith Symonds
Usability Evaluation Methods (UEM) are plentiful in the literature. However, there appears to be a new interest in usability testing from the... Sample PDF
Usability Evaluation Meets Design: The Case of bisco Office™
Chapter 11
Ivan Ka-Wai Lai, Joseph M. Mula
Soft Systems Methodology (SSM) has been employed to increase the effectiveness of organizational requirement analysis in Information Systems (IS)... Sample PDF
An Analysis-Form of Soft Systems Methodology for Information Systems Maintenance
Chapter 12
Raul Valverde, Mark Toleman, Aileen Cater-Steel
Recently, many organisations have become aware of the limitations of their legacy systems to adapt to new technical requirements. Trends towards... Sample PDF
Design Science: A Case Study in Information Systems Re-Engineering
Chapter 13
Shaligram Pokharel
Information and communication technology (ICT) refer to a family of technologies that facilitate information capturing, storing, processing... Sample PDF
Analyzing the Use of Information Systems in Logistics Industry
Chapter 14
William Yeoh
Engineering asset management organisations (EAMOs) are increasingly motivated to implement business intelligence (BI) systems in response to... Sample PDF
Empirical Investigation of Critical Success Factors for Implementing Business Intelligence Systems in Multiple Engineering Asset Management Organisations
Chapter 15
Ping Li, Joseph M. Mula
A review of the literature showed that there appears to be very little research undertaken on Electronic Data Interchange (EDI) adoption by small to... Sample PDF
Electronic Data Interchange (EDI) Adoption: A Study of SMEs in Singapore
Chapter 16
Hatem F. Halaoui
Using geographical information systems (GIS) has been of great interest lately. A lot of GIS applications are being introduced to regular and... Sample PDF
Towards Google Earth: A History of Earth Geography
Chapter 17
Sergio Di Martino, Filomena Ferrucci, Carmine Gravino
Web technologies are being even more adopted for the development of public and private applications, due to the many intrinsic advantages. Due to... Sample PDF
Empirical Studies for Web Effort Estimation
Chapter 18
Mobile Marketing  (pages 328-341)
Kazuhiro Takeyasu
Recently, cellular phones capable of accessing the Internet are prevailing rapidly in Japan. First, their functions and features are examined... Sample PDF
Mobile Marketing
Chapter 19
Ross A. Malaga
Online auctions are an increasingly popular avenue for completing electronic transactions. Many online auction sites use some type of reputation... Sample PDF
The Retaliatory Feedback Problem: Evidence from eBay and a Proposed Solution
About the Contributors