Creating Social Technologies to Assist and Understand Social Interactions

Creating Social Technologies to Assist and Understand Social Interactions

Anton Nijholt (University of Twente, The Netherlands), Dirk Heylen (University of Twente, The Netherlands) and Rutger Rienks (University of Twente, The Netherlands)
DOI: 10.4018/978-1-60566-264-0.ch028
OnDemand PDF Download:


In this chapter the authors discuss a particular approach to the creation of socio-technical systems for the meeting domain. Besides presenting a methodology this chapter will present applications that have been constructed on the basis of the method and applications that can be envisioned. Throughout the chapter, illustrations are drawn from research on the development of meeting support tools. The chapter concludes with a section on implications and considerations for the on-going development of social technical systems in general and for the meeting domain in particular.
Chapter Preview

Assimilation into the Borg Collective might be inevitable, but we can still make it a more human place to live.

—Pentland, 2005



Socio-technical computing inherits the complexity related to software engineering and system integration whilst embedding the human in the loop. It also inherits the difficulties of understanding and modeling human-human and human-computer interaction in the context of a changing environment (see Clancey, 1997). In this chapter we will outline an approach to the development of Social Technical Systems, with the focus on meeting support. This approach can be characterized as theory-informed data-driven. In essence the method consists of the following four steps.

  • Step 1: Collection of a multimodal corpus of social activity signals

  • Step 2: Description of a myriad of aspects of system relevant activities (annotation) in the collected material

  • Step 3: Discovery of interdependencies between recorded signals and annotations, annotations and annotations, and signals and signals (e.g. by means of machine learning.)

  • Step 4: System creation based on knowledge obtained from the previous steps

In the collection and annotation steps, the process relies heavily on the insights provided by the social sciences; in particular sociology, social psychology and linguistics. In return, the annotated collection and the machine learning effort may provide important insights for social theorizing as the annotated corpus provides the researcher with statistics about the occurrence and distribution of certain phenomena and interesting correlations. Increased insight into how people behave can point out problems they encounter in their activities that may be relieved by technologies that are based on this understanding of their activities as derived through Steps 1 to 3. This means that these steps can be viewed both as a way into requirements engineering and as providing the basic data and algorithms to build the tools that can solve some of these problems.

Technology that inherits these possibilities can be said to be social for three reasons. The first is in the way in which the system supports social activities. The second relates to the way the technology can provide insight into social processes which occurs when correlations between phenomena are found. The third reason in which the qualifier social relates to the term technical system is in how social theories are at the basis of the construction of the technical applications. Given theories on how humans ‘operate’, technology is equipped with the manual in order to understand and support their operating.

As example case for this chapter our focus is on small business meetings. Currently several projects worldwide are investigating the way technology can support the needs of people in meetings and how it can relieve them of some of the frustrations that meetings seem to impose upon them. Examples in this chapter will be drawn mainly from studies in a series of European projects on meeting analysis and meeting support: M4, AMI, and AMIDA. These projects investigated how human-centred computing techniques can detect and interpret activities of participants in smart meeting rooms and how these techniques can be used to design tools that support meeting participants in their encounters and activities.

This chapter discusses a variety of methodological issues and charts several results showing the rationale behind the scientific drive to develop technological support for social gatherings and events. The chapter also contains a short discussion on ethical issues and potential pitfalls on the road ahead.

Key Terms in this Chapter

Smart Meeting Room: A smart meeting room uses multi-modal sensors to detect and capture the verbal and nonverbal behavior of meeting participants. This is done in order to provide real-time support to these participants and to record meeting activity for off-line intelligent browsing and retrieval of meeting activities. Modeling multi-party human-to-human interaction, e.g. by using machine learning approaches, helps to recognize important activities and events during a meeting.

Sensor Information: Sensors in smart environments provide us with information about its inhabitants, their activities, and their interactions. Cameras and microphones allow audio-visual processing of perceived activity. Proximity and pressure sensors tell us about the location of inhabitants. Such sensors allow us to track the inhabitants and their activities in the environment. Devices that measure physiological information, including brain activity, can provide detailed information about the affective state of a user.

Machine Learning: Machine Learning: This subfield of artificial intelligence is concerned with the design, analysis, implementation and applications of programs that learn from experience. The discovery of general rules from large data sets using computational and statistical methods is an important application area. Such large data sets can, for example, be corpora that contain audio and video recorded human-human or human-computer nteraction.

Multimodal Interface: Interface to a computer system (from a mobile device to a smart environment) that allows multiple modes of interaction. Among the modalities can be speech, touch, gaze, or gestures. Modalities can supplement one another, but also complement one another.Combining different input modalities is called fusion. It allows a system to disambiguate user input in order to get a more complete understanding of a user’s commands or behavior.

Corpus-based Research: Traditionally a corpus is a collection of language examples: written or spoken examples of words, sentences, phrases or texts. Nowadays a corpus can be any collection of examples, for example, human-human interactions, protoin interaction, video fragments, maintenance information, etc. A corpus is collected in order to learn from it, that is, to extract domain-specific information. Examples can be analysed and rules and models underlying the examples can be discovered. Machine learning algorithms are used to extract relationships between examples. Manual structuring of such data (annotation) allows the integration of human preferences and knowledge in machine learning algorithms.

Nonverbal Behaviour: Nonverbal behaviour not only supports verbal communication. By observing nonverbal behavior, the observer, whether it is a computer system or a human observer, can learn about the intentions, the attitudes and the feelings of its human partner. Nonverbal behavior includes gaze behavior, facial expressions, body posture, gestures, and prosodic information, but it can also include physiological information. Hence, supporting verbal communication, issuing nonverbal commands, and allowing our human or computer partners to learn about our feelings, intentions, and preferences are the main reasons for needing to detect and interprete nonverbal behavior.

Annotation Process: A corpus of examples, whether these are language or interaction examples (distinguishing between different kinds of interaction) can be annotated with human knowledge that makes it possible to distinguish characteristics of these examples. Machine learning algorithms can be guided and supported by such annotations and machine learning results provide feedback about our intuition and heuristics concerning which features of the examples help to distinguish them into classes. To support human annotators, tools are developed that visualize and otherwise emphasize characteristics of the examples in the corpus.

Complete Chapter List

Search this Book:
Editorial Advisory Board
Table of Contents
Ben Shneiderman
Brian Whitworth, Aldo de Moor
Brian Whitworth, Aldo de Moor
List of Reviewers
Prologue: General Socio-Technical Theory
Chapter 1
Brian Whitworth
A socio-technical system (STS) is a social system built upon a technical base. An STS adds social requirements to human-computer interaction (HCI)... Sample PDF
The Social Requirements of Technical Systems
Chapter 2
Matti Tedre
This chapter introduces the reader to some social research characteristics that are central to the social study of computer science. It introduces... Sample PDF
The Social Study of Computer Science
Chapter 3
Ann Borda, Jonathan P. Bowen
This chapter introduces the concept of a Virtual Organization (VO), using the Internet to link geographically separated participants in an efficient... Sample PDF
Virtual Collaboration and Community
Chapter 4
David Davenport
This chapter analyses the effect that social values have on the design of technical systems. Beginning with an examination of the role technology... Sample PDF
The Social Derivation of Technical Systems
Chapter 5
Ken Eason, José Abdelnour-Nocera
This chapter sets the traditional focus of socio-technical systems theory on primary work systems in a modern context where information and... Sample PDF
Socio-Technical Theory and Work Systems in the Information Age
Chapter 6
Peter Day
This chapter introduces the community engagement strategy of the Community Network Analysis (CNA) project and considers its significance to research... Sample PDF
An Engagement Strategy for Community Network Research and Design
Chapter 7
Cleidson R.B. de Souza, David F. Redmiles
This chapter reviews the socio-technical relationship between organizational and software structure. It describes the early theoretical work about... Sample PDF
On the Alignment of Organizational and Software Structure
Ronald K. Stamper
Prologue: Socio-Technical Perspectives
Chapter 8
Catherine Heeney
The chapter discusses the traditional expectations about privacy protection and argues that current models for the governance of data do not... Sample PDF
Privacy and the Identity Gap in Socio-Technical Systems
Chapter 9
Ronald Leenes
Second Life can be seen as a social microcosmos in which fairly normal people lead a social life and where social needs develop. Privacy is one of... Sample PDF
Privacy Regulation in the Metaverse
Chapter 10
David Tuffley
This chapter introduces a process reference model of leadership for integrated teams operating in virtual environments. Geographically dispersed... Sample PDF
Leadership of Integrated Teams in Virtual Environments
Chapter 11
Monique Janneck
For a technology use to be successful, the circumstance of its introduction into a use context—or recontextualization— is crucial. The users of a... Sample PDF
Recontextualising Technology in Appropriation Processes
Chapter 12
Petter Bae Brandtzæg, Jan Heim
The last few years have seen a substantial growth in online communities such as MySpace and Facebook. In order to survive and increase in size... Sample PDF
Explaining Participation in Online Communities
Chapter 13
Malcolm Shore
This chapter is about the way in which computer hackers invoke social networking paradigms to support and encourage their activities. It reviews the... Sample PDF
Cyber Security and Anti-Social Networking
Chapter 14
Wilson Huang, Shun-Yung Kevin Wang
This chapter examines the gaps that arise between reactive social control systems and proactive technology systems. The authors further link these... Sample PDF
Emerging Cybercrime Variants in the Socio-Technical Space
Chapter 15
Elayne W. Coakes, Peter Smith, Dee Alwis
This chapter presents the argument that service innovation is promoted by supporting divergent interpretations, enlarging the scope of employee and... Sample PDF
Developing Innovative Practice in Service Industries
Mark Aakhus
Prologue: Socio-Technical Analysis
Chapter 16
Hans Weigand
Often socio-technical systems are designed simply on the basis of what the user asks, and without considering explicitly whether the required... Sample PDF
Using Communication Norms in Socio-Technical Systems
Chapter 17
Jonas Sjöström, Göran Goldkuhl
This chapter introduces the theoretical framework of Socio-Instrumental Pragmatism (SIP) and illustrates how it has been used as an analytic... Sample PDF
Socio-Instrumental Pragmatism in Action
Chapter 18
Paul J. Bracewell
Analytics provides evidence for objective corporate decision-making. Lack of understanding of analytical techniques can create confusion amongst... Sample PDF
A Framework for Using Analytics to Make Decisions
Chapter 19
Mikael Lind, Peter Rittgen
Setting up co-design processes involving several stakeholders is a complex task. In this chapter the authors have looked upon experiences from... Sample PDF
The Challenges of Co-Design and the Case of e-Me
Chapter 20
Harry S. Delugach
Automated tools are often used to support software development workflows. Many of these tools are aimed toward a development workflow that relies... Sample PDF
Formal Analysis of Workflows in Software Development
Chapter 21
Dorit Nevo, Brent Furneaux
This chapter reviews the significance of expectations to information systems development with particular emphasis on the process of requirements... Sample PDF
The Role of Expectations in Information Systems Development
Chapter 22
Jeff Axup
With mobile technologies increasingly weaving themselves into the fabric of our communities, it would be beneficial to increase our understanding of... Sample PDF
Building a Path for Future Communities
Thomas Erickson
Prologue: Socio-Technical Design
Chapter 23
Thomas Herrmann
Socio-technical systems integrate technical and organizational structures and are related to various stakeholders and their perspectives. The design... Sample PDF
Systems Design with the Socio-Technical Walkthrough
Chapter 24
Anders I. Mørch
This chapter presents a translational approach to socio-technical design, as a new approach to the theorybased design of user interfaces, supported... Sample PDF
Applied Pragmatism and Interaction Design
Chapter 25
Manuel Kolp, Yves Wautelet
Information systems are deeply linked to human activities. Unfortunately, development methodologies have been traditionally inspired by programming... Sample PDF
A Social Framework for Software Architectural Design
Chapter 26
Designing for Trust  (pages 388-401)
Piotr Cofta
Designing for trust is a methodology that attempts to design our perception of trust in information systems, in the long-term expectation that such... Sample PDF
Designing for Trust
Chapter 27
Dan Dixon
Three decades ago the concept of pattern languages were introduced in the field of architecture and they have since become widely used in... Sample PDF
Pattern Languages for CMC Design
Chapter 28
Anton Nijholt, Dirk Heylen, Rutger Rienks
In this chapter the authors discuss a particular approach to the creation of socio-technical systems for the meeting domain. Besides presenting a... Sample PDF
Creating Social Technologies to Assist and Understand Social Interactions
Chapter 29
Jos Benders, Ronald Batenburg, Paul Hoeken, Roel Schouteten
This chapter sketches an Organization Design perspective called “Modern Socio-technical Design”, and subsequently discusses the implementation of... Sample PDF
A Modern Socio-Technical View on ERP-Systems
Chapter 30
Mary Allan, David Thorns
The chapter introduces the Bourdieuean habitus and field theory as a framework for an alternative way of investigating how perceptions of Media Rich... Sample PDF
Being Face to Face: A State of Mind or Technological Design?
Chapter 31
Rebecca M. Ellis
This chapter introduces the work of sociologist Pierre Bourdieu and his concepts of “the field” and “capital” in relation to eBay. In any given... Sample PDF
Applying Bourdieu to eBay's Success and Socio-Technical Design
Chapter 32
Christopher A. Miller
This chapter focuses not on technology mediation of human relationships, but rather on human-like relationships with technology itself. The author... Sample PDF
Relationships and Etiquette with Technical Systems
Anton Nijholt
Prologue: Socio-Technical Implementation
Chapter 33
Laura Anna Ripamonti, Ines Di Loreto, Dario Maggiorini
The necessity of supporting more and more social interaction (and not only mere information sharing) in online environments is the disruptive force... Sample PDF
Augmenting Actual Life Through MUVEs
Chapter 34
Mohamed Ben Ammar, Mahmoud Neji, Adel M. Alimi
Affective computing is a new artificial intelligence area that deals with the possibility of making computers able to recognize human emotions in... Sample PDF
The Role of Affect in an Agent-Based Collaborative E-Learning System Used for Engineering Education
Chapter 35
Pernilla Qvarfordt, Shumin Zhai
Eye-gaze plays an important role in face-to-face communication. This chapter presents research on exploiting the rich information contained in human... Sample PDF
Gaze-Aided Human-Computer and Human-Human Dialogue
Chapter 36
Licia Calvi
The chapter presents and combines the results of two case studies dealing with online communities1 in order to understand under which conditions... Sample PDF
How to Engage Users in Online Sociability
Chapter 37
Ivan Launders
The UK National Health Service (NHS) provides the opportunity to undertake local socio-technical system design to help staff maximize the... Sample PDF
Socio-Technical Systems and Knowledge Representation
Chapter 38
Claire de la Varre, Julie Keane, Matthew J. Irvin, Wallace Hannum
This chapter describes the design of a sociotechnical system to support rural high school students in an online distance education (ODE) course. The... Sample PDF
Social Support for Online Learning
Chapter 39
Jeremy Birnholtz, Emilee J. Rader, Daniel B. Horn, Thomas Finholt
This chapter uses the theoretical notion of common ground to explore remote participation in experimental research. On one hand, there is a desire... Sample PDF
Enabling Remote Participation in Research
Starr Roxanne Hiltz
Prologue: Socio-Technical Evaluation
Chapter 40
John M. Carroll, Mary Beth Rosson, Umer Farooq, Jamika D. Burge
Socio-technical systems are social systems that incorporate technological infrastructures. At the group level of analysis, the most important... Sample PDF
Community Collective Efficacy
Chapter 41
Tanguy Coenen, Wouter Van den Bosch, Veerle Van der Sluys
This chapter views social networking sites as supporting social capital and the advantages which derive from it, namely emotional support... Sample PDF
An Analysis of the Socio-Technical Gap in Social Networking Sites
Chapter 42
Olga Kulyk, Betsy van Dijk, Paul van der Vet, Anton Nijholt, Gerrit van der Veer
This chapter addresses awareness support to enhance teamwork in co-located collaborative environments. In particular, the authors focus on the... Sample PDF
Situational Awareness In Collaborative Work Environments
Chapter 43
Janet L. Holland
This chapter deals with research on the development and use of an assessment instrument for measuring affective satisfaction in online learning. The... Sample PDF
A Scale of Affective Satisfaction in Online Learning Communities
Chapter 44
David Hinds, Ronald M. Lee
In this chapter, the authors suggest how measures of “social network health” can be used to evaluate the status and progress of a virtual community.... Sample PDF
Assessing the Social Network Health of Virtual Communities
Chapter 45
Bertram C. Bruce, Andee Rubin, Junghyun An
This chapter introduces situated evaluation as an approach for evaluating socio-technical innovation and change. Many current evaluations simply... Sample PDF
Situated Evaluation of Socio-Technical Systems
Chapter 46
Heike Winschiers-Theophilus
Communities all over the world have established their own value systems which do not necessarily correlate with the intrinsic values of technology.... Sample PDF
Cultural Appropriation of Software Design and Evaluation
Charles Steinfield
Prologue: The Future of Socio-Technical Systems
Chapter 47
Peter J. Denning
Wicked problems (messes) are tangled social situations that are too costly to stay in and too intransigent to get out of. Collaboration is essential... Sample PDF
Resolving Wicked Problems through Collaboration
Chapter 48
Rachel McLean
As a social activity, the shopping experience can not be recreated or improved through technical design alone. This chapter proposes that there is... Sample PDF
The Myth of the e-Commerce Serf to Sovereign Powershift
Chapter 49
Theresa Dirndorfer Anderson
This chapter explores the challenges associated with teaching the principles of socio-technical systems in the dynamic climate that characterizes... Sample PDF
Teaching the Socio-Technical Practices of Tomorrow Today
Chapter 50
Isa Jahnke
The chapter describes an empirical study of a socio-technical community—as an extended part of an institution— with the aim of revealing its... Sample PDF
Socio-Technical Communities: From Informal to Formal?
Chapter 51
Laurence Claeys, Johan Criel
This chapter introduces the concept of critical user participation as a means to see the socio-technical gap in context aware applications as an... Sample PDF
Future Living in a Participatory Way
Chapter 52
Paul Hodgson
This chapter analyses the formation and generation of social trust through communications technology in postmodern society, and presents some... Sample PDF
The Impact of Communications Technology on Trust
Chapter 53
Kenneth E. Kendall, Julie E. Kendall
This chapter explores the social, organizational, and individual impacts of emerging information technologies using the advent of recent... Sample PDF
Good and Evil in the Garden of Emerging Information Technologies
About the Contributors