Training Coordination Proxy Agents Using Reinforcement Learning

Training Coordination Proxy Agents Using Reinforcement Learning

Myriam Abramson (Naval Research Laboratory, USA)
DOI: 10.4018/978-1-60566-236-7.ch011
OnDemand PDF Download:


In heterogeneous multi-agent systems, where human and non-human agents coexist, intelligent proxy agents can help smooth out fundamental differences. In this context, delegating the coordination role to proxy agents can improve the overall outcome of a task at the expense of human cognitive overload due to switching subtasks. Stability and commitment are characteristics of human teamwork, but must not prevent the detection of better opportunities. In addition, coordination proxy agents must be trained from examples as a single agent, but must interact with multiple agents. We apply machine learning techniques to the task of learning team preferences from mixed-initiative interactions and compare the outcome results of different simulated user patterns. This chapter introduces a novel approach for the adjustable autonomy of coordination proxies based on the reinforcement learning of abstract actions. In conclusion, some consequences of the symbiotic relationship that such an approach suggests are discussed.
Chapter Preview


Advances in communication technologies has led to increased agent interactions and increased complexity in the decision-making process. To deal with this added burden, the coordination role is delegated to a proxy agent. Coordination proxy agents [Scerri et al., 2003] are personal agents that take on the coordination role on behalf of a human user (Figure 1). While the optimization of the global task can be better achieved by the self-organization of proxy agents in dynamic environments, switching roles or teams involves preferences, such as loyalty, boredom, and persistence thresholds, in addition to interpretations that might need to be elicited from the human in the loop. For example, individual drivers differ in their tendency to switch lanes in urban traffic; truck drivers might prefer a less optimal route going through their favorite spots. This chapter addresses issues in determining when switching roles or teams is appropriate to satisfy both the urgency of the subtask relative to the global task, the preferences of the user, and when input from the user is warranted. We hypothesize that a distinct class of agents, proxy agents, will emerge at the junction of the human and non-human worlds that will take on not only decision-making tasks such as coordination, but also the social interactive task and the adaptation task on our behalf. We envision those agents to be embedded in personal mobile devices such as cell phones and personal digital assistants and personalized through a training process.

Figure 1.

Example of coordination proxies helping in traffic by negotiating the road

In this chapter, we claim that through result-driven reinforcement learning, the human can train coordination proxies in a task with examples biasing the way the task is achieved with respect to the outcome of the task in a multiagent system. Similarly, in mixed-initiative planning involving goal selection, directives from the user are obtained interactively in case of plan conflict or provided a priori in the form of plan constraints. Mixed-initiative interactions in multi-agent systems provide a flexible way to harness the cognitive capabilities of the human in the loop in solving a problem while delegating more mundane tasks to the proxy agents. As in the turn-taking problem found in dialog management [Allen, 1999], the key decisions for mixed-initiative interactions, as applied to the adjustable autonomy of proxy agents, include knowing when to ask for help, when to ask for more information, and when to inform the user of a decision. This chapter claims that learning user preferences is not sufficient for training coordination proxies if those preferences conflict with other agents’ preferences and affect the outcome of the task. As long as preferences are inconsistent with each other as evidenced by the outcome of the task, a proxy agent must keep training and continue interacting while suggesting alternatives.

This chapter is organized as follows. A learning approach for training coordination proxies in making decisions is first introduced. We then motivate experiments in the prey/predator canonical coordination domain and present empirical results and an analysis of our evaluation. Finally, we conclude with a summary of related work and extrapolate on the consequences of such interactions. The key contribution of this work is a mixed-initiative approach based on the reinforcement learning of abstract actions and its algorithm scalable to large state space for the adjustable autonomy problem of coordination proxy agents.

Complete Chapter List

Search this Book:
List of Reviewers
Table of Contents
Georgi Stojanov
Chapter 1
R. Keith Sawyer
Sociology should be the foundational science of social emergence. But to date, sociologists have neglected emergence, and studies of emergence are... Sample PDF
The Science of Social Emergence
Chapter 2
Christopher Goldspink, Robert Kay
This chapter critically examines our theoretical understanding of the dialectical relationship between emergent social structures and agent... Sample PDF
Agent Cognitive Capabilities and Orders of Social Emergence
Chapter 3
Joseph C. Bullington
Social interaction represents a powerful new locus of research in the quest to build more truly human-like artificial agents. The work in this area... Sample PDF
Agents and Social Interaction: Insights from Social Psychology
Chapter 4
M. Afzal Upal
This chapter will critically review existing approaches to the modeling transmission of cultural information and advocate a new approach based on a... Sample PDF
Predictive Models of Cultural Information Transmission
Chapter 5
Jorge A. Romero
Despite the popularity of agents for the information technology infrastructure, questions remain because it is not clear what do e-business agents... Sample PDF
Interaction of Agent in E-Business: A Look at Different Sources
Chapter 6
Adam J. Conover
This chapter presents a description of ongoing experimental research into the emergent properties of multi-agent communication in “temporally... Sample PDF
A Simulation of Temporally Variant Agent Interaction via Passive Inquiry
Chapter 7
Richard Schilling
This chapter presents a generalized messaging infrastructure that can be used for distributed agent systems. The principle of agent feedback... Sample PDF
Agent Feedback Messaging: A Messaging Infrastructure for Distributed Message Delivery
Chapter 8
Yu Zhang, Mark Lewis, Christine Drennon, Michael Pellon, Coleman
Multi-agent systems have been used to model complex social systems in many domains. The entire movement of multi-agent paradigm was spawned, at... Sample PDF
Modeling Cognitive Agents for Social Systems and a Simulation in Urban Dynamics
Chapter 9
Scott Watson, Kerstin Dautenhahn, Wan Ching (Steve) Ho, Rafal Dawidowicz
This chapter discusses certain issues in the development of Virtual Learning Environments (VLEs) populated by autonomous social agents, with... Sample PDF
Developing Relationships Between Autonomous Agents: Promoting Pro-Social Behaviour Through Virtual Learning Environments Part I
Chapter 10
Martin Takác
In this chapter, we focus on the issue of understanding in various types of agents. Our main goal is to build up notions of meanings and... Sample PDF
Construction of Meanings in Biological and Artificial Agents
Chapter 11
Myriam Abramson
In heterogeneous multi-agent systems, where human and non-human agents coexist, intelligent proxy agents can help smooth out fundamental... Sample PDF
Training Coordination Proxy Agents Using Reinforcement Learning
Chapter 12
Deborah V. Duong
The first intelligent agent social model, in 1991, used tags with emergent meaning to simulate the emergence of institutions based on the principles... Sample PDF
The Generative Power of Signs: The Importance of the Autonomous Perception of Tags to the Strong Emergence of Institutions
Chapter 13
Josefina Sierra, Josefina Santibáñez
This chapter addresses the problem of the acquisition of the syntax of propositional logic. An approach based on general purpose cognitive... Sample PDF
Propositional Logic Syntax Acquisition Using Induction and Self-Organisation
Chapter 14
Giovanni Vincenti, James Braman
Emotions influence our everyday lives, guiding and misguiding us. They lead us to happiness and love, but also to irrational acts. Artificial... Sample PDF
Hybrid Emotionally Aware Mediated Multiagency
Chapter 15
Samuel G. Collins, Goran Trajkovski
In this chapter, we give an overview of the results of a Human-Robot Interaction experiment, in a near zerocontext environment. We stimulate the... Sample PDF
Mapping Hybrid Agencies Through Multiagent Systems
Chapter 16
Scott Watson, Kerstin Dautenhahn, Wan Ching (Steve) Ho, Rafal Dawidowicz
This chapter is a continuation from Part I, which has described contemporary psychological descriptions of bullying in primary schools and two... Sample PDF
Developing Relationships Between Autonomous Agents: Promoting Pro-Social Behaviour Through Virtual Learning Environments Part II
Chapter 17
Mario Paolucci, Rosaria Conte
This chapter is focused on social reputation as a fundamental mechanism in the diffusion and possibly evolution of socially desirable behaviour... Sample PDF
Reputation: Social Transmission for Partner Selection
Chapter 18
Adam J. Conover
This chapter concludes a two part series which examines the emergent properties of multi-agent communication in “temporally asynchronous”... Sample PDF
A Simulation of Temporally Variant Agent Interaction via Belief Promulgation
Chapter 19
David B. Newlin
Following the discovery in Rhesus monkeys of “mirror neurons” that fire during both execution and observation of motor behavior, human studies have... Sample PDF
The Human Mirror Neuron System
Chapter 20
Eric Baumer, Bill Tomlinson
This chapter presents an argument that the process of emergence is the converse of the process of abstraction. Emergence involves complex behavior... Sample PDF
Relationships Between the Processes of Emergence and Abstraction in Societies
Chapter 21
Vern R. Walker
In modern legal systems, a large number of autonomous agents can achieve reasonably fair and accurate decisions in tens of thousands of legal cases.... Sample PDF
Emergent Reasoning Structures in Law
Chapter 22
Theodor Richardson
Network Intrusion Detection Systems (NIDS) are designed to differentiate malicious traffic, from normal traf- fic, on a network system to detect the... Sample PDF
Agents in Security: A Look at the Use of Agents in Host-Based Monitoring and Protection and Network Intrusion Detection
Chapter 23
Michael J. North, Thomas R. Howe, Nick Collier, Eric Tatara, Jonathan Ozik, Charles Macal
Search has been recognized as an important technology for a wide range of software applications. Agentbased modelers often face search challenges... Sample PDF
Search as a Tool for Emergence
About the Contributors