Learned Behavior: Enabling Believable Virtual Characters through Reinforcement

Introduction

In creating virtual environments, whether for entertainment or serious purposes, believable virtual characters have long been a challenge to developers and users. For large immersive virtual worlds, users have come to expect the presence of computer-generated characters as an essential element of the experience. These characters must act in a way that is reasonable and consistent with their personae, the world, and the context in which they are operating; in other words, they must act believably.

The requirements for achieving a believable virtual character are extensive and non-trivial, but the rewards in terms of immersion and satisfaction make this an important problem to solve (Bailey & Katchabaw, 2008). As written in (Rizzo et al, 1997), characters “are considered believable when they are viewed by an audience as endowed with thoughts, desires, and emotions, typical of different personalities”. This definition of believability is not necessarily a definition of character but an illusion of life, permitting the audience’s suspension of disbelief and acceptance of the virtual as real, at least for a time. While this idea of believability has long been studied in other disciplines, its difficulty in relation to virtual environments invariably comes back to the required interactivity of a virtual character (Loyall, 1997). This level of interactivity requires autonomous and flexible behavior that is not defined a priori. Traditionally, character behavior has been driven by static scripting, pre-coded expert knowledge, or search algorithms. These approaches, however, are simply not sufficient to enable believability; only with narrow problem domains are they sufficient as a solution (Rabin, 2010). Consequently, a more robust cognitive solution is clearly necessary.

In this chapter, we present an approach to character believability enabled by reinforcement learning (Mitchell, 1997) in which behavior is guided by a model of a character’s own unique motivations. In this way, developers are able to develop virtual characters by directly referencing character traits as defined by the underlying narrative or the needs of the virtual environment. The characters will, in turn, learn what actions they must take to benefit their respective motives, in real-time. Not only does this result in interesting and realistic unscripted behavior tuned to the particulars of each individual character, but it does so without the complexity and expense of an extensive and exhaustive programming exercise, as appropriate behaviors can be learned without explicit coding.

The theory behind this approach is based on work done by Reiss (Reiss, 2004; Reiss & Havercamp, 1998), in which motives are the reason that causes a person to initiate and perform a voluntary behavior, and unusually strong or weak desires are used to characterize an individual. In Reiss’s model, motives capture the basic desires of an individual, including power, curiosity, independence, status, social contact, vengeance, honor, idealism, physical exercise, romance, family, order, eating, acceptance, tranquility, and saving. Studies have shown that this theory can be used to describe diverse higher-order aspects of behavior such as religious beliefs (Reiss, 2000), athleticism (Reiss et al, 2001), and lack of scholastic achievement (Reiss, 2004). In general, motives are said to affect perception, cognition, emotion, and ultimately the resultant behaviors. A person must have a motive to perform any particular action, even if that person is not consciously aware of the motive, and so this model is a reasonable basis for believable virtual characters (Bailey et al, 2012). In this approach, when a particular motive is far from satisfied, a character will take appropriate actions that address this shortfall. Similarly, when a motive is fulfilled more than desired, the character will take actions that compensate and avoid actions that contribute to its further advancement.