Abstract
The internet, social networks, emerging virtual/augmented/mixed reality technology platforms and portals are beginning to utilize and display interactive, spatially relative, three-dimensional versions of objects, persons, and environments. The human need to document and archive one's form, behavior, beliefs, experiences, and wishes is an inherent need and desire of our species to preserve and tell their unique life stories. An ability to track and/or capture human movement, expression, environment, and experience with technology designed to acquire hand gestures, body and facial tracking inputs, as well as speech will play an important role in the life documentation process. The eventual goal will be for humankind to interact with, and be remembered as, autonomous virtual agents beyond the scope of physical life, providing “virtual immortality” to any and all that adopt the capability as it evolves in our culture, and with the machines and applications that we utilize. This chapter explores the age of avatarism.
TopBackground
Naming Conventions For Human Simulation Constructs
As any important development or aspect of, history, culture or technology begins to appear, it becomes necessary to name and describe its relevance, meaning and future. The field of human simulation is a broad area of study and will continue to require further definition using common, easy to understand language.
A variety of example attempts to describe aspects of human simulation have already been defined and are being used. Certain words have gained traction and are easily recognizable, others less so. Figure 1 shows a chart that attempts to illustrate some of the more commonly used word fragments and the resulting combinations that have the potential to simplify understanding and discussion related to the field.
Figure 1. Some of the vocabulary surrounding human simulation
The word Avatar tops the chart, as it has become the most widely adopted word to describe a human simulated construct, although now the word is not specifically anthropomorphic in meaning. Today an Avatar can take any form to represent a person. Some of the other familiar terms in the chart have more specific intentions or meanings that most English-speaking people can associate with aspects of human simulation, however, we do not, as yet, have specific, universal understandings of these terms.
Rather than adherence to any specific word, this chapter will remain general in discussing the art, science and philosophy of human simulation. The author’s personal choice would be to use the word “construct” in most cases, although, it is not listed on the chart and seldom appears in literature and presentations. It is for this reason the word is less likely to be confused or constrained to any specific context, so that it can serve as a valuable device, and thus it will be used freely in this regard within the remainder of the text.
Key Terms in this Chapter
Time-of-Flight: A technique used to measure the time taken by an invisible, infrared light pulse to travel from an emitter to a subject and return to a detector (sensor). This information can be used to establish a method to measure approximate distance to many points on the surface of a target object.
Deep Fakery (aka Deep Fake): A portmanteau of “deep learning” and “fake” (coined in 2017) is a technique for human image synthesis based on artificial intelligence and machine learning. The combination and superimposition of existing images and videos onto source images or videos using a technique known as generative adversarial network (GAN). Deep fakes have been used to create illicit, simulated celebrity pornographic videos and more recently are being used to create fake news and malicious hoaxes, easily spread on social media sites ( https://en.wikipedia.org/wiki/Deepfake ).
Healthmap: A detailed, interactive data set, fully referenced and aligned with a personalized 3D human model.
Machine Learning: The scientific study of algorithms and statistical models that computer systems use to perform a specific task without using explicit instructions, relying on patterns and inference instead. Seen as a subset of artificial intelligence, machine learning algorithms build a mathematical model based on sample data, known as “training data”, in order to make predictions or decisions without being explicitly programmed to perform a task ( https://en.wikipedia.org/wiki/Machine_learning ).
Volumetric Video: A video technique that captures a 3D spatial volume in real-time, such as a location or human performance. The acquired 3D data that can be viewed on flat screens as well as using 3D Displays and VR goggles. The main, compelling feature of this technique is the ability for a viewer (or audience) to navigate and change their point-of-view when observing or exploring the generated capture volume. Various methods such as photogrammetry, 3D depth camera arrays and multi-point visible light video camera arrays are used to generate a volumetric video database.
3D Digitizing Rig: A graphics input system that records x, y and z coordinates of a real object. Contact is made with various points on the object's surface by a light sensor, sound sensor, robotic instrument, or pen.
Point cloud: A set of data points in space, generally produced by 3D scanners and used to represent 3D objects in visualization, animation, rendering and 3D manufacturing applications ( https://en.wikipedia.org/wiki/Point_cloud ).
Structure From Motion: A range imaging technique used to estimate the 3D form / structure of an object or scene. A camera (or the object) must be moving during a video captured clip and the resulting video frames are output as 2D images for photogrammetric processing (by computer) resulting in the creation of a 3D point cloud representation of an object / scene captured. The generated output is then topologized into a suitable 3D format for use in simulation, re-construction using 3D printing and for many other applications including the creation of 3D topological maps, 3D model building and for ground plane detection required to perform augmented reality (AR) spatial anchoring from an observed scene ( https://en.wikipedia.org/wiki/Structure_from_motion ).
Structured Light: A process of projecting a known pattern (often grids / horizontal bars) on to a scene that deform when striking surfaces, allows vision systems (cameras) to calculate depth and surface information of objects in view. 3D scanners can utilize invisible (often infrared) structured light, without visible detection, as a reliable, accurate way to create 3D objects and measure distances in an observed scene ( https://en.wikipedia.org/wiki/Structured_light ).
Avatarism: A term that characterizes the recent manifestation of technology applications applied to the creation and usage of personal virtual representations for various purposes.
Construct: In the context of this chapter, a “construct” can be interpreted as a virtual embodiment (or representation/imitation of a person or thing. Also referred to in this context as “simulacra”, avatar and many other terms. See https://en.wikipedia.org/wiki/Construct for disambiguation.
Vactor: A virtual actor, first coined and subsequently trademarked by SimGraphic Engineering in the early 1990’s to describe a proprietary computer system which allows a live, human performer to control a real-time computer-generated character or construct for entertainment and other applications.
Waldo: A device used by an actor that translated the motion of the face to control a generated character. A word taken from a Robert Heinlein science fiction story: Waldo and Magic, Inc . (1950) about a handicapped man who uses remote manipulators to interact with things he cannot physically.
Depth-Sensing Camera: A camera which is able to determine accurate distance (range) information in addition to visual information using a variety of technologies.
Fiducial Markers: An object, image or printed pattern, when placed in the field of view of an imaging system, a fiducial can be used as a point of reference or a measure to determine a position, often within a sequence of elapsed time. See: https://en.wikipedia.org/wiki/Fiducial_marker .
Performance Animation: Control and creation of an animated virtual character in real-time with the use digitally tracked body and facial action by a performer, puppeteer, or actor.
Holographic Recordings: Captured video sequences, usually of a person’s performance obtained by the use of volumetric capture, depth camera acquisition, or other methods. A holographic recording, one created and edited, can be seen from any angle or perspective and re-positioned with an area of 3D space as viewed in virtual/augmented reality or through 2D stereographic video on a flat screen.
Holoportation: A type of real-time, 3D human capture technology that allows a visual re-creation of people to be compressed, transmitted and reconstructed at a remote location for live, immediate, interpersonal communication and engagement.