Digitizing the Intangible: Machine Learning Applications Over Complex 3D Trajectories

Introduction

In performing arts, such as dance, classical or contemporary, body signals, i.e. movements and gestures, are intentionally used to punctuate a storyline, in an aesthetically pleasing and thorough way. The observed kinesiology is a form of Intangible Cultural Heritage (ICH), directly connected to local culture and identity (Marolt et al., 2009). ICH preservation is of great interest to both the scientific and cultural communities, as well as, the general public. The most prominent challenges involved are associated with the complex structure of ICH; i.e. its dynamic nature, the interactions among objects and environment, and emotional elements, e.g. dancers’ expressions and style (Aristidou et al., 2015).

The preservation of folk dances is, nowadays, a basic requirement (A. D. Doulamis et al., 2017). The history and style of dance should become available to the public, through a system that includes descriptive information, videos, movement, and 3D modelled data relevant to it. Digital documentation of tangible and intangible heritage, data formats and standards, metadata and semantics, linked data, crowdsourcing and cloud, the use and reuse of data and copyright issues are some of the rising challenges in the field (Nikolaos Doulamis et al., 2017). However, prior to any of the above challenges the digitization of the information itself, i.e. moving patterns and tempo remains the main task.

Recent technological advancements, including ubiquitous mobile devices, pervasive video capturing sensors and software, increased camera and display resolutions, cloud storage solutions, and motion capture technologies, have unleashed tremendous possibilities in capturing, documenting and storing ICH content. However, utilizing the full potential of the massive, high-quality multimodal (text, image, video, 3D, mocap) ICH data is not an easy feat. Researchers need to appropriately adopt state-of-the-art techniques or invent new ones; multiple fields are involved as artificial intelligence (AI), computer vision, and image processing. Existing knowledge is essential for the ICH—in our case, dance—content’s efficient and effective organization and management, fast indexing, browsing, and retrieval, but also semantic analysis, such as automatic recognition (Kosmopoulos et al., 2013; A. S. Voulodimos et al., 2012) and classification (N. D. Doulamis et al., 2010; A. Voulodimos et al., 2011).

Ever since the introduction of the first Kinect sensor, depth cameras were widely used as low-cost peripherals for several applications. Furthermore, the advent of motion sensing devices and depth cameras has boosted the fields of motion analysis and monitoring, including human tracking (N. Doulamis & Voulodimos, 2016; Lalos et al., 2014), action recognition (Kosmopoulos et al., 2010; A. S. Voulodimos et al., 2014), and pose estimation(Bakalos et al., 2019). The main advantage of a depth camera is that produces dense and reliable depth measurements, albeit over a limited range and offers balance in usability and cost.

These, relatively recent, advances in depth sensors lead to the development of low-cost 3D capturing systems, such as Microsoft Kinect (Zhang, 2012) or Intel RealSense (Keselman et al., 2017), and allowed for easy capturing of human skeleton joints in 3D space, which are then properly analyzed to extract dance kinematics (A. Voulodimos, Rallis, et al., 2018). Nevertheless, digitization does not guarantee preservation in the case of folklore performing arts. The documentation and the development of interactive frameworks, that enhances the learning procedure, or generate additional knowledge, is required; e.g. the creation serious game platforms, allowing the users to achieve a rich learning experience (Kitsikidis et al., 2015).