Statistical Machine Learning Approaches for Sports Video Mining Using Hidden Markov Models

Introduction

Video mining is to discover knowledge, structures, patterns and events of interest in the video data, and its benefits range from efficient browsing and summarization of video content to facilitating video access and retrieval in a large database or online multimedia repository. There are various types of video data. According to different production and edition styles, videos can be classified into two main categories: scripted and non-scripted videos (Xiong, Zhou, Tian, Rui, & Huang, 2006). Scripted videos are produced according to a certain script or plan that are later edited, compiled, and distributed to users for consumption. News and movies are the examples of scripted videos. Most research efforts on scripted videos focus on the development of a Table-of-Content (TOC) that provides an overview of the video’s naturally organized content to support efficient browsing and indexing. On the other hand, the events in non-scripted videos happen spontaneously and usually in a relatively fixed setting, such as meeting videos, sports videos, and surveillance videos. The research on non-scripted videos involves detecting high-lights and events-of-interest. In this chapter sports video mining is studied using American football as the case study.

Although sports videos are non-scripted content, there are still definite or repetitive structures and patterns. Using these structures and patterns, we can develop some flexible and effective tools for video browsing/indexing. Currently, there are two kinds of approaches for sports video mining, structure-based (Kokaram, et al., 2006; Xie, Chang, Divakaran, & Sun, 2003) and event-based (Assfalg, Bertini, Colombo, Bimbo, & Nunziati, 2003; T. Wang, et al., 2006). The former uses either supervised or unsupervised learning to recognize some basic semantic structures (such as canonical view in a baseball game or play/break in a soccer game) that can serve as an intermediate representation supporting semantics-oriented video retrieval, but usually cannot deliver high-level semantics directly. The latter one provides a better understanding of the video content by detecting and extracting the events-of-interest or the highlights, which could be very specific and task-dependent and requires sufficient and diverse training examples. Because these two approaches are complementary in nature, researchers have investigated how to integrate both of them in one computational framework. For example, a mid-level representation framework was proposed for semantic sports video analysis involving both temporal structures and events hierarchy (L. Y. Duan, Xu, Chua, Q. Tian, & Xu, 2003) and a mosaic-based generic scene representation was developed from video shots and used to mine both events and structures (Mei, Ma, Zhou, Ma, & Zhang, 2005).

Machine learning is one of the most feasible approaches for semantic video analysis (Kokaram, et al., 2006). Hidden Markov models (HMMs) have been the most popular tool in this area (Xie, Chang, Divakaran, & Sun, 2003). Our goal is try to establish a HMM-based machine learning framework that supports both structure analysis and event analysis and delivers the rudimentary building blocks for high-level semantic analysis. There are two major distinctions in our research. One is the concept of semantic space which supports explicit semantic modeling and specifies both mid-level and high-level semantic structures as well as relevant visual features, so that the task is formulated as two inference problems. The other is a new HMM that offers multi-faceted advantages for semantic video analysis.