What is Segmental HMMs

This chapter summarizes the authors’ recent research on the hidden Markov model (HMM)-based machine learning approaches to sports video mining. They will advocate the concept of semantic space that provides explicit semantic modeling at three levels, high-level semantics, mid-level semantic structures, and low-level visual features. Sports video mining is formulated as two related statistical inference problems. One is from low-level features to mid-level semantic structures, and the other is from midlevel semantic structures to high-level semantics. The authors assume that a sport video is composed of a series of consecutive play shots each of which contains variable-length frames and can be labelled with certain mid-level semantic structures. In this chapter, the authors present several HMM–based approaches to the first inference problem where the hidden states are directly associated with mid-level semantic structures at the shot level and observations are the visual features extracted from frames in a shot. Specifically, they will address three technical issues about HMMs: (1) how to enhance the observation model to handle variable-length frame-wise observations; (2) how to capture the interaction between multiple semantic structures in order to improve the overall mining performance; (3) how to optimize the model structure and to learn model parameters simultaneously. This work is the first step toward the authors’ long-term goal that is to develop a general sports video mining framework with explicit semantic modeling and direct semantic computing.