Real-Time Classification of Water Spray and Leaks for Robotic Firefighting

Introduction

To limit the exposure and risk of firefighters and to improve effectiveness of indoor firefighting operations, robots have been proposed as a support tool. One aspect of robotic firefighting is a fire suppression system that autonomously detects and extinguishes fires effectively and efficiently. Many versions of firefighting robots (Kim, Y.D., Kim, Y.G., Lee, Kang, & An, 2009; Penders et al., 2011; Liljeback, Stavdahl, & Beitnes, 2006; Pack, Avanzato, Ahlgren, & Verner, 2004; Dearie, Fisher, Rajala, & Wasson, 2001; Miyazawa, 2002) have been developed; however, most use an operator to remotely control them or have not been tested in realistic fire scenarios. In the shipboard autonomous firefighting robot (SAFFiR) program, a humanoid robot was developed to autonomously locate and suppress fires located onboard naval ships in indoor structures. Advancements including artificial intelligent algorithms and perception systems (Kim, Starr, & Lattimer, 2015; Kim, Keller, & Lattimer, 2013; Starr & Lattimer, 2012, 2013, 2014) along with autonomous fire suppression approaches (McNeil, Starr, & Lattimer 2013) were developed through the program to further advance firefighting robots. In addition, a probabilistic classification technique was proposed to distinguish fire and smoke from reflections or hot objects (Kim & Lattimer, 2015) for use in determining a heading direction for the robot to locate a fire. Due to the smoke-filled environments that are encountered in firefighting, IR cameras were selected over visual RGB cameras. Handheld thermal infrared cameras (TIC) are typically used to aid in firefighting tasks within smoke-filled environments (Amon, Benetis, Kim, & Hamins, 2004; Amon, Bryner, & Hamins, 2004; Amon & Ducharme, 2009; Maxwell, 1971). For TICs and IR cameras that absorb IR radiation in the long wavelength IR (LWIR, 7-14 micron range), they are able to image surfaces even in dense smoke and zero visibility environments (Starr & Lattimer, 2014; Chacon-Murguia & Perez-Vargas, 2011). Classification of objects with thermal imagers is needed to provide the robotic system with the information required to efficiently perform firefighting tasks even in clear and low visibility environments.

Water is an unstructured, moving object that makes its segmentation and classification complex. There are many different approaches to segmenting and classifying moving objects in video sequences but active water segmentation with classification techniques in IR have not been widely studied. Some spray segmentation approaches include edge detection (Pastor, Arregle, & Palomares, 2001; Jeong, Lee, & Ikeda, 2007), which relies on high contrast between warm temperature diesel spray and a static blank background. Water spray or leaks in IR images contain a high level of noise and can be very low contrast making traditional edge-based detection algorithms unreliable (Nanda & Davis, 2002). Water does not have a distinct structure so classification techniques relying on shape cannot be used in low contrast scenarios. Image segmentation based on Otsu’s method, a cluster-based image thresholding technique, has been used when the object or spray has high contrast with the background (Shao & Yan, 2008; Zhao-dong & Zeng-ping, 2010; Kang & Chen, 2009) and can be separated as foreground. In the suppression system, the water object is often low intensity (low temperature) and merges with parts of the background. Background subtraction techniques such as Gaussian Mixture Models (GMM) (Stauffer & Grimson, 2000) and adaptive background subtraction are unreliable in IR video due to the low signal-to-noise ratio (Latecki, Miezianko, & Pokrajac, 2005). A reliable background model cannot be used in dynamic firefighting scenes involving fire, water, smoke, and reflections due to the large differences between an image and a background model that are produced as the scene changes or the system moves due to robot motion or from scanning the area.