As real-time video streaming moves to the mobile Internet, there is a greater need to protect fragile compressed bit-streams from the impact of lossy wireless channels. Though forward error correction (FEC) has a role, if it is applied without adaptation, it may introduce excessive communication overhead. Alternatively, error resilience methods provide additional protection at the application layer of the protocol stack, without replication of any protection already provided at the data-like layer. In this chapter, a case study shows that these resilience methods can be applied adaptively through stream switching according to channel conditions. Error resilience can work hand-in-hand with error concealment, again applied through source coding. There are many error resilience and concealment methods, which this chapter surveys at a tutorial level. The chapter also includes an overview of video streaming for those unfamiliar with the topic. Though error concealment is a non-normative feature of the H.264/AVC (Advanced Video Coding) codec standard, there is a range of new techniques that have been included within the Standard such as flexible macroblock ordering and stream switching frames. The chapter additionally reviews error concealment provision, including spatial, temporal, and hybrid methods. Results show that there are tradeoffs between the level of protection and the level of overhead, according to the severity of the wireless channel impairment.
TopIntroduction
Video compression efficiency improves with each successive codec, with each bit carrying more information. Consequently, a video bit-stream becomes more error sensitive, resulting in significant quality degradation in error-prone channels. This is especially the case for the extension of video services such as Internet Protocol TV (IPTV) (Park & Jeong, 2009)to mobile, broadband wireless networks. The efficiency is also a problem both for sensor networks and mobile ad hoc networks, because of the risk of routing link failures. This Chapter presents source coding methods to combat these errors, resulting in robust video streaming in such networks.
We have considered source coding approaches to providing robust video streaming. The aim of source coding is to reduce the bit-rate of the uncompressed video by exploiting redundancies or correlations within the uncompressed video. Source coding algorithms are organized within a codec. A codec consists of an encoder/decoder pair corresponding to the sender/receiver in a transmission path. Suppose one takes the smallest of spatial resolutions commonly now employed, Common Intermediate Format (CIF) (352 × 288 pixel/picture) with 8-bit precision for each luminance and chrominance component (with half resolution for each chrominance component to account for the reduced sensitivity to color information of the human visual system (HVS)). Taking a typical frame rate of 25 frame/s then the raw video data-rate is 30 Mbps. The most common capacity of the widely-deployed Asymmetric Digital Subscriber Line (ADSL) form of access network is typically 1.25 Mbps. Therefore, even if it were commercially prudent to stream raw video, there are practical limits, due to the potential bottleneck at the access network. Moreover, a one-minute uncompressed video would occupy 225 Mbytes of storage, which explains why even short range transmission of uncompressed video (as in High-Definition Multimedia Interface (HDMI)) effectively employs streaming. Of course, apart from being employed in networked and broadcast communication of video, codecs are also necessary for storage of video on digital media such as DVDs.
Standard hybrid video codecs (Ghanbari, 2003) exploit various forms of redundancy to achieve efficient data compression. The correlation of the three color signals output from a video camera is reduced prior to codec processing by conversion into a different color space (for example, RGB to recommendation CCIR-601). Perceptual coding is not generally exploited for video compression, though it may have a future role (Tan &Wu, 2006). In fact, there are basically two forms of source-coded video picture, depending on whether spatial or temporal redundancy is exploited. Both these forms of compression are ‘lossy’ in the sense that the original video picture cannot be exactly reproduced after decoding. However, as the viewer is not normally aware of the exact nature of any natural scenes captured by a camera and as the HVS does not hold an exact representation of natural scenes, loss of fidelity can be tolerated.
Of these two forms of redundancy, still-image codecs such as JPEG2000 (Taubman & Marcellin, 2002) only exploit spatial redundancy. However, it is intuitively obvious that the similarity between successive video pictures (except when there are scene cuts or rapid motion of the camera or objects within the scene) in a video sequence can potentially lead to large reductions in bit rate. Another form of redundancy, statistical redundancy, is exploited through entropy coding at the output stage of both still-image and video codecs. Entropy coding is a lossless form of compression. Therefore, it is particularly susceptible to data loss, as each successive codeword depends on the preceding sequence of codewords since the last reset point. On the other hand, due to the tolerance of the HVS to some loss of spatio-temporal information it is possible to replace lost data with similar data, for example by blocks of data from the previous picture. In fact, error resilience together with error concealment, the topics of this Chapter can both act to help restore a video picture after data is lost or arrives in corrupted form at a wireless receiver.