Mobile devices are replacing the desktop computer in most spheres outside the workplace. This development brings a problem to video streaming services, as wireless channels are fundamentally error-prone, whereas video compression depends for most of its gains on predictive coding. The H.264 codec family has included a good number of error resilience facilities to counter-act the spatio-temporal error propagation brought on by packet loss. This chapter outlines these facilities before examining ways in which predictive coding can be temporally restrained. In particular, intra-refresh techniques are the focus, as these bring additional utility to the video stream. For example, the chapter compares periodic and gradual intra-refresh, each of which provides recovery points for the decoder and also allow stream switching or joining at these points. Thus, in intra-coding, the more normal temporal prediction is temporally replaced by spatial prediction, at a cost in coding efficiency but allowing a decoder in a mobile device to reset itself. After a review of research into this area, the chapter provides a case study in non-periodic intra-refresh before considering possible future research directions.
TopIntroduction
As fixed and wireless networks converge (Watson, 2009; Ahson & Ilyas, 2010; Paul, 2011), there is a need to bring the user’s video experience within the mobile wireless network closer to that of the fixed network. Real-time delivery of video is required for streaming applications such as Web TV, Internet Protocol TV (IPTV) in its various forms (Simpson, 2008), and the class of interactive video applications such as video conferencing and soft videophone. However, wireless access continues to present a bandwidth constriction, especially as high-definition video (720p, i.e. 1280×720 pixels/frame in progressive display at 30 frames/s) is extended (Bing, 2010) to displays on laptops and other mobile devices such as smart-phones. In terms of streaming to mobiles, the trend (e.g. Apple's FaceTime) is towards full VGA resolution (640 × 480 pixels/frame at 30 frames/s). This implies that compression is still very necessary and, because of the predictive nature of that compression, there is an ever present risk of spatio-temporal error propagation. The main differentiating feature of wireless networks is the various channel impairments that can occur. Therefore, ways are sought to arrest error propagation and intra-refresh techniques provided by the video codec itself are a way to do so, along with other forms of error resilience (Stockhammer & Zia, 2007).
Intra-coded video data within I-pictures relies on spatial reference within the video picture and, hence, is unaffected by the corruption of previous pictures. This is in contrast to inter-coded video data, which takes reference from past (P-pictures) or even future video pictures (or both temporal directions) (B-pictures). In both these forms of coding, it is the difference image (or residual data) that is processed in subsequent stages of a hybrid video encoder (Ghanbari, 2011). For networked TV, I-pictures can serve as a point at which the TV channel can be switched or zapped. For streaming of live video, periodic I-pictures can also act as the point in time of joining the broadcast stream. Pseudo-Video Cassette Recorder (VCR) functions (otherwise known as trick modes) such as fast forward, rewind and so on, can also be based on I-pictures. Periodic I-pictures are, in fact, the usual way to provide intra-refresh in those ways but it is also possible to provide Gradual Decoding Refresh (GDR) (Hannuksela, et al., 2004) by including intra-refresh Macroblocks (MBs) (the compression building blocks) within inter-coded pictures, provided the refreshment pattern is carefully considered.
Related to the issue of intra-refresh is the question of the Group-of-Picture (GoP) structure and size, as this can be varied statically and dynamically. It is also dependent on codec profile. For streaming to mobile devices it may actually be advisable not to use traditional intra-refresh I-pictures (or slices), provided distributed intra-refresh MBs can be used in some way to provide the functionality once served by I-pictures (see the previous remarks on GDR). It should also be borne in mind that some intra-coding is naturally inserted even on nominally inter-coded pictures and that in some circumstances it will be necessary to constrain intra-coding reference to avoid temporal error corruption arising from spatial reference to inter-coded MBs that themselves are corrupted. The authors of this chapter (Ali, et al., 2012) have shown that the insertion of a cyclic intra-coded line of MBs on a per-video picture basis as a convenient way to mitigate error propagation, if less-active video sequences are transmitted over wireless or other ‘lossy’ links. For more-active sequences (ones with substantial inter-picture motion), periodic insertion of I-pictures is preferable. Randomized insertion of intra-coded MBs (Haskell & Messerschmitt, 1992; Côté & Kossentini, 1999) is an alternative to a cyclic intra-coded line of MBS or some other more visually pleasing pattern. However, a random pattern may result in duplication of intra-coded MBs in successive pictures. However, there are some subtleties involved in this, as the randomized insertion of intra-coded MBs in the Joint Model (JM) reference software for the H.264/Advanced Video Coding (AVC) codec (Wiegand, et al., 2003) is not ‘completely random’ in the sense of earlier writers (Haskell & Messerschmitt, 1992; Côté & Kossentini, 1999). In fact, previous MBs are not duplicated and, once all MBs have been replaced with intra-coded MBs, the random cycle repeats, exactly as before.