The abstraction of a long video is often useful to a user in determining whether the video is worth viewing or not. In particular, video abstraction guarantees users of digital libraries with the fast, safe and reliable access of video data. Two approaches, such as summary sequences and highlights are possible in video abstraction. The summary sequences are good for documentaries because they give an overview of the contents of the entire video, whereas highlights are good for movie trailers because they contain only the most interesting video segments. The video abstraction can be generated by three steps: analyzing video, selecting video clips, and synthesizing the output. In the analyzing video step, salient features, structures, or patterns in visual information, audio information, and textual information are detected. In the selecting step, meaningful clips are selected from detected features in the previous step. In the output synthesis step, the selected video clips are composed into the final form of the abstract. In this chapter, we will discuss various video abstraction techniques for digital libraries. In addition, we will also discuss a context-based video abstraction method in which contextual information of the video shot is computed. This method is useful in generating highlights because the contextual information of the video shot reflects semantics in video data.