MusicStory: An Autonomous, Personalized Music Video Creator

Abstract

This chapter covers some of the challenges in storytelling with music. We describe the MusicStory system which creates music videos from personal media (audio and image) collections. Using a song’s lyrics, MusicStory finds and presents word/image associations. It takes the emotional experience of listening to music, amplifies it and heightens its visceral appeal by externalizing concrete and visual imagery intrinsic in the music. The retrieved images vary in their association - some semantically on point and some distant. The flow of imagery moves with the pace of the song: providing quick transitions through fast songs, and leisurely transitions through slower songs. In addition, MusicStory uses vocal segmentation to direct the video and alter how images are displayed. We discuss the creative and technical challenges in this work as well as how it was deployed in several fashions.