This chapter gives an overview of the challenges that have to be mastered while working with audio. The vision of ubiquitous computing involves challenges for the future workplace. Tasks to be performed by workers are becoming more and more complex, which leads to an ever-increasing need to deliver information to workers. This can be information from a manual or instructions on how to proceed with the current task. Workers typically have their hands busy and use of a mouse or a keyboard will force them to stop working. Mouth and ear interaction can be performed without focusing attention on the device. But audio is also a medium that is not easy to handle. This chapter provides an understanding of why audio-based interfaces are difficult to handle and you will also be provided with some pointers as to how these challenges can be mastered to improve the quality of applications involving mouth & ear interaction.