THE INTEGRAL MUSIC CONTROLLER: INTRODUCING A DIRECT EMOTIONAL INTERFACE TO GESTURAL CONTROL OF SOUND SYNTHESIS R. Benjamin Knapp, PhD BioControl Systems, LLC Pr Sebastopol, CA 95472 ABSTRACT This paper describes the concept of the integral music controller (IMC), a controller that combines gestural interface with direct emotional control of a digital musical instrument. This new controller enables the performer to move smoothly between direct physical interaction with an acoustical musical instrument, and gestural/emotional control of the instrument's physical model. The use of physiological signals to determine gesture and emotion is an important component of the IMC. The design of a wireless IMC using physiological signals is described and possible mappings to sound synthesis parameters are explored. Controlling higher level musical systems such as conducting and style modelling is also proposed. 1. INTRODUCTION One (arguably the primary) goal of a musical performer is to express abstract thoughts and emotions through sound by interaction with a musical instrument and have this sound ultimately create a thoughtful and, hopefully, emotional response in the listener. Thus, a musical performance can be defined as the communication of emotion through sound. It would then follow that one definition of a musical instrument is a device that enables the expression of emotion through sound. Musical instruments can take many forms and can be classified as either a traditional acoustic instrument or, as defined by Wanderley [1] and others, a digital musical instrument. Interaction with the traditional instrument requires physical gestures that, as Wanderley states, are inseparable from the sound production. Many times, in order to convey the intended emotions of the composer and performer, the physical gestures require a level of proficiency that is difficult to obtain. It is indeed possible that regardless of performer proficiency, the physical interface does not allow for the expression of the desired emotion. As stated, the digital musical instrument enables the separation of gesture from sound production. This creates a revolutionary opportunity for the creation of an integral music controller (IMC) defined as a controller that: 1. Creates a direct interface between emotion and sound production unencumbered by the physical interface. 2. Enables the musician to move between this direct emotional control of sound synthesis and the physical interaction with a traditional acoustic instrument and through all of the possible levels of interaction in between. Perry R. Cook, PhD inceton Computer Science (also Music) Princeton, NJ 08540 This paper will introduce the IMC in three parts: 1. Definition of the IMC 2. Measurement of emotion in the context of a musical instrument (including an example system) 3. Use of the IMC as input to sound synthesis 2. THE INTEGRAL MUSIC CONTROLLER The term "integral" in "integral music controller" refers to the integration into one controller of the pyramid of interface possibilities as shown in Figure 1. Using an IMC, a performer can move up and down through the interface possibilities. For example [2] [3], in a piece entitled "Tibet", composed for NIME 2002, a multimodal control system was used that allowed the performer, Atau Tanaka, to: * play a Tibetan bowl (traditional Physical Interface) * control pre-recorded components of bowl sound and parameters of synthesized sound simultaneously while playing the bowl (augmented interface) * and step away from the instrument (the bowl) and continue to control the pre-recorded and physical model components of the bowl (remote interface) This multimodal controller used physiological signals, specifically EMG, and motion sensors to achieve the multiple levels of control of the bowl. The introduction of a direct interface between emotion and sound would have made this device a true integral controller. As shown in Figure 1, the introduction of direct measurement of emotion to digital musical instrument control represents the completing of the pyramid of possible interfaces. Only with a direct interface to emotion is a truly integral controller possible. /A Emotion Interface Remote Interface Augmented Interface / // Traditional Physical Interface 4- Number of Existing Interface Devices Figure 1: Pyramid of interfaces for controlling a digital musical instrument (categories loosely adapted from [1]). Note the decreasing number of existing interface devices as you move up the pyramid. The integral music controller (IMC) has elements of all interfaces.
Top of page Top of page